Commits on Source (26)
-
DETCHART Jonathan authored
-
-
DETCHART Jonathan authored
-
STEVAN Antoine authored
this adds three badges to the README: - the latest release (links to the [release page](https://gitlab.isae-supaero.fr/dragoon/komodo/-/releases)) - the GitLab CI status (links to the [GitLab pipeline dashboard](https://gitlab.isae-supaero.fr/dragoon/komodo/-/pipelines)) - the GitHub CI status (links to the [GitHub workflow runs dashboard](https://github.com/dragoon-rs/komodo/actions))
-
STEVAN Antoine authored
this MR adds two new badges to the README - the crate (links to the [Komodo crate](https://crates.io/crates/komodo)) - the documentation (links to the [doc](https://docs.rs/komodo/latest/komodo/))
-
STEVAN Antoine authored
see the successful CI on GitHub: https://github.com/dragoon-rs/komodo/actions/runs/11950087413/job/33310881686
-
STEVAN Antoine authored
FRI protocol from [`dragoon/fri`](https://gitlab.isae-supaero.fr/dragoon/fri) ## changelog - add binary assets to be used as inputs - add `fri` and `fri_test_utils` as local dependencies until [`dragoon/fri`](https://gitlab.isae-supaero.fr/dragoon/fri) becomes public - add `fri` feature and module (see section below for the public definitions) - fix bug in 32bd6566 - bump Rust in e7a2c244 - add a versatile example - add Nushell pipeline to run benchmarks and plot results - add some tests ## `fri` module API ```rust struct Block<F: PrimeField, H: Hasher> ``` ```rust fn evaluate<F: PrimeField>(bytes: &[u8], k: usize, n: usize) -> Vec<Vec<F>> ``` ```rust fn encode<F: PrimeField>( bytes: &[u8], evaluations: Vec<Vec<F>>, k: usize, ) -> Vec<fec::Shard<F>> ``` ```rust fn prove<const N: usize, F: PrimeField, H: Hasher, P>( evaluations: Vec<Vec<F>>, shards: Vec<fec::Shard<F>>, blowup_factor: usize, remainder_plus_one: usize, nb_queries: usize, ) -> Result<Vec<Block<F, H>>, KomodoError> where P: DenseUVPolynomial<F>, for<'a, 'b> &'a P: Div<&'b P, Output = P>, <H as rs_merkle::Hasher>::Hash: AsRef<[u8]>, ``` ```rust fn verify<const N: usize, F: PrimeField, H: Hasher, P>( block: Block<F, H>, domain_size: usize, nb_queries: usize, ) -> Result<(), KomodoError> where P: DenseUVPolynomial<F>, for<'a, 'b> &'a P: Div<&'b P, Output = P>, <H as rs_merkle::Hasher>::Hash: AsRef<[u8]>, ``` ```rust fn decode<F: PrimeField, H: Hasher>(blocks: Vec<Block<F, H>>, n: usize) -> Vec<u8> ``` ## results ### times      ### sizes    
-
This MR adds an option to perform erasure coding using FFT rather than using a matrix. It also adds the field FP128 in the list of curves Note that there is a redundant function `random_loss` into benchmarks/bin/fec.rs and examples/fec.rs
-
STEVAN Antoine authored
this replaces the "_nothing to do_" messages from Nushell commands with more explicite errors.
-
STEVAN Antoine authored
## changelog - add more snippets and instructions - add missing imports - update the "atomic operations" section - add a table of contents
-
STEVAN Antoine authored
in order to help catch issues with the various Nushell scripts and modules in the source base, i propose to add a CI script that will check them all. below is an example error, e.g. when introducing a syntax error in `.env.nu`: ``` Error: × Failed to parse content: │ file: .env.nu │ err: Expected keyword. ```
-
STEVAN Antoine authored
this should avoid the following error when trying to rust the LSP in a toolchain which does not have the `rust-analyzer` component and without running `rustup component add rust-analyzer` manually: ``` error: Unknown binary 'rust-analyzer' in official toolchain 'stable-x86_64-unknown-linux-gnu'. ```
-
STEVAN Antoine authored
mainly fixes a few typos and improves `algebra` and `semi_avid` tests ## changelog - fix some typos and notes in the documentation - test more cases for `algebra::split_data_into_field_elements` and `algebra::merge_elements_into_bytes`, more data lengths and more modulus respectively - remove a useless `::<Vec<_>>` on a `collect` in `fec` - `semi_avid::tests` - refactor `bls12-381` into a constant - write an "_attack_" function to alter a particular block, `attack<F, G>(block: Block<F, G>, c: usize, base: u128, pow: u64) -> Block<F, G>` - pass a list of attacks, i.e. `attacks: Vec<(usize, usize, u128, u64)>`, to `verify_with_errors_template`, use the same as the previous hardcoded one - pass a list of "_recodings_", i.e. `recodings: Vec<Vec<usize>>`, to `verify_recoding_template` and assert the number of blocks, e.g. `vec![vec![2, 3], vec![3, 5]]` means that recoding 2 and 3 together should verify, and same with 3 and 5, and also that there must be at least 6 blocks - pass a list of "_recodings_", i.e. `recodings: Vec<(Vec<Vec<usize>>, bool)>`, to `end_to_end_with_recoding_template` and assert the number of blocks and the number of source shards, e.g. `vec![(vec![vec![0, 1], vec![2], vec![3]], true)]` means that trying to decode with blocks 2, 3 and a recoded block from 0 and 1 should work, and also that there must be at least 4 blocks and at most 3 source shards - pass `k` and `n` to `run_template`
-
STEVAN Antoine authored
the idea is to make the installation of Nushell easier to maintain, especially regarding versions, currently pinned to `0.95.0` > successful run on GitHub: [13032726635](https://github.com/dragoon-rs/komodo/actions/runs/13032726635) this is also to allow easier testing locally with the same Nushell version as in the CI, e.g. ```bash # install in `~/.local/bin/` and have multiple versions make install-nu hash=$(/tmp/nu --no-config-file --commands 'version | get commit_hash') nu_bin=$"$HOME/.local/bin/nu-$hash" cp /tmp/nu $nu_bin make NU=$nu_bin show test ``` or ```bash # install in the repo and overwrite each time make NU_DEST=. install-nu make NU=./nu show test ``` # changelog - Makefile - split the global `.PHONY` rule into _atomic_ rules next to each _phony_ rule - define `NU` and `NU_FLAGS` to allow changing which and how Nushell runs - define `NU_ARCH`, `NU_VERSION`, `NU_BUILD` and `NU_DEST` for Nushell installation - tweak the output of `make show` a bit - add `print-%` rules to print `Makefile` variables, e.g. `make print-NU_FLAGS` would print `--no-config-file` - add `make install-nu` to replace the ones from the CIs - GitLab CI - use `make install-nu` - export `PATH` with `make print-NU_DEST` - GitHub CI - use `make install-nu` instead of [github.com:hustcer/setup-nu](https://github.com/hustcer/setup-nu) - export `PATH` with `make print-NU_DEST` and `GITHUB_ENV`
-
STEVAN Antoine authored
this is an attempt at making benchmarks easier to work with - `benchmarks run` will read benchmarks from NUON data and run them - `benchmarks plot` will plot benchmark results - the output format will be something - a directory whose name is the hash of the CPU spec and the Komodo commit hash - contains `cpu.json` with the CPU info - contains `komodo.txt` with the Komodo commit hash - contains NDJSON result files >
💡 **Note** > > results will typically be uploaded to https://gitlab.isae-supaero.fr/dragoon/komodo-benchmark-results >💡 **Note** > > this MR goes alongside the [`komodo-benchmark-results@restart`](https://gitlab.isae-supaero.fr/dragoon/komodo-benchmark-results/-/compare/main...restart) branch ## changelog - bump Nushell to 0.101.0 - parallel `$in` => remove useless `let input = $in` when possible, e.g. still required when using the function's `$in` in a `for` loop - `group-by` changed => `group-by x --to-table` will now produce a table with columns `x` and `items` instead of `group` and `items` as in 0.95.0 - add link to results repo - the Nushell benchmarks lib - rename `--force` to `--no-confirm (-y)` - add `--append` - reject columns that GPLT will complain about, e.g. `$.points.k` for the FRI plots - add `--save` to the FRI plot - move the "field" and "curve group" benchmarks from `benchmarks/src/bin/operations/` to `benchmarks/src/bin/` - remove `benchmarks/params/fri.nu` because it's been uniformized with the other methods - rewrite the README - add main function to `benchmarks/` that runs the benchmarks from a NUON record specification - simplify the output of FRI run ## TODO - [x] fix "_atomic operations_" (done in 4f69a1d6) - [x] check that _plotting_ still works ## images ### Field     ### Linear algebra    ### FEC      ### ZK   ### FRI         -
STEVAN Antoine authored
this is to avoid having to change the toolchain in the CI everytime. `rust-toolchain.toml` and `.gitlab-ci.yml` will have to be updated in pairs when bumping / changing the Rust toolchain.
-
Added a html header files, an added metadata line in Cargo.toml and some formatting changes in the doc to have Latex formulas in docs.rs. It works locally, should work online
-
STEVAN Antoine authored
## usage ```bash nu scripts/check-mirror.nu ...[ https://gitlab.isae-supaero.fr/dragoon/komodo https://github.com/dragoon-rs/komodo main ] ``` will - check the Nushell version and throw a warning if mismatch - add temporary remotes with random names - fetch the temporary remotes - get the revision of the `$branch` for both temporary remotes - clean the temporary remotes - show a message if the two revisions are not the same and how many commits they are different > **Note** > > this does not conflict with #17 -
STEVAN Antoine authored
this should close #17. SACLIN has been moved to [`gitlab.isae-supaero.fr:dragoon/komodo.nu`](https://gitlab.isae-supaero.fr/dragoon/komodo.nu).
-
STEVAN Antoine authored
This is to complete the `check-mirror.nu` script with information about the CI runs. The current output of ```bash nu scripts/check-mirror.nu ...[ https://gitlab.isae-supaero.fr/dragoon/komodo https://github.com/dragoon-rs/komodo main ] ``` is ``` [INF] adding remotes [INF] fetching [INF] cleaning [ OK] mirror is up to date [INF] pulling mirror runs #─┬────id─────┬────────────────head_sha────────────────┬─status──┬conclusion┬run_started_at 0 │11950217125│de4266c0│completed│success │4 months ago 1 │11950604920│202d8bcc│completed│success │4 months ago 2 │11950873603│702cd5eb│completed│success │4 months ago 3 │12028360471│57a96c02│completed│success │4 months ago 4 │12028432364│07602875│completed│success │4 months ago 5 │12028781541│f2a76fbb│completed│success │4 months ago 6 │12631165433│11a5b59d│completed│success │2 months ago 7 │13008608156│cc412625│completed│success │2 months ago 8 │13033206333│5511404f│completed│success │2 months ago 9 │13113244065│ab0d9a8b│completed│success │2 months ago 10│13679188387│3c8afac4│completed│success │3 weeks ago 11│14191103891│b5381fda│completed│success │a day ago 12│14214853530│b7e2d99d│completed│success │an hour ago 13│14215431531│0d7a5cd6│completed│success │32 minutes ago ──┴───────────┴────────────────────────────────────────┴─────────┴──────────┴────────────── ``` -
STEVAN Antoine authored
that's an attempt at using Rust to build itself. this is using [`gitlab.isae-supaero.fr:a.stevan/nob.rs@e4b03cdd`](a.stevan/nob.rs@e4b03cdd). >
💡 **Note** > > to be honest, this is not a 100% replacement of the `Makefile`... > > `make.rs` does a lot more and provides a full CLI with easy-to-use options, e.g. instead of `make fmt` and `make fmt-check`, we now have `./make.rs fmt` and `./make.rs fmt --check` > > (see the API below) ## the API ``` Usage: make [OPTIONS] [COMMAND] Commands: fmt Formats the code check Checks the code clippy Runs Clippy test Runs the tests version Shows the version of all the tools used, doc Builds the documentation help Print this message or the help of the given subcommand(s) Options: -h, --help Print help -V, --version Print version ``` ``` Usage: make fmt [OPTIONS] Options: -c, --check Only checks instead of really formatting ``` ``` Usage: make check ``` ``` Usage: make clippy ``` ``` Usage: make test [OPTIONS] Options: -v, --verbose Be extra verbose with the output of the tests -e, --examples Run the examples instead of regular tests ``` ``` Usage: make version ``` ``` Usage: make doc [OPTIONS] Options: -o, --open Open the documentation in the browser -p, --private Document private items -f, --features Document all features ``` ## running the pipeline in the GitHub mirror ```bash const GH_API_OPTIONS = [ -H "Accept: application/vnd.github+json" -H "X-GitHub-Api-Version: 2022-11-28" ] let res = gh api ...$GH_API_OPTIONS /repos/dragoon-rs/komodo/actions/runs | from json ``` ```bash let runs = $res.workflow_runs | where head_branch == "use-nob-to-build" | select id head_sha status conclusion run_started_at | into datetime run_started_at | sort-by run_started_at ``` ```bash $runs | update id { $"[`($in)`]\(https://github.com/($GITHUB_MIRROR)/actions/runs/($in)\)" } | update run_started_at { format date "%Y-%m-%dT%H:%M:%S" } | to md --pretty ``` | id | head_sha | status | conclusion | run_started_at | | ------------------------------------------------------------------------------ | ---------------------------------------- | --------- | ---------- | ------------------- | | [`14237650542`](https://github.com/dragoon-rs/komodo/actions/runs/14237650542) | d67f1cfd | completed | success | 2025-04-03T07:44:14 | | [`14237741570`](https://github.com/dragoon-rs/komodo/actions/runs/14237741570) | 9ef598a1 | completed | success | 2025-04-03T07:49:40 | | [`14238086977`](https://github.com/dragoon-rs/komodo/actions/runs/14238086977) | 0a79edf3 | completed | success | 2025-04-03T08:09:13 | | [`14238175174`](https://github.com/dragoon-rs/komodo/actions/runs/14238175174) | a84b2b12 | completed | success | 2025-04-03T08:13:52 | | [`14239395984`](https://github.com/dragoon-rs/komodo/actions/runs/14239395984) | 8594c9bf | completed | success | 2025-04-03T09:16:00 | -
STEVAN Antoine authored
this removes the `check-nushell-files.nu` script i left in !200.
-
STEVAN Antoine authored
This should close #19. ## changelog - triggers pipeline in the GitHub mirror on each push - add `./scripts/get-mirror-pipelines.nu` ## the pipelines >
❗ **Important** > > i just had to add `run-mirror-pipeline-everytime` to the list of protected branches and trigger the push with an empty commit and it runs > > and then a single call to `nu scripts/get-mirror-pipelines.nu "run-mirror-pipeline-everytime"` gives the table below | id | head_sha | status | conclusion | run_started_at | | ------------------------------------------------------------------------------ | ---------------------------------------- | --------- | ---------- | ------------------- | | [`14238533729`](https://github.com/dragoon-rs/komodo/actions/runs/14238533729) | 2f32ee59 | completed | success | 2025-04... -
STEVAN Antoine authored
This adds a Dockerfile to build an image for the pipelines, once and for all, until we decide to bump the Rust toolchain. ## changelog - add `.gitlab-ci.dockerfile` which - uses `rust:latest` as a base (it appears using `alpine:latest` is both bigger and slower
🤷 ) - installs basic system dependencies - uses `rust-toolchain.toml` to install the toolchain of the project - installs `cargo-script` - use [`gitlab-registry.isae-supaero.fr/dragoon/komodo:bcb0e6b5`](https://gitlab.isae-supaero.fr/dragoon/komodo/container_registry/42) in the GitLab pipelines and [`ghcr.io/dragoon-rs/dragoon/komodo:bcb0e6b5`](https://github.com/orgs/dragoon-rs/packages/container/dragoon%2Fkomodo/388672772?tag=bcb0e6b5f73420762f6208700a43291e0066c2c3) in the GitHub pipelines and remove the "manual" dependency installation - add `./make.rs container`, `./make.rs container --login` and `./make.rs container --push` - uses `nob.rs@7ea6be8` to capture output of... -
STEVAN Antoine authored
## examples ```bash ./make.rs container list --json | from ndjson | into datetime CreatedAt | into filesize Size VirtualSize | reject CreatedSince ``` or ```bash ./make.rs container list --json | from ndjson | into datetime CreatedAt | into filesize Size VirtualSize | reject CreatedSince | select ID Repository Tag CreatedAt VirtualSize | update Tag { str substring 0..<7 } ``` ## changelog - transform options of `container` to sub-subcommands - `container` --> `container build` - `container --login` --> `container login` - `container --push` --> `container push` - add `container list` to print the local images for the GitLab and GitHub repositories - `container list` will print in a pretty table - `container list --json` will print as NDJSON, i.e. one image per line as JSON - use wrappers around `nob::run_cmd_as_vec_and_fail!` - `extend_and_run` to run a partial command with an extra vector ... -
STEVAN Antoine authored
This MR explains the use of `make.rs` and `nob.rs` in the README for newcomers.
Showing
- .env.dockerfile 14 additions, 0 deletions.env.dockerfile
- .github/workflows/ci.yml 32 additions, 0 deletions.github/workflows/ci.yml
- .gitignore 6 additions, 0 deletions.gitignore
- .gitlab-ci.yml 6 additions, 23 deletions.gitlab-ci.yml
- .nu.cfg 2 additions, 0 deletions.nu.cfg
- Cargo.toml 15 additions, 2 deletionsCargo.toml
- Makefile 0 additions, 45 deletionsMakefile
- README.md 41 additions, 19 deletionsREADME.md
- assets/128_4096_3.bin 0 additions, 0 deletionsassets/128_4096_3.bin
- assets/128_4_3.bin 0 additions, 0 deletionsassets/128_4_3.bin
- benchmarks/Cargo.toml 3 additions, 2 deletionsbenchmarks/Cargo.toml
- benchmarks/README.md 141 additions, 65 deletionsbenchmarks/README.md
- benchmarks/mod.nu 259 additions, 0 deletionsbenchmarks/mod.nu
- benchmarks/nu-lib/commit.nu 19 additions, 12 deletionsbenchmarks/nu-lib/commit.nu
- benchmarks/nu-lib/fec/plot.nu 9 additions, 9 deletionsbenchmarks/nu-lib/fec/plot.nu
- benchmarks/nu-lib/fec/run.nu 22 additions, 11 deletionsbenchmarks/nu-lib/fec/run.nu
- benchmarks/nu-lib/fri/mod.nu 2 additions, 0 deletionsbenchmarks/nu-lib/fri/mod.nu
- benchmarks/nu-lib/fri/plot.nu 108 additions, 0 deletionsbenchmarks/nu-lib/fri/plot.nu
- benchmarks/nu-lib/fri/run.nu 78 additions, 0 deletionsbenchmarks/nu-lib/fri/run.nu
- benchmarks/nu-lib/linalg.nu 19 additions, 12 deletionsbenchmarks/nu-lib/linalg.nu
.env.dockerfile
0 → 100644
.github/workflows/ci.yml
0 → 100644
.nu.cfg
0 → 100644
| [package] | ||
| name = "komodo" | ||
| version = "1.0.0" | ||
| version = "1.0.1" | ||
| edition = "2021" | ||
| description = "Komodo: cryptographically-proven erasure coding for distributed systems" | ||
| repository = "https://gitlab.isae-supaero.fr/dragoon/komodo" | ||
| ... | ... | @@ -25,26 +25,31 @@ thiserror = "1.0.50" |
| tracing = "0.1.40" | ||
| tracing-subscriber = "0.3.17" | ||
| ark-poly-commit = { git = "https://gitlab.isae-supaero.fr/a.stevan/poly-commit", version = "0.4.0", rev = "19fc0d4", optional = true } | ||
| dragoonfri = { version = "0.1.0", optional = true} | ||
| [workspace] | ||
| members = [ | ||
| "benchmarks", | ||
| "bins/rank", | ||
| "bins/saclin", | ||
| ] | ||
| [dev-dependencies] | ||
| ark-bls12-381 = "0.4.0" | ||
| clap = { version = "4.5.17", features = ["derive"] } | ||
| itertools = "0.13.0" | ||
| rand = "0.8.5" | ||
| dragoonfri-test-utils = "0.1.0" | ||
| hex = "0.4.3" | ||
| [features] | ||
| kzg = ["dep:ark-poly-commit"] | ||
| aplonk = ["dep:ark-poly-commit"] | ||
| fri = ["dep:dragoonfri"] | ||
| fs = [] | ||
| [package.metadata.docs.rs] | ||
| features = ["kzg", "aplonk"] | ||
| rustdoc-args = [ "--html-in-header", "katex.html" ] | ||
| [[example]] | ||
| name = "kzg" | ||
| ... | ... | @@ -53,3 +58,11 @@ required-features = ["kzg"] |
| [[example]] | ||
| name = "aplonk" | ||
| required-features = ["aplonk"] | ||
| [[example]] | ||
| name = "fri" | ||
| required-features = ["fri"] | ||
| [[example]] | ||
| name = "fec" | ||
| required-features = ["fri"] |
Makefile
deleted
100644 → 0
assets/128_4096_3.bin
0 → 100644
File added
assets/128_4_3.bin
0 → 100644
File added
benchmarks/nu-lib/fri/mod.nu
0 → 100644
benchmarks/nu-lib/fri/plot.nu
0 → 100644
benchmarks/nu-lib/fri/run.nu
0 → 100644