- Feb 03, 2025
-
-
STEVAN Antoine authoredthis is an attempt at making benchmarks easier to work with - `benchmarks run` will read benchmarks from NUON data and run them - `benchmarks plot` will plot benchmark results - the output format will be something - a directory whose name is the hash of the CPU spec and the Komodo commit hash - contains `cpu.json` with the CPU info - contains `komodo.txt` with the Komodo commit hash - contains NDJSON result files >
💡 **Note** > > results will typically be uploaded to https://gitlab.isae-supaero.fr/dragoon/komodo-benchmark-results >💡 **Note** > > this MR goes alongside the [`komodo-benchmark-results@restart`](https://gitlab.isae-supaero.fr/dragoon/komodo-benchmark-results/-/compare/main...restart) branch ## changelog - bump Nushell to 0.101.0 - parallel `$in` => remove useless `let input = $in` when possible, e.g. still required when using the function's `$in` in a `for` loop - `group-by` changed => `group-by x --to-table` will now produce a table with columns `x` and `items` instead of `group` and `items` as in 0.95.0 - add link to results repo - the Nushell benchmarks lib - rename `--force` to `--no-confirm (-y)` - add `--append` - reject columns that GPLT will complain about, e.g. `$.points.k` for the FRI plots - add `--save` to the FRI plot - move the "field" and "curve group" benchmarks from `benchmarks/src/bin/operations/` to `benchmarks/src/bin/` - remove `benchmarks/params/fri.nu` because it's been uniformized with the other methods - rewrite the README - add main function to `benchmarks/` that runs the benchmarks from a NUON record specification - simplify the output of FRI run ## TODO - [x] fix "_atomic operations_" (done in 4f69a1d6) - [x] check that _plotting_ still works ## images ### Field     ### Linear algebra    ### FEC      ### ZK   ### FRI        
-
- Nov 26, 2024
-
-
STEVAN Antoine authored
## changelog - add more snippets and instructions - add missing imports - update the "atomic operations" section - add a table of contents
-
- Nov 21, 2024
-
-
STEVAN Antoine authored
FRI protocol from [`dragoon/fri`](https://gitlab.isae-supaero.fr/dragoon/fri) ## changelog - add binary assets to be used as inputs - add `fri` and `fri_test_utils` as local dependencies until [`dragoon/fri`](https://gitlab.isae-supaero.fr/dragoon/fri) becomes public - add `fri` feature and module (see section below for the public definitions) - fix bug in 32bd6566 - bump Rust in e7a2c244 - add a versatile example - add Nushell pipeline to run benchmarks and plot results - add some tests ## `fri` module API ```rust struct Block<F: PrimeField, H: Hasher> ``` ```rust fn evaluate<F: PrimeField>(bytes: &[u8], k: usize, n: usize) -> Vec<Vec<F>> ``` ```rust fn encode<F: PrimeField>( bytes: &[u8], evaluations: Vec<Vec<F>>, k: usize, ) -> Vec<fec::Shard<F>> ``` ```rust fn prove<const N: usize, F: PrimeField, H: Hasher, P>( evaluations: Vec<Vec<F>>, shards: Vec<fec::Shard<F>>, blowup_factor: usize, remainder_plus_one: usize, nb_queries: usize, ) -> Result<Vec<Block<F, H>>, KomodoError> where P: DenseUVPolynomial<F>, for<'a, 'b> &'a P: Div<&'b P, Output = P>, <H as rs_merkle::Hasher>::Hash: AsRef<[u8]>, ``` ```rust fn verify<const N: usize, F: PrimeField, H: Hasher, P>( block: Block<F, H>, domain_size: usize, nb_queries: usize, ) -> Result<(), KomodoError> where P: DenseUVPolynomial<F>, for<'a, 'b> &'a P: Div<&'b P, Output = P>, <H as rs_merkle::Hasher>::Hash: AsRef<[u8]>, ``` ```rust fn decode<F: PrimeField, H: Hasher>(blocks: Vec<Block<F, H>>, n: usize) -> Vec<u8> ``` ## results ### times      ### sizes    
-
- Jul 12, 2024
-
-
STEVAN Antoine authored
this MR is two-fold - it restructures the two main Nushell modules so that they are easier to read and use - it improves the "run" and "plot" modules for the benchmarks ## changelog - `.nushell/` is now renamed to `nu-utils/` - `benchmarks/` is now a valid Nushell module which exports a bunch of modules - `benchmarks linalg`: measure and plot linear algebra operations - `benchmarks setup`: measure and plot trusted setup building - `benchmarks commit`: measure and plot crafting commitments - `benchmarks recoding`: measure and plot the recoding of shards - `benchmarks fec`: measure and plot FEC operations, such as encoding and recoding, and allow combining these results with the pure recoding ones - the submodules of `benchmarks` typically have a `run` and a `plot` command, whith the exception of `benchmarks fec` which has a `run` module and multiple "plot" commands in `benchmarks fec plot` - the "run" commands will create a random temp file by default and ask for confirmation otherwise if the output file already exists, unless `--force` is used - snippetds in `benchmarks/README.md` have been updated
-
STEVAN Antoine authored
## results     
-
- Jul 05, 2024
-
-
STEVAN Antoine authored
this adds a new custom field to the `benchmarks::fields` new module and fixes the "linalg" commands to run the benchmarks ## results >
💡 **Note** > > using the data from !156   
-
- May 28, 2024
-
-
STEVAN Antoine authored
## new structure for the repository - benchmarks are in `./benchmarks/` and can be run with either `cargo run --package benchmarks --bin <bench>` or the commands in `./benchmarks/README.md` ``` ├── Cargo.toml ├── README.md └── src └── bin ├── commit.rs ├── fec.rs ├── linalg.rs ├── operations │ ├── curve_group.rs │ └── field.rs ├── recoding.rs ├── setup.rs └── setup_size.rs ``` - examples are now in `./bins/` as standalone binaries and can be run either with `cargo run --package <pkg>` or with the help of the `cargo bin` command from `.nushell/cargo.nu` ``` ├── curves │ ├── Cargo.toml │ ├── README.md │ └── src │ └── main.rs ├── inbreeding │ ├── build.nu │ ├── Cargo.toml │ ├── consts.nu │ ├── mod.nu │ ├── plot.nu │ ├── README.md │ ├── run.nu │ └── src │ ├── environment.rs │ ├── main.rs │ └── strategy.rs ├── rank │ ├── Cargo.toml │ └── src │ └── main.rs └── rng ├── Cargo.toml └── src └── main.rs ``` - Nushell modules are now located in `./.nushell/` ## changelog apart from the changes to the general structure of the repo: - `binary.nu` -> `.nushell/binary.nu` - new `cargo bin` command from `.nushell/cargo.nu` - `error throw` is now defined in `.nushell/error.nu` - main TOML has been greatly simplified because the dependencies of "examples" have been moved to the associated crates - the rest is basically the same but in the new structure
-
- May 23, 2024
-
-
STEVAN Antoine authored
up until now, elliptic curves have been hardcoded in the benchmarks, forcing to run them on all supported curves... this MR makes it possible to use only a subset of curves. >
💡 **Note** > when running the same commands from !104, minus the "inbreeding" ones which are not affected by this MR, the time goes from 12min 33sec to 4min 28sec🎉 ## TODO - [x] setup - [x] commit - [x] recoding - [x] fec - [ ] linalg - [ ] setup size - [ ] field operations - [ ] group operations >💡 **Note** > because all the unticked bullet points above are far from critical to the paper and do require to measure all curves, i won't change these for now -
STEVAN Antoine authored
this MR moves run and plot commands from `examples/benches/README.md` to - `scripts/setup/`: `run.nu` and `plot.nu` - `scripts/commit/`: `run.nu` and `plot.nu` - `scripts/recoding/`: `run.nu` and `plot.nu` - `scripts/fec/`: `run.nu` and `plot.nu` - `scripts/inbreeding/`: `build.nu`, `run.nu` and `plot.nu` to generate all the figures at once ```bash use scripts/setup/run.nu; seq 0 13 | each { 2 ** $in } | run --output data/setup.ndjson use ./scripts/setup/plot.nu; plot data/setup.ndjson --save ~/setup.pdf use scripts/commit/run.nu; seq 0 13 | each { 2 ** $in } | run --output data/commit.ndjson use ./scripts/commit/plot.nu; plot data/commit.ndjson --save ~/commit.pdf use scripts/recoding/run.nu; seq 0 18 | each { 512 * 2 ** $in } | run --ks [2, 4, 8, 16] --output data/recoding.ndjson use ./scripts/recoding/plot.nu; plot data/recoding.ndjson --save ~/recoding.pdf use scripts/fec/run.nu; seq 0 18 | each { 512 * 2 ** $in } | run --ks [2, 4, 8, 16] --output data/fec.ndjson use ./scripts/fec/plot.nu; plot encoding data/fec.ndjson --save ~/encoding.pdf use ./scripts/fec/plot.nu; plot decoding data/fec.ndjson --save ~/decoding.pdf use ./scripts/fec/plot.nu; plot e2e data/fec.ndjson --save ~/e2e.pdf use ./scripts/fec/plot.nu; plot combined data/fec.ndjson --recoding data/recoding.ndjson --save ~/comparison.pdf use ./scripts/fec/plot.nu; plot ratio data/fec.ndjson --recoding data/recoding.ndjson --save ~/ratio.pdf ./scripts/inbreeding/build.nu ./scripts/inbreeding/run.nu --output data/inbreeding.nuon ./scripts/inbreeding/plot.nu data/inbreeding.nuon --save ~/inbreeding.pdf ``` >💡 **Note** > this took around 27min 18sec in total on my machine with 14min 45sec for the inbreeding section only and 12min 33sec for the rest
-
- May 21, 2024
-
-
STEVAN Antoine authored
- update `benches/README.md` to use `cargo run --release --example ...` - add `build-examples` to `Makefile` to build all examples in release ### minor change add two `eprintln!` in `inbreeding.rs` to show the experiment parameters
-
STEVAN Antoine authored
- new `scripts/plot.nu` with common tools and options - better sets of parameters - better commands in `benches/README.md`
-
- May 13, 2024
-
-
STEVAN Antoine authored
this MR makes the plot a bit nicer. ## new figures       
-
- Apr 29, 2024
-
-
we want to compare - _naive recoding_: k-decoding followed by 1-encoding - _komodo recoding_: k-recoding with $k = \#\text{shards}$ # results >💡 **Note** > we see that the _naive recoding_ is around 100 times slower compared to the _komodo recoding_🎉 >💡 **Note** > the format of the labels is always `{curve} / {k}`   -
STEVAN Antoine authored
i've moved the plotting scripts to [GPLT](https://gitlab.isae-supaero.fr/a.stevan/gplt) which allows to install a single command, called `gplt`, with two subcommands - `gplt plot` which is the same as old `python scripts/plot/plot.py` - `gplt multi_bar` which is the same as old `python scripts/plot/multi_bar.py`
-
STEVAN Antoine authored
otherwise, k doesn't play any role in the "recoding" benchmark
-
- Apr 26, 2024
-
-
STEVAN Antoine authored
this MR adds - `examples/benches/bench_fec.rs` to the list of example benches - instructions on how to run the new benchmark and plot the results ## results   
-
STEVAN Antoine authored
- fix the path to the "bench" readme and remove it from the plot scripts - "BLS-12-381" to "BLS12-381" for consistency
-
STEVAN Antoine authored
this MR goes from ```rust let xs = seq 0 5 | each { 2 ** $in } | wrap x let twice = $xs | insert measurement { 2 * $in.x } | insert error { 0.1 + 0.5 * $in.x } let square = $xs | insert measurement { $in.x ** 2 } | insert error { 1 + 1.5 * $in.x } python scripts/plot/plot.py --title title --x-label x --y-label y --fullscreen ([ { group: "x ^ 2", items: $square }, { group: "2 * x", items: $twice } ] | to json) ``` to ```rust let xs = seq 0 5 | each { 2 ** $in } let twice = $xs | wrap x | insert y { 2 * $in.x } | insert e { 0.1 + 0.5 * $in.x } let square = $xs | wrap x | insert y { $in.x ** 2 } | insert e { 1 + 1.5 * $in.x } python scripts/plot/plot.py --title title --x-label x --y-label y --fullscreen ([ { name: "x ^ 2", points: $square }, { name: "2 * x", points: $twice } ] | to json) ``` updates the "bench" README and adds type annotations to the `plot.py` script. -
STEVAN Antoine authored
this MR - moves the last "recoding" benchmark to `examples/benches/` - moves the README, which is now all alone, to `examples/benches/` - adds a mention to `examples/benches/README.md` in `README.md` - some minor improvements to the bench README ## TODO - [x] find a way to plot the "recoding" results (thanks to !90)
-
- Apr 25, 2024
-
-
STEVAN Antoine authored
## changelog - use `out> *.ndjson` in the README to simplify running the benchmarks - create a `scripts/math.nu` module with `ns-to-ms` and `compute-stats` to refactor some of the most common operations - add `--fullscreen` to `plot.py` and `multi_bar.py` - add `--x-scale` and `--y-scale` to `plot.py`
-
STEVAN Antoine authored
## changelog - benchmarks - _commit_ has been removed in favor of `examples/benches/commit.rs` - _linalg_ has been migrated to `examples/benches/` as `bench_linalg` - _setup_ has been migrated to `examples/benches/` as `bench_setup` - `read-atomic-ops` command has been moved to `scripts/parse.nu` module - `scripts/plot/bench_commit.py` has been made more general and renamed to `scripts/plot/plot.py` - `scripts/plot/benches.py` has been removed because it's not required anymore => `plot.py` and `multi_bar.py` are general enough
-
STEVAN Antoine authored
this MR - bumps PLNK to 0.6.0 - update all existing code - uses the PLNK lib in `examples/benches/commit.rs` - fixes the y label of the plot in `scripts/plot/bench_commit.py`: was _ns_, should be _ms_
-
- Apr 24, 2024
-
-
STEVAN Antoine authored
i've basically refactored the whole "bench" framework that was inlined in `examples/benches/operations/field.rs` and `examples/benches/operations/curve_group.rs` into a new repo called [PLNK](https://gitlab.isae-supaero.fr/a.stevan/plnk). nothing effectively changes on the side of Komodo but now the code is much simpler here :)
-
STEVAN Antoine authored
this idea is to not use `criterion` and measure exactly what we want ## results    
-
STEVAN Antoine authored
we see that the Arkworks / Komodo versions of the same curves are basically the same. 
-
- Apr 23, 2024
-
-
STEVAN Antoine authored
as per title
-
- Apr 22, 2024
-
-
STEVAN Antoine authored
this MR improves the "_atomic_" script in `benches/README.md` to allow filtering _species_ to show in the _multibar_ plot. in addition to this, the warmup time and the number of samples of Criterion have been increased back to 3sec and 100 respectively. > **Note** > the benchmarks take 15min on my machine, i.e. by running the following two commands in Nushell > ```bash > cargo criterion --output-format verbose --message-format json --bench field_operations out> field.ndjson > cargo criterion --output-format verbose --message-format json --bench curve_group_operations out> curve.ndjson > ``` ## results   
-
STEVAN Antoine authored
this MR adds two now benchmarks: - `field_operations` in `benches/operations/field.rs` - `curve_group_operations` in `benches/operations/curve_group.rs` as well as `scripts/plot/multi_bar.py` to plot the results, see `benches/README.md` for the commands to run. ## results  
-
- Apr 12, 2024
-
-
STEVAN Antoine authored
-
- Apr 10, 2024
-
-
STEVAN Antoine authored
i ended up adding a bunch of changes to the benchmarks
😊 ## changelog * 805a2454 reduce the number of loops and the warmup time * f7ce05c3 don't serialize for real to save time * 37a2a7e2 don't try to compress with validation * 409f3e3c don't multiply degree by 1_024 * 610024a9 fix setup * 3d5e7c58 fix setup * 3d3167fb run benchmarks on BLS12-381, BN-254 and PALLAS * da2a71a1 pass name of the curve as parameter * 954fd6d3 plot commit for all curves * f980b30f plot all curves in linalg * 5e41df1d rename `labels` to `keys` in commit * 8bb64f99 filter setup by curves * 0163c8f9 plot all curves in setup * 8c91c6d8 split the setup of Komodo and the serde benchmarks * 0784f294 add a manual benchmark to measure the commit * 608a3fd1 move the "example benches" to `examples/benches/` * 10f9a37c add a script to plot results from `bench_commit` * 6d512fa6 move plot script from `benches/` to `scripts/plot/` * a4e6ffbc measure VESTA
-
- Apr 02, 2024
-
-
STEVAN Antoine authored
as per title ## changelog - add a `setup.rs` benchmark which measures - the creation of a random setup - the serialization of a setup - the deserialization of a setup - refactor `plot.py` a bit to - use `argparse` - take `--bench` to plot either _linalg_ or _setup_ results - write a complete `plot_setup` function - add a bit of documentation here and there ## example results 
-
- Mar 26, 2024
-
-
STEVAN Antoine authored
this MR - adds `criterion` as a dependency - creates a `linalg.rs` benchmark file - makes the following function `pub`lic - `Matrix::transpose` - `Matrix::invert` - `Matrix::mul` - creates a new `benches/` directory containing - a README with commands - a `plot.py` file to plot results - a `linalg.rs` file with the benchmarks ## example results 
-
