this comes from https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idcontainer.

This should close #19.

## changelog
- triggers pipeline in the GitHub mirror on each push
- add `./scripts/get-mirror-pipelines.nu`

## the pipelines
> ❗ **Important**
>
> i just had to add `run-mirror-pipeline-everytime` to the list of protected branches and trigger the push with an empty commit and it runs
>
> and then a single call to `nu scripts/get-mirror-pipelines.nu "run-mirror-pipeline-everytime"` gives the table below

| id                                                                             | head_sha                                 | status    | conclusion | run_started_at      |
| ------------------------------------------------------------------------------ | ---------------------------------------- | --------- | ---------- | ------------------- |
| [`14238533729`](https://github.com/dragoon-rs/komodo/actions/runs/14238533729) | 2f32ee59 | completed | success    | 2025-04-03T08:32:31 |
| [`14238724666`](https://github.com/dragoon-rs/komodo/actions/runs/14238724666) | e7587675 | completed | success    | 2025-04-03T08:42:19 |
| [`14240494187`](https://github.com/dragoon-rs/komodo/actions/runs/14240494187) | 3c223dd6 | completed | success    | 2025-04-03T10:13:02 |
| [`14240642763`](https://github.com/dragoon-rs/komodo/actions/runs/14240642763) | cbc1f7f9 | completed | success    | 2025-04-03T10:21:04 |
| [`14240848529`](https://github.com/dragoon-rs/komodo/actions/runs/14240848529) | 7b34fc8a | completed | success    | 2025-04-03T10:32:06 |

this removes the `check-nushell-files.nu` script i left in !200.

that's an attempt at using Rust to build itself.

this is using [`gitlab.isae-supaero.fr:a.stevan/nob.rs@e4b03cdd`](a.stevan/nob.rs@e4b03cdd).

> 💡 **Note**
>
> to be honest, this is not a 100% replacement of the `Makefile`...
>
> `make.rs` does a lot more and provides a full CLI with easy-to-use options, e.g. instead of `make fmt` and `make fmt-check`, we now have `./make.rs fmt` and `./make.rs fmt --check`
>
> (see the API below)

## the API
```
Usage: make [OPTIONS] [COMMAND]

Commands:
  fmt      Formats the code
  check    Checks the code
  clippy   Runs Clippy
  test     Runs the tests
  version  Shows the version of all the tools used,
  doc      Builds the documentation
  help     Print this message or the help of the given subcommand(s)

Options:
  -h, --help      Print help
  -V, --version   Print version
```
```
Usage: make fmt [OPTIONS]

Options:
  -c, --check  Only checks instead of really formatting
```
```
Usage: make check
```
```
Usage: make clippy
```
```
Usage: make test [OPTIONS]

Options:
  -v, --verbose   Be extra verbose with the output of the tests
  -e, --examples  Run the examples instead of regular tests
```
```
Usage: make version
```
```
Usage: make doc [OPTIONS]

Options:
  -o, --open      Open the documentation in the browser
  -p, --private   Document private items
  -f, --features  Document all features
```

## running the pipeline in the GitHub mirror
```bash
const GH_API_OPTIONS = [
    -H "Accept: application/vnd.github+json"
    -H "X-GitHub-Api-Version: 2022-11-28"
]
let res = gh api ...$GH_API_OPTIONS /repos/dragoon-rs/komodo/actions/runs | from json
```
```bash
let runs = $res.workflow_runs
    | where head_branch == "use-nob-to-build"
    | select id head_sha status conclusion run_started_at
    | into datetime run_started_at
    | sort-by run_started_at
```
```bash
$runs
    | update id { $"[`($in)`]\(https://github.com/($GITHUB_MIRROR)/actions/runs/($in)\)" }
    | update run_started_at { format date "%Y-%m-%dT%H:%M:%S" }
    | to md --pretty
```
| id                                                                             | head_sha                                 | status    | conclusion | run_started_at      |
| ------------------------------------------------------------------------------ | ---------------------------------------- | --------- | ---------- | ------------------- |
| [`14237650542`](https://github.com/dragoon-rs/komodo/actions/runs/14237650542) | d67f1cfd | completed | success    | 2025-04-03T07:44:14 |
| [`14237741570`](https://github.com/dragoon-rs/komodo/actions/runs/14237741570) | 9ef598a1 | completed | success    | 2025-04-03T07:49:40 |
| [`14238086977`](https://github.com/dragoon-rs/komodo/actions/runs/14238086977) | 0a79edf3 | completed | success    | 2025-04-03T08:09:13 |
| [`14238175174`](https://github.com/dragoon-rs/komodo/actions/runs/14238175174) | a84b2b12 | completed | success    | 2025-04-03T08:13:52 |
| [`14239395984`](https://github.com/dragoon-rs/komodo/actions/runs/14239395984) | 8594c9bf | completed | success    | 2025-04-03T09:16:00 |

This is to complete the `check-mirror.nu` script with information about the CI runs.

The current output of
```bash
nu scripts/check-mirror.nu ...[
    https://gitlab.isae-supaero.fr/dragoon/komodo
    https://github.com/dragoon-rs/komodo
    main
]
```
is
```
[INF] adding remotes
[INF] fetching
[INF] cleaning
[ OK] mirror is up to date
[INF] pulling mirror runs
#─┬────id─────┬────────────────head_sha────────────────┬─status──┬conclusion┬run_started_at
0 │11950217125│de4266c0│completed│success   │4 months ago
1 │11950604920│202d8bcc│completed│success   │4 months ago
2 │11950873603│702cd5eb│completed│success   │4 months ago
3 │12028360471│57a96c02│completed│success   │4 months ago
4 │12028432364│07602875│completed│success   │4 months ago
5 │12028781541│f2a76fbb│completed│success   │4 months ago
6 │12631165433│11a5b59d│completed│success   │2 months ago
7 │13008608156│cc412625│completed│success   │2 months ago
8 │13033206333│5511404f│completed│success   │2 months ago
9 │13113244065│ab0d9a8b│completed│success   │2 months ago
10│13679188387│3c8afac4│completed│success   │3 weeks ago
11│14191103891│b5381fda│completed│success   │a day ago
12│14214853530│b7e2d99d│completed│success   │an hour ago
13│14215431531│0d7a5cd6│completed│success   │32 minutes ago
──┴───────────┴────────────────────────────────────────┴─────────┴──────────┴──────────────
```

this should close #17.

SACLIN has been moved to [`gitlab.isae-supaero.fr:dragoon/komodo.nu`](https://gitlab.isae-supaero.fr/dragoon/komodo.nu).

## usage
```bash
nu scripts/check-mirror.nu ...[
    https://gitlab.isae-supaero.fr/dragoon/komodo
    https://github.com/dragoon-rs/komodo
    main
]
```
will
- check the Nushell version and throw a warning if mismatch
- add temporary remotes with random names
- fetch the temporary remotes
- get the revision of the `$branch` for both temporary remotes
- clean the temporary remotes
- show a message if the two revisions are not the same and how many commits they are different

> **Note**
>
> this does not conflict with #17

HEME Clement authored 1 month ago and STEVAN Antoine committed 1 month ago

Added a html header files, an added metadata line in Cargo.toml and some formatting changes in the doc to have Latex formulas in docs.rs. It works locally, should work online

this is to avoid having to change the toolchain in the CI everytime.

`rust-toolchain.toml` and `.gitlab-ci.yml` will have to be updated in pairs when bumping / changing the Rust toolchain.

this is an attempt at making benchmarks easier to work with
- `benchmarks run` will read benchmarks from NUON data and run them
- `benchmarks plot` will plot benchmark results
- the output format will be something 
  - a directory whose name is the hash of the CPU spec and the Komodo commit hash
  - contains `cpu.json` with the CPU info
  - contains `komodo.txt` with the Komodo commit hash
  - contains NDJSON result files

> 💡 **Note**
>
> results will typically be uploaded to https://gitlab.isae-supaero.fr/dragoon/komodo-benchmark-results

> 💡 **Note**
>
> this MR goes alongside the [`komodo-benchmark-results@restart`](https://gitlab.isae-supaero.fr/dragoon/komodo-benchmark-results/-/compare/main...restart) branch

## changelog
- bump Nushell to 0.101.0
  - parallel `$in` => remove useless `let input = $in` when possible, e.g. still required when using the function's `$in` in a `for` loop
  - `group-by` changed => `group-by x --to-table` will now produce a table with columns `x` and `items` instead of `group` and `items` as in 0.95.0
- add link to results repo
- the Nushell benchmarks lib
  - rename `--force` to `--no-confirm (-y)`
  - add `--append` 
  - reject columns that GPLT will complain about, e.g. `$.points.k` for the FRI plots
  - add `--save` to the FRI plot
- move the "field" and "curve group" benchmarks from `benchmarks/src/bin/operations/` to `benchmarks/src/bin/`
- remove `benchmarks/params/fri.nu` because it's been uniformized with the other methods
- rewrite the README
- add main function to `benchmarks/` that runs the benchmarks from a NUON record specification
- simplify the output of FRI run

## TODO
- [x] fix "_atomic operations_" (done in 4f69a1d6)
- [x] check that _plotting_ still works

## images

### Field
![complex_curve_group_operations](/uploads/57b36926cce041cf405a9b44f190b8b8/complex_curve_group_operations.png)
![complex_field_operations](/uploads/0747c85dbaff8980561aa9d922fcd5e7/complex_field_operations.png)
![simple_curve_group_operations](/uploads/974cf70fed68f8d8ac898d54be3f27be/simple_curve_group_operations.png)
![simple_field_operations](/uploads/a3d4d0dcdeb35d4c434eaa38fb51e7b5/simple_field_operations.png)

### Linear algebra
![linalg-inverse](/uploads/bc290ffa39459ce0f9bbd393b50b7b98/linalg-inverse.png)
![linalg-mul](/uploads/96d8c2a63ed48d6a0d3508b4a948153b/linalg-mul.png)
![linalg-transpose](/uploads/128e35eca91497d8aadb0130c05aeee3/linalg-transpose.png)

### FEC
![encoding](/uploads/405c4d3ef9ec5135ebdd7ce2e6c96bfe/encoding.png)
![decoding](/uploads/d793234d44e9fc6f34f0c2a9372863cd/decoding.png)
![recoding](/uploads/413021de997c86d45b1287fcfe7804c7/recoding.png)
![combined](/uploads/6d1c3ae6d3bf5547434ca29ae80b5536/combined.png)
![ratio](/uploads/6cebd7a0bcef57d1b1256bc3941c1b0a/ratio.png)

### ZK
![setup](/uploads/1feb169452aa3274dc924edf772c9a5b/setup.png)
![commit](/uploads/0a2775c2116ca7d3fa7b956b934d1565/commit.png)

### FRI
![commits_single](/uploads/4602725e551a025d42815183a61d11b2/commits_single.png)
![commits_single_normalized](/uploads/2ffebea940af1cbbe9bd64499343e0e9/commits_single_normalized.png)
![end_to_end](/uploads/46917abd2f5976dfa6d3039cc0ab2c0e/end_to_end.png)
![evaluating](/uploads/cf4dc496cd5144615bf5f9b06d27dccd/evaluating.png)
![proofs](/uploads/b0828bfa62c2226c8d63c9ecd387049f/proofs.png)
![proofs_normalized](/uploads/8fbf0d713884e2e147c62313a001f379/proofs_normalized.png)
![proving](/uploads/161539dc412330be1878cdda82c3d966/proving.png)
![verifying_single](/uploads/2df7e777481d7789478386f8e83e0783/verifying_single.png)

the idea is to make the installation of Nushell easier to maintain, especially regarding versions, currently pinned to `0.95.0`

> successful run on GitHub: [13032726635](https://github.com/dragoon-rs/komodo/actions/runs/13032726635)

this is also to allow easier testing locally with the same Nushell version as in the CI, e.g.
```bash
# install in `~/.local/bin/` and have multiple versions
make install-nu

hash=$(/tmp/nu --no-config-file --commands 'version | get commit_hash')
nu_bin=$"$HOME/.local/bin/nu-$hash"
cp /tmp/nu $nu_bin

make NU=$nu_bin show test
```
or
```bash
# install in the repo and overwrite each time
make NU_DEST=. install-nu
make NU=./nu show test
```

# changelog
- Makefile
  - split the global `.PHONY` rule into _atomic_ rules next to each _phony_ rule
  - define `NU` and `NU_FLAGS` to allow changing which and how Nushell runs
  - define `NU_ARCH`, `NU_VERSION`, `NU_BUILD` and `NU_DEST` for Nushell installation
  - tweak the output of `make show` a bit
  - add `print-%` rules to print `Makefile` variables, e.g. `make print-NU_FLAGS` would print `--no-config-file`
  - add `make install-nu` to replace the ones from the CIs
- GitLab CI
  - use `make install-nu`
  - export `PATH` with `make print-NU_DEST`
- GitHub CI 
  - use `make install-nu` instead of [github.com:hustcer/setup-nu](https://github.com/hustcer/setup-nu)
  - export `PATH` with `make print-NU_DEST` and `GITHUB_ENV`

mainly fixes a few typos and improves `algebra` and `semi_avid` tests

## changelog
- fix some typos and notes in the documentation
- test more cases for `algebra::split_data_into_field_elements` and `algebra::merge_elements_into_bytes`, more data lengths and more modulus respectively
- remove a useless `::<Vec<_>>` on a `collect` in `fec`
- `semi_avid::tests`
  - refactor `bls12-381` into a constant
  - write an "_attack_" function to alter a particular block, `attack<F, G>(block: Block<F, G>, c: usize, base: u128, pow: u64) -> Block<F, G>`
  - pass a list of attacks, i.e. `attacks: Vec<(usize, usize, u128, u64)>`, to `verify_with_errors_template`, use the same as the previous hardcoded one
  - pass a list of "_recodings_", i.e. `recodings: Vec<Vec<usize>>`, to `verify_recoding_template` and assert the number of blocks, e.g. `vec![vec![2, 3], vec![3, 5]]` means that recoding 2 and 3 together should verify, and same with 3 and 5, and also that there must be at least 6 blocks
  - pass a list of "_recodings_", i.e. `recodings: Vec<(Vec<Vec<usize>>, bool)>`, to `end_to_end_with_recoding_template` and assert the number of blocks and the number of source shards, e.g. `vec![(vec![vec![0, 1], vec![2], vec![3]], true)]` means that trying to decode with blocks 2, 3 and a recoded block from 0 and 1 should work, and also that there must be at least 4 blocks and at most 3 source shards
  - pass `k` and `n` to `run_template`

this should avoid the following error when trying to rust the LSP in a toolchain which does not have the `rust-analyzer` component and without running `rustup component add rust-analyzer` manually:
```
error: Unknown binary 'rust-analyzer' in official toolchain 'stable-x86_64-unknown-linux-gnu'.
```

in order to help catch issues with the various Nushell scripts and modules in the source base, i propose to add a CI script that will check them all.

below is an example error, e.g. when introducing a syntax error in `.env.nu`:
```
Error:   × Failed to parse content:
  │     file: .env.nu
  │     err:  Expected keyword.
```

## changelog
- add more snippets and instructions
- add missing imports
- update the "atomic operations" section
- add a table of contents

this replaces the "_nothing to do_" messages from Nushell commands with more explicite errors.

DETCHART Jonathan authored 5 months ago and STEVAN Antoine committed 5 months ago

This MR adds an option to perform erasure coding using FFT rather than using a matrix. It also adds the field FP128 in the list of curves

Note that there is a redundant function `random_loss` into benchmarks/bin/fec.rs and examples/fec.rs