## changelog
- _semi\_avid_, _kzg_ and _aplonk_ examples have been added
- the `fs` module has been hidden behind an `fs` feature
- the `conversions` module has been properly hidden behind the `test` config feature
- the documentation has been completed
- some error messages have been improved

> **Note**
>
> the documentation of aPlonK has been left as-is for now

`algebra`, `field` and `linalg` were doing extremely similar things before...

this MR merges them into a single module `algebra`
- old `algebra` and `field` are at the root of the new `algebra`
- old `linalg` is now `algebra::linalg`

all references to these have been fixed in the rest of the codebase and the features have been tuned to work fine.

## changelog
- `src/main.rs` has been moved to a new crate: `bins/saclin` which stands for **S**emi-**A**VID **CLI** in **N**ushell
- dependencies of `komodo` have been fixed
- Nushell and Rust tests have been split in the Makefile: by default, only Rust tests will run locally and Nushell tests and examples can be run manually if desired. The CI will still run everything.
- the README has been updated
- test images have been moved to `assets/`
- the majority of the old `./nu-utils/` module have been moved to internals of `./benchmarks/` and imports have been fixed
- `cargo.nu` has been moved to `./bins/` and a new `./bins/README.md` mentions it
- `./bins/saclin/` has been created and should be a self-contained Rust crate + Nushell module

see [`dragoon/nc-diversity`](https://gitlab.isae-supaero.fr/dragoon/nc-diversity)

they have been moved to [dragoon/binaries](https://gitlab.isae-supaero.fr/dragoon/binaries).

this is the trivial case where it is always possible to decode the original data, so there is no need to have it here.

this MR
- moves the Nushell modules from `bins/inbreeding/` to `bins/inbreeding/src/.nushell/`
- creates a `NUSHELL` constant in `consts.nu` to allow the following more robust construct
```bash
use consts.nu
use $consts.NUSHELL ...
```
- updates the README

this simply makes "parsing" in the inbreeding modules more robust and centralized.

- early return when format is bad twice
- refactor

wait for
- !134 

## description
assuming !134 has been merged, this MR allows to run something like
```bash
inbreeding list | input list --fuzzy | inbreeding load $in | inbreeding plot
```
which will fuzzy find the available experiments and then plot it after load 

this can be used by `plot` without passing extra arguments, i.e. the pipeline becomes
```bash
use bins/inbreeding

let experiment = "..."
```
```bash
inbreeding load $experiment | inbreeding plot
```
instead of having to know the value of `k` and do
```bash
inbreeding load $experiment | inbreeding plot --options { k: $k }
```

- addresses #9 
- needed !133 to work

this MR simply adds `+ Clone` to `rng` and removes the `mut` from the `rng` of the `measure_inbreeding` function.

running the same snippet of code from #9 yields the following two images with a schedule of $1$ and $5$ respectively

![000_1.1](/uploads/ace74e299c6d4374df713ab5004f3c1d/000_1.1.png)
![000_1.2](/uploads/c784f5b62c41004971fd3048f7b5babc/000_1.2.png)

we see that all measurements on $t$ where $t = 0 \mod 5$ are the same in both images 

related to
- #9

in order for the measurements not to influence the experiment, the seeds passed to the runs need to not include the _measurement schedule_ parameters!

EDIT: in the end, it's more than that, we want to only include things related to the environment in its hash, nothing related to the measurements, i.e. we want to either
- exclude `strategies`, `nb_scenarii`, `measurement_schedule`, `measurement_schedule_start`, `nb_measurements` and `max_t`
- include `nb_bytes`, `k`, `n` and `environment`

## output sample
let's say the current seed is `"b239e48345ac457b492cc164f58c010d07292c88e4791e607d91796baec7f334"` and the experiment has ID `fixed:0-single:3-5-50-10240`, and that we are watching both the creation of the first experiment and a few runs, e.g.
- `002d8de28913efbf7dbd111b817ae901fee2d47882ba7aa76d293c2d95d9652c`
- `015672b37c9cf1a6b475937987294f9a503a922ffbcfdfc5d18ef839fac91b8c`
- `080a51a17ac43fcbdf08f77f3bc993983fef589bd9672f04382af4b16dd09b13`

then the output would look like
```
b239e48  fixed:0-single:3-5-50-10240            at 2024-05-30 15:24:12
b239e48  fixed:0-single:3-5-50-10240  002d8de   at 2024-05-30 15:24:14
b239e48  fixed:0-single:3-5-50-10240  015672b   at 2024-05-30 15:24:16
b239e48  fixed:0-single:3-5-50-10240  080a51a   at 2024-05-30 15:24:18
```

we are switching
- _naive recoding_ to _$(k, 1)$-re-encoding_
- _true recoding_ to _$k$-recoding_

this is to have a complete identifier for each run, which does not depend at all on the time at which the experiment runs.

related to
- !122

## description
!122 introduced a `draw_unique_indices` function which uses a `HashSet` to accumulate unique indices in the range `0..<len`.
however, a `HashSet` does not preserve the order of insertion when iterating over the elements of the set... which results in apparent randomness, even though the RNG seed is the same  

this MR switches back to using `shuffle` which used to work, even though a bit less performant    
it's basically a revert of !122, while keeping the refactoring into `random.rs`.

## measuring the performance
i did run the same timing experiment from !122 but with `main` on `bb55005f` and the MR on `fix-shuffle`

| env     | main                   | mr                      | improvement         |
| ------- | ---------------------- | ----------------------- | ------------------- |
| fixed:0 | 6sec 244ms 238µs 45ns | 8sec 734ms 929µs 328ns | -39.88783363238997  |
| fixed:1 | 639ms 720µs 39ns      | 731ms 360µs 261ns      | -14.325051024390373 |

we loose a bit

- add optional `$.help` to argument `err` of `error throw`
- parse `prng_seed: [u8; 32]` in `rng` and `inbreeding`
- compute the "_local_" seed by hashing the "_global_" seed, the strategy and the iteration index
- pass `--prng-seed: string`, a 64-char long seed to `inbreeding run`

because the "_environment_" can contain an experiment separator, e.g. `random-fixed`, more powerful `parse` patterns needs to be used.

* 7b44a886 refactor constants from `inbreeding load`
* 64e446f7 move `remove-cache-prefix` to new `path.nu`
* 822b03fd add `inbreeding inspect`

in order to add more information in the experiment names, e.g. $k$, $n$ and $\#\text{bytes}$, this MR changes the cache files format from
```
{seed}/{timestamp}/{env}/{strategy}/...
```
to
```
{seed}/{timestamp}-{env}-{strategy}-{k}-{n}-{nb_bytes}/...
```

this MR turns `./.nushell/` into a directory module by
- adding `mod.nu`
- exporting all the modules

all uses of `.nushell/` have been fixed to not mention `.nu` internal modules anymore.

>  **Note**  
> the `.nushell venv` module has been removed because, when the `$venv.VENV` activation script is not there, Nushell can't parse the whole `.nushell` module, which is very annoying to have to rely of the state of the external filesystem to be able to simply parse a module...

in `./bins/inbreeding/`, this MR does
- refactor the "list item drawing" from `environment.rs` and `strategy.rs` into the `draw_unique_elements` function of new `random.rs` module
- use a `HashSet` to draw unique indices in the slice of "things" to draw from and then extracts the items corresponding to these indices

## results
```bash
use ./bins/inbreeding
use std bench

const PRNG_SEED = 0
const OPTS = {
    nb_bytes: (10 * 1_024),
    k: 10,
    n: 20,
    nb_measurements: 100,
    nb_scenarii: 10,
    measurement_schedule: 1,
    measurement_schedule_start: 2_000,
    max_t: 2_000,
    strategies: [ "single:5" ],
    environment: null,
}

def run [rev: string] {
    git co $rev

    inbreeding build

    let a = bench --rounds 5 {
        inbreeding run --options ($OPTS | update environment "fixed:0") --prng-seed $PRNG_SEED
    }
    let b = bench --rounds 5 {
        inbreeding run --options ($OPTS | update environment "fixed:1") --prng-seed $PRNG_SEED
    }

    {
        0: $a,
        1: $b,
    }
}

let main = run a29b511d
let mr = run fix-shuffle
```
```bash
let table = [
    [env, main, mr, improvement];

    ["fixed:0", $main."0".mean, $mr."0".mean, (($main."0".mean - $mr."0".mean) / $main."0".mean * 100)],
    ["fixed:1", $main."1".mean, $mr."1".mean, (($main."1".mean - $mr."1".mean) / $main."1".mean * 100)],
]

$table | to md --pretty
```

| env     | main                    | mr                      | improvement        |
| ------- | ----------------------- | ----------------------- | ------------------ |
| fixed:0 | 8sec 504ms 794µs 784ns | 6sec 353ms 206µs 645ns | 25.298530930431912 |
| fixed:1 | 727ms 648µs 292ns      | 639ms 443µs 782ns      | 12.12186037811795  |

the improvement is quite nice, even though not huge, but the code is cleaner anyways 

this adds `$.diversity.e` to the output of `inbreeding load` and input of `inbreeding plot` to show errors bars in the final plot.

these can be discarded by running something similar to
```bash
inbreeding load $experiment
    | reject diversity.e
    | inbreeding plot --options { k: $OPTS.k }
```

plotting "inbreeding" results is now done in two steps
- `load` the data from raw experiment files with `inbreeding load`
- `plot` the results by piping this data into `inbreeding plot`

## new structure for the repository
- benchmarks are in `./benchmarks/` and can be run with either `cargo run --package benchmarks --bin <bench>` or the commands in `./benchmarks/README.md`
```
├── Cargo.toml
├── README.md
└── src
    └── bin
        ├── commit.rs
        ├── fec.rs
        ├── linalg.rs
        ├── operations
        │   ├── curve_group.rs
        │   └── field.rs
        ├── recoding.rs
        ├── setup.rs
        └── setup_size.rs
```

- examples are now in `./bins/` as standalone binaries and can be run either with `cargo run --package <pkg>` or with the help of the `cargo bin` command from `.nushell/cargo.nu`
```
├── curves
│   ├── Cargo.toml
│   ├── README.md
│   └── src
│       └── main.rs
├── inbreeding
│   ├── build.nu
│   ├── Cargo.toml
│   ├── consts.nu
│   ├── mod.nu
│   ├── plot.nu
│   ├── README.md
│   ├── run.nu
│   └── src
│       ├── environment.rs
│       ├── main.rs
│       └── strategy.rs
├── rank
│   ├── Cargo.toml
│   └── src
│       └── main.rs
└── rng
    ├── Cargo.toml
    └── src
        └── main.rs
```

- Nushell modules are now located in `./.nushell/`

## changelog
apart from the changes to the general structure of the repo:
- `binary.nu` -> `.nushell/binary.nu`
- new `cargo bin` command from `.nushell/cargo.nu`
- `error throw` is now defined in `.nushell/error.nu`
- main TOML has been greatly simplified because the dependencies of "examples" have been moved to the associated crates
- the rest is basically the same but in the new structure