# `mix test.baseline`
[🔗](https://github.com/riddler/statifier-ex/blob/v2.10.0/lib/mix/tasks/test.baseline.ex#L1)

Finds conformance tests that pass but are not yet in the regression
registry, and adds them once they are verified.

## Usage

    # Report what could be ratcheted in, without writing anything
    mix test.baseline

    # Report, then add everything that passed
    mix test.baseline --add

    # Verify specific files and add them
    mix test.baseline add test/scion_tests/basic/basic0_test.exs

    # An authored case is named by its JSON file
    mix test.baseline add conformance/cases/send/registered_immediate.json

## Options

  * `--add` - ratchet in every newly passing test the scan found
  * `--only` - restrict the scan to one suite, `scion`, `w3c` or `statifier`
  * `--registry` - registry to update, defaults to `test/passing_tests.json`

Both forms run each candidate on its own before writing anything, so a test
can only enter the registry by passing. `add` is all-or-nothing: if any named
file fails, the registry is left untouched.

The `statifier` suite is the cases this repository authors under
`conformance/cases/` (ADR-0070 decision 5). They have no generated test
module, so a candidate there is the case's JSON file, and it is run through
`Mix.Statifier.Corpus.Runner`, as `mix statifier.corpus` runs it, instead
of `mix test`.

The ratchet only moves forward. Nothing here removes an entry - a test that
used to pass and now does not is a regression to fix, not a line to delete.

A scan also prints a per-corpus coverage block: for each suite the scan
covered, `passing/total (percent%)` against the suite's emitted corpus
files. The numerator is every registry-tracked file plus whatever this scan
found newly passing - tracked files this invocation skipped re-running are
still counted, because `mix test.regression` is what guarantees them. `add`
prints no such block: it verifies named files and never scans, so it has no
denominator in hand.

# `execute`

```elixir
@spec execute(argv :: [String.t()], opts :: keyword()) :: :ok | {:error, String.t()}
```

Runs the task and reports the outcome instead of halting.

`opts[:runner]` replaces the `mix test` shell-out with a function of the
argument list returning an exit status, `opts[:case_runner]` replaces
running the authored cases with a function of their paths returning what
`Mix.Statifier.Corpus.Runner.run_paths/2` returns, `opts[:root]` moves the
corpus scan to a fixture tree, and `opts[:today]` fixes the date stamped
into the registry. All four exist so the tests can drive this without
spawning a nested `mix test` or starting the session runtime.

---

*Consult [api-reference.md](api-reference.md) for complete listing*