# Replicate harness

`scripts/replicate_study.py` runs K replicates of `scripts/run_editorial_prequential_study.py` with the same
arguments and summarises the run-to-run spread, so a later change can be compared with a baseline.

    python scripts/replicate_study.py --base-args-file study-args.txt --replicates 5 \
        --out-dir runs/baseline --parallel 2 \
        --max-total-optimizer-calls 300 --max-total-decision-calls 40000

`--dry-run` prints the exact commands and runs nothing.

- `--base-args-file`: the study's own arguments (shell-style text with `#` comments, or a JSON list of strings),
  without `--output` and `--seed`. Include `--confirm-live` there if the replicates should spend; the harness adds nothing.
- `--seeds a,b,c` (verbatim, sets K) or `--replicates K` with `--base-seed S` giving seeds `S:rep-1` ... `S:rep-K`
  (default base `editorial-prequential-v1`). Each replicate writes to `DIR/rep-N` (log in `DIR/rep-N.log`).
- `--parallel P` (default 2), `--window-size` (default 70), `--mode` (study mode directory to read; default the
  first with a `results.json`).
- `--resume`: replicates whose output is complete are skipped (and counted against the global caps from their
  ledgers). A partial replicate is reported as incomplete and not relaunched or counted. Without `--resume`,
  existing replicate directories are refused.

## What a seed varies

In the study script the seed feeds the replay plan (`build_plan`: the hash order of stories within each class, and
the label-independent development/training split when `--development-rate` is set) and the feedback policy
(which labels are revealed under selective policies). It does not change the decision model or the optimizer's
sampling, so remaining variance there is the services' own nondeterminism.

## Global cap guard

`--max-total-optimizer-calls` / `--max-total-decision-calls` bound all replicates together. The per-run caps are
read from the base args (`--max-optimizer-calls`, default 60; `--max-decision-calls`, default 6000). Before each
launch: finished replicates count at the calls consumed in their `global_budget_ledger.json` (the per-run cap if
the ledger is missing), running replicates and the candidate count at the per-run cap; if the total would exceed
a global cap nothing more is launched. Any non-zero exit also stops further launches (running ones finish).
The guard is conservative by design, so runs that consume far less than their caps can still be blocked.

## Environment and secrets

The harness never reads `.env` files and never prints or writes environment values. Child processes inherit the
caller's environment, so export keys in the shell that launches it. No key value is written to any output file.

## Metrics (per replicate, percent)

Reference labels come from the operational corpus (`operational_corpus` in `protocol.json` if recorded, else the
`--operational-corpus` in the base args; id key, label key `simulated_label` or `label`), because `results.json` holds a
human label only for reviewed stories. Without a readable corpus the recorded labels are used and the summary says
`recorded (reviewed stories only)` when some are missing.

Computed read-only from `results.json` and `runtime.sqlite3` over the first floor(cycles/window) x window cycles:
cyclotron accuracy per window (`predicted_label` vs `human_label`); decision-model-alone accuracy per window
(`prediction` events' `decision_model_label` vs the human label); majority-class baseline per window (the share of
the most common human label, the best constant answer); first `classifier-activated` cycle and the number of
activations; whether a fitted head was used and the share of predictions with `fitted_head`; Jev and optimizer
requests from the ledger; post-warm-up accuracy (mean of windows 2..last); final-window accuracy; area under the
learning curve (mean of all window accuracies).

Aggregate (complete replicates only): median, min, max and a seeded bootstrap 90% interval of the median
(2000 resamples) for final-window and post-warm-up accuracy; reliability = share of replicates whose final window
beats its majority baseline by at least 5 points, and the minimum post-warm-up accuracy. Output:
`replicates-summary.json` and `replicates-summary.md` in `--out-dir`.