# Which items to review: least-confident, random, or mixed

Task decision-flywheel-b0cf2975. The hypotheses were pre-registered in a task
comment before any run.

## Design

All arms use:
- Jev 1.13.0 as the decision model, chosen in
  `decision-model-comparison-20261009.md`.
- The 1,200-item extended corpus with the same seed.
- An empty starting rubric and the recorded gate configuration.
- gpt-4.1-mini as the optimizer, and the item text only (no title).

The arms differ only in which items are reviewed:
- **Review everything (reference):** this run stopped at cycle 1,031 on the
  decision-request cap I set too low (8,000). It is compared over cycles 401
  to 1,000.
- **Random 25%:** a seeded hash picks the items.
- **Least-confident 25%:** an item is reviewed when its confidence is at or
  below the 25th percentile of the previous 100 issued confidences. The first
  20 cycles are random.
- **Mixed:** least-confident 20%, plus a random audit of about 5% of all
  items.

Selection uses only the issued confidence, never the hidden label.

Jev's confidences take few distinct values, so many items tie at the
percentile cut, and "at or below" reviews all of them. The two
confidence-based arms therefore reviewed more than 25%: least-confident 33.5%
and mixed 35.7%, against random's 25.6%. Compare them with that in mind; the
per-label row below corrects for it.

Whole-system accuracy counts reviewed items as right. Learning means the
cyclotron's own accuracy on all items, evaluator-only.

## Results, cycles 401 to 1,200 (reference arm: 401 to 1,000)

| Arm | Labels (all cycles) | Labels 401+ | Learning accuracy | ECE | Whole-system accuracy | Mistakes that slipped through | Mistakes caught per 100 labels | Worst whole-system window after 200 | USD | Minutes |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| Review everything | 1,000 | 600 | 84.3% | 0.094 | 100% | 0 | 15.7 | 100% | 2.69 | 68 |
| Random 25% | 307 | 207 | **86.4%** | **0.079** | 89.6% | 83 | 12.6 | 83% | 0.28 | 18 |
| Least-confident 25% | 402 | 289 | 83.4% | 0.092 | 94.4% | 45 | **30.4** | 88% | 0.45 | 22 |
| Mixed (20% + 5% audit) | 428 | 310 | 85.4% | 0.095 | **95.1%** | **39** | 25.2 | 84% | 0.48 | 23 |

Per-window tables are in `analysis/selection-*.json`.

## Hypotheses

- **H1, least-confident's whole-system accuracy is at least 5 points above
  random's: not supported, narrowly.** The gap was 4.8 points, and
  least-confident also reviewed about 30% more items. Per label it is clearly
  better: it caught 30.4 mistakes per 100 labels against random's 12.6.
- **H2, least-confident's ECE is at least 0.03 worse than random's: not
  supported.** The gap was 0.013. Calibration did not collapse without an
  audit.
- **H3, mixed is within 2 points of least-confident on whole-system accuracy
  and within 0.02 of random on ECE: supported.** It scored 95.1% against
  94.4%, and ECE 0.095 against 0.079.
- **H4, least-confident learns at least as well as random: not supported.**
  Random learned best: 86.4% against 83.4%. Reviewing everything did not
  learn better either (84.3% over 401 to 1,000).
  - One reading is that review volume beyond about 25% does not raise this
    plateau.
  - Fewer labels also means fewer rubric rewrites. The arms ran 22 (first 1,000 cycles), 7, 8 and 9 optimizer calls (everything, random, least-confident, mixed), so the rubric churned less.

## Recommendation

**Mixed is the practical default.** It catches mistakes nearly as well as
least-confident: it had the fewest slipped (39) and the highest whole-system
accuracy (95.1%), at a sixth of review-everything's cost. Its random audit
keeps an unbiased sample for calibration and promotion checks.

**Random-only is the right choice when the goal is the model's own accuracy
rather than catching mistakes.**

Caveats:
- One run per arm, and window-to-window spread on this corpus is about ±5
  points.
- The realized rates differ because of tied confidences. The percentile cut
  should break ties at random so the rate holds at 25%; that is a follow-up.
- The reference arm is short by 200 cycles.

Fixtures (schema 2) are in `exports/selection-*-v2.json` under
`~/Documents/Codex/2026-10-09/editorial-study/`. The databases are compressed
after export.