# Why no rubric proposal was promoted, and a gate that lets one be

Task decision-flywheel-b1b1827a. The hypotheses were pre-registered in the
task before either replay ran.

## Offline: attempt 4's 24 rubric steps

Attempt 4 is the 1,124-cycle run in `editorial-long-run-20261009.md`.

**Every rubric step took the provisional branch.** In `staged_optimization`
that branch runs while `EvaluationPolicy.recency_allowance(development
counts)` is above zero, which lasts until both classes have
`recency_decay_per_class` = 20 development labels. The study's plan drew
development as a fixed 20 per class from the whole corpus, so the set only
completes at the corpus's last item. Development counts at each attempt:

| Cycle | Development (publish / reject) | Allowance | Incumbent balanced Brier | Candidate balanced Brier | Activated |
| ---: | --- | ---: | ---: | ---: | --- |
| 94 | 1 / 5 | 1.9 | 0.130 | 0.118 | yes |
| 185 | 5 / 6 | 1.5 | 0.173 | 0.248 | yes |
| 238 | 6 / 7 | 1.4 | 0.195 | 0.420 | yes |
| 320 | 8 / 11 | 1.2 | 0.366 | 0.455 | yes |
| 561 | 10 / 15 | 1.0 | 0.551 | 0.483 | yes |
| 901 | 12 / 20 | 0.8 | 0.514 | 0.387 | yes |
| 1,081 | 19 / 22 | 0.1 | 0.463 | 0.462 | yes |

(The table shows 7 of the 24 steps; all are in
`analysis/attempt4-rubric-versions.json`.)

The promotion gate (`DecisionFlywheel.improve`) was never called. The reason
is the minimum evidence, not the promotion margin.

**The provisional margin was also no filter.** It starts at 2.0 balanced-Brier
units, which is the whole range, so every candidate was activated, including
clear regressions on development:
- Cycle 238: balanced Brier 0.195 to 0.420.
- Cycle 320: 0.366 to 0.455, and development accuracy 79% to 58%.

The rubric adopted at 320 was 110 characters long: "Include items with a
meaningful constructive or public-benefit outcome; reject harm-dominant or
routine items." That is the task's own question, returned as the rubric.
While it was active (cycles 321 to 367) the decision model was right on 53% of
items.

**The proposals were otherwise sensible.** The 25 activations held 13
distinct texts; the optimizer often returned the same rubric again. Typical
ones:
- Cycle 45 (1,376 characters): "Label as 'publish' those news articles that
  describe situations where a constructive or beneficial outcome is
  demonstrated or strongly implied. Examples … technological achievements …
  international cooperation … legal or economic rulings with fair outcomes …"
- Cycle 367 (637 characters): "Include items where a concrete constructive or
  public-benefit outcome or advancement is central, such as improved rights or
  access, scientific or technological progress … Exclude items dominated by
  harm, conflict, or controversy without established constructive effects, as
  well as routine, procedural, or informational reports …"

**Development measurements tracked operational accuracy poorly.** On 13 to 41
development items they were noisy, and they sat well below operational
accuracy: development accuracy 57% to 77% from cycle 413 on, against about 82% operational.

## Configuration G and the replay

Configuration G:
- `--development-rate 0.25`: development is a label-independent hash share of
  the stream, the engine's default rate. 100 of the 400 items get the
  development role.
- `--provisional-allowance 0.05`.
- Recency decay unchanged at 20 per class.

Everything else matches the recorded runs: the 400-item corpus, the same seed,
gpt-4.1-mini for both decisions and the optimizer, an empty starting rubric,
and every item reviewed. Exact decision requests already answered in the
412-cycle replicate were served from its answers (`--seed-answers-from`).

There were two replicates, G1 and G2. The baseline is the five recorded runs
with the current configuration.

| Run | Cycles 1–100 | 101–200 | 201–300 | 301–400 | Mean 201–400 | ECE 301–400 | 0.80 rule whole-system right, 301–400 | Rubric promoted |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| 400-cycle recording (v1) | 52% | 89% | 84% | 82% | 83% | 0.017 | 89 | never |
| Attempt 1 | 58% | 86% | 78% | 80% | 79% | | | never |
| Attempt 3 | 56% | 86% | 78% | 78% | 78% | | | never |
| Attempt 4 | 57% | 85% | 84% | 66% | 75% | 0.062 | 92 | never |
| Replicate (cancelled 500 run) | 51% | 85% | 80% | 70% | 75% | 0.095 | 89 | never |
| **G1** | 55% | 83% | 77% | 75% | **76%** | 0.177 | 78 | at 367 |
| **G2** | 58% | 85% | 80% | 78% | **79%** | 0.100 | 78 | at 281 |

The baseline averages 78% over cycles 201 to 400, with a range of 75% to 83%.

**Cost:** G1 $0.77 over 19 minutes (1,439 decision calls billed, 7 optimizer
calls). G2 $1.01 over 23 minutes (1,719 decision calls, 7 optimizer calls).
The seeded answers served the rest.

## Hypotheses

- **H1, the promotion gate is reached: supported.** In both replicates the
  development set reached 20 per class by cycle 238. From then on, rubric
  steps went through `improve`:
  - G1 rejected three candidates on per-class recall safeguards and promoted
    one at cycle 367, on development balanced Brier 0.345 to 0.315.
  - G2 promoted one at 281 and rejected three.
- **H2, no accepted regressions: supported.** No provisional rubric was
  activated with development balanced Brier more than 0.05 worse. The largest
  accepted increase was 0.007. G1 rejected one candidate at 185 (0.213 to
  0.312), and G2 rejected one at 140 (0.245 to 0.283).
- **H3, accuracy at least 81% over cycles 201 to 400 in each replicate: not
  supported.** G1 reached 76% and G2 79%, against the baseline's 78% mean.
  Promoting rubrics through the gate did not raise accuracy within 400 cycles.
- **Secondary, ECE in cycles 301 to 400 no worse than the baseline by more
  than 0.03: not supported.** G1 had 0.177 and G2 0.100, against 0.017 to
  0.095. The 0.80 rule's whole-system right in cycles 301 to 400 was 78 in
  both replicates, against 89 to 92 in the three baselines that have it.

## Side effect: the head was replaced by raw probabilities

In G1, the classifier step at cycle 200 promoted the `raw_decision` feature
set, which is the decision model's own probabilities with no head. It scored
balanced Brier 0.177 against the head's 0.218 on 54 development items. From
cycle 201 to 367 the cyclotron therefore reported the decision model's raw
confidence, with mean 0.964 and ECE 0.19, until the rubric promotion at 367
refit a head.

A development set four times larger let a selection that looks good on
development but is badly calibrated in operation win. With a quarter of the
reviews held out for development, the head also trains on fewer labels.

## Conclusion

The rubric gate never promoted because the study's development set completes
only at the end of the corpus, and the provisional margin accepts anything.

Configuration G fixes both. Rubrics are then promoted (once per run) and
regressions are blocked. Accuracy did not improve (76% and 79% against 78%),
and calibration got worse. On this corpus with gpt-4.1-mini, the rubric is not
the lever that lifts the plateau.

Recommendation:
- Keep the narrow provisional allowance (0.05) as the safer default for
  review-everything runs.
- Do not adopt the 25% development share for this study without a head
  safeguard: `raw_decision` should not be selected over a fitted head unless
  it is also calibrated on development.
- Go on to the decision-model comparison, where the hypothesis is that the
  model, not the review volume, sets the plateau.

Data is under `~/Documents/Codex/2026-10-09/editorial-study/`:
- `gate/g1/`, `gate/g2/`: databases compressed after export.
- `exports/gate-g1-400-v2.json`, `exports/gate-g2-400-v2.json`: schema 2
  fixtures.
- `analysis/`: scripts and tables.