Decision model comparison at 400 cycles
Task decision-flywheel-c9c5b6e2. Hypotheses H1 to H3 were pre-registered in the task before the runs. H4, the ceiling check, was registered in a task comment after the two runs and before its own calls.
Design
All runs use:
- The 400-item editorial corpus with the same seed.
- Every item reviewed, starting from an empty rubric.
- The recorded runs' gate configuration.
- gpt-4.1-mini as the optimizer in every arm.
Only the decision model changes:
- Jev 1.13.0: one SDK request per item answers the decision and every discovered element. Price: $0.042 per million input tokens.
- gpt-4.1-mini: the five recorded runs; no new run.
- gpt-4.1: $2.00 input, $0.50 cached input and $8.00 output per million tokens (OpenAI model page, read 2026-10-09).
Accuracy and ECE are scored against the frozen reference labels, evaluator-only. "Raw" is the decision model's own label and stated confidence.
Results
| Decision model | 1–100 | 101–200 | 201–300 | 301–400 | Mean 201–400 | ECE 301–400 (cyclotron / raw) | Labels differ from raw | Rubric promoted | ML-model promotions | USD (decisions + optimizer) | Minutes |
|---|---|---|---|---|---|---|---|---|---|---|---|
| gpt-4.1-mini (5 runs) | 51–58% | 85–89% | 78–84% | 66–82% | 78% (75–83) | 0.017–0.095 / 0.152–0.324 | 0 (replicate) to 38 (attempt 4) | never | 2 per run | 0.78 (v1) to 0.94 (replicate, 412 cycles) | 19 (replicate, current code) |
| gpt-4.1 | 55% | 83% | 79% | 77% | 78% | 0.066 / 0.175 | 10 (cyclotron right 5, raw right 5) | never | 2 | 4.08 (3.74 + 0.33) | 25 |
| Jev 1.13.0 | 80% | 83% | 81% | 78% | 79.5% | 0.070 / 0.140 | 28 (cyclotron right 17, raw right 11) | never (one invalid proposal dropped) | 1 (one refit rejected) | 0.37 (0.05 + 0.32) | 7 |
In the last window (cycles 301 to 400), measured on the cyclotron's confidence:
| 0.80 rule: approved | Approved right | Whole system right | Share of mistakes in least-confident 15% | In least-confident 30% | |
|---|---|---|---|---|---|
| gpt-4.1-mini (v1 recording) | 74 | 63 | 89 | 25% | |
| gpt-4.1 | 67 | 56 | 89 | 22% | 47% |
| Jev 1.13.0 | 71 | 62 | 91 | 41% | 61% |
Hypotheses
- H1, a stronger model raises the plateau: not supported. gpt-4.1 scored 78% over cycles 201 to 400, the same as gpt-4.1-mini's mean and below its long-run plateau of 82%. It cost five times as much.
- H2, Jev within 5 points of gpt-4.1-mini: supported. Jev scored 79.5%.
- Jev started at 80% with no rubric at all, against 51% to 58% for both OpenAI models.
- It was the cheapest arm ($0.05 of decision calls) and the fastest.
- Its mistakes are easiest to route: the least-confident 15% held 41% of them.
- Its ML head changed more labels (28), and was right on 17 of them against the raw model's 11. In cycles 201 to 300 the cyclotron scored 81% while raw Jev scored 74%.
- H3, the cyclotron's calibration beats the raw model's whatever the model: supported. It held in every run, with cyclotron ECE 0.02 to 0.10 against raw 0.14 to 0.32.
Ceiling check (H4)
The reference labels were generated by gpt-4.1-mini at temperature 0 under a
stored rubric (task-2/semantic_v2_label.py), from the title plus the article
excerpt. The study's decision model receives only the item's text field,
which is the same excerpt without the title.
We gave gpt-4.1-mini, at temperature 0, the labeling rubric verbatim, with no learning and no optimizer:
| Input | Agreement with reference | Publish recall | Reject recall | USD |
|---|---|---|---|---|
| Text only (what the study's decision model sees) | 89.0% | 79% | 95% | 0.16 |
| Title + text (what the labeler saw) | 95.5% | 99% | 93% | 0.17 |
- H4, text-only agreement at most 88%: not supported, narrowly. It reached 89.0%. The plateau is therefore not purely an input ceiling.
- What remains:
- Even a perfect rubric reaches about 89% on the study's input. The learning loop reaches 78% to 82%, so about 7 to 11 points are left for rubric learning.
- About 6.5 more points are lost because the study never shows the title. Without it, the labeling model itself misses 21% of publish items.
Conclusion
On this corpus, the decision model's strength does not set the plateau: gpt-4.1 at 5x the cost matched gpt-4.1-mini. Two things do matter:
- The input: the title is missing from what the decision model sees.
- The rubric: the gap between learned rubrics and the labeling rubric.
Jev 1.13.0 matches the OpenAI models' accuracy from the first window, at a fraction of the cost, and its confidence routes mistakes best. That makes it the model for the next experiments.
Next, pre-registered separately: show the title to the decision model, as the labeler saw it. Then run the selection and taper experiments on Jev with the title.
Data is under ~/Documents/Codex/2026-10-09/editorial-study/:
models/gpt41,models/jev113: databases compressed after export.exports/models-gpt41-400-v2.json,exports/models-jev113-400-v2.json.ceiling/: the check script, per-item answers and summary.
Follow-up: showing the title (decision-flywheel task "show the decision model the title")
The title experiment was pre-registered before its runs. Its design matches
the runs above, with --item-fields title,text.
| Decision model | 1–100 | 101–200 | 201–300 | 301–400 | Mean 201–400 | Without the title | ECE 301–400 (cyclotron / raw) | USD | Minutes |
|---|---|---|---|---|---|---|---|---|---|
| Jev 1.13.0 | 67% | 85% | 77% | 77% | 77% | 79.5% | 0.054 / 0.208 | 0.37 | 5 |
| gpt-4.1-mini | 56% | 85% | 85% | 82% | 83.5% | 78% (75–83) | 0.039 / 0.150 | 0.91 | 20 |
- H1, Jev with the title at least 85%: not supported. It reached 77%, lower than without the title. Its first window also fell from 80% to 67%.
- H2, gpt-4.1-mini with the title at least 83%: supported, narrowly. It reached 83.5%, against a no-title range of 75% to 83%. One run is within the recorded spread, so this is weak evidence.
- H3, cyclotron ECE below the raw model's: supported in both.
With the labeling rubric itself, the title is worth 6.5 points. Rubrics the loop learns from reviewers' explanations do not reliably capture that gain. The remaining gap is in what the optimizer learns, not in what the model sees.