Cyclotron

Editorial intelligence for publications.

Contents

Measuring what a cyclotron learned

This page defines Cyclotron's headline evidence: a before-and-after number on a fixed held-out set of human labels. It covers the metric, how the held-out set is chosen and frozen, the report format, and how the shared UI shows it in an embedded application (Papyrus) and in the central lab.

The question it answers

On reviews it never trained on, how much better does the cyclotron agree with editors now than when it was created?

Two classifiers answer the same items:

Both are scored on every item of one frozen held-out set, with the same request clock, so the only difference between the two columns is what the cyclotron learned from reviews.

The metric

The headline is balanced accuracy: for each label, the share of items with that label the classifier got right, averaged over labels. A rare label (include in Papyrus) counts as much as a common one, so a classifier that says exclude to everything scores 50%, not the 85% plain accuracy would give it.

The report also carries accuracy, precision and recall for the positive label (macro averages when a classifier has no positive label) and agreement on each label, for both endpoints.

Weighting. Once the review rate tapers, confident decisions are reviewed only at a sampled rate. Each held-out item therefore counts with the inverse of the probability it was sent to review (its recorded selection propensity), so the numbers stand for every decision in the audit share and not only the uncertain ones that were reviewed. While every decision is reviewed, every weight is 1.

Is the change real? The report puts a 95% interval on the headline before and after and on the paired change (headline.beforeInterval, afterInterval, changeInterval; the change in percentage points). The intervals come from a bootstrap over the held-out items: 1,000 resamples with replacement, each recomputing the weighted balanced accuracy of both endpoints, from a generator seeded by the set id so the same set and predictions always give the same intervals. The verdict follows the weighted change interval, so it cannot contradict the headline number:

Verdict When
nothing-learned-yet The active classifier is still version 1 with no ML model.
too-few-labels The rarest label has fewer than minimumPerLabel held-out items (default 10). Numbers are shown, no claim is made.
improved The change interval is entirely above zero.
worse The change interval is entirely below zero.
no-clear-change The change interval includes zero.

The report still counts the items the active classifier newly gets right (gained) and newly gets wrong (lost) and runs an exact two-sided sign test on those pairs (pValue). Those are unweighted item counts shown for transparency; once the review rate tapers they can point the other way from the weighted headline, and change.summary says the verdict follows the weighted interval. Reports written before the intervals existed carry no interval fields.

Only improved supports a public claim such as the marketing site's.

The held-out set

The engine already splits items by identity before any label is seen: about 20% of items fall in the audit share, and their reviews never train, tune, prime or select examples for any classifier. A held-out set is frozen from that share:

Sets and reports are stored in the cyclotron's own store, so they work the same on the local SQLite store and the S3 cloud store, and travel with snapshot and restore.

Cost

Freezing makes no model call. Evaluating makes at most two decision-model requests per scored item (held_out_request_bound()), through the store's shared request cache: re-evaluating after no change costs nothing, and after a new version only the "after" column is new. An evaluation that could exceed the open's request ceiling is refused before any call. Nothing is learned, fitted or promoted during an evaluation.

Engine API

summary = cyclotron.freeze_held_out_set()                # no model call
bound = cyclotron.held_out_request_bound()               # most new requests an evaluation can make
report = await cyclotron.evaluate_held_out()             # cyclotron-held-out-report/v1
history = cyclotron.held_out_reports()                   # the last 10, oldest first

Each takes classifier= for a cyclotron with several outputs, and evaluate_held_out takes set_id=, minimum_per_label= and the same execution_guard= as decide.

Report format

cyclotron-held-out-report/v1 is described by src/decision_flywheel/schemas/cyclotron-held-out-report.v1.schema.json, with an example beside it and TypeScript types in the package (cyclotron/held-out-report). Its parts:

Showing it in either deployment mode

The package exports HeldOutReportView (render a report) and HeldOutPanel (load, freeze and evaluate through the host's transport) from cyclotron/components/held-out-report, and createHeldOutClient from cyclotron/held-out-report. Styles are in styles/components.css.

Both modes serve the same two GraphQL operations, so the card is the same component on the same data:

query    { heldOutOverview(cyclotronId: ID!, classifier: String): JSON }
mutation { heldOutCommand(action: "freeze" | "evaluate", payload: JSON!): JSON }

decision_flywheel.held_out_service.HeldOutService implements them. The host supplies open_cyclotron(cyclotron_id, max_requests=None), which opens the authorized store for one call and raises for any other (evaluating reopens with max_requests equal to the confirmed request bound); a scale-to-zero host such as Papyrus holds the store's lease only while the call runs.

evaluate needs live authority (allow_live=True) and the caller must send confirmedRequestBound equal to the bound the overview showed; the panel's button says "Evaluate (up to N model requests)".