Topic identity
BERTopic numbers its clusters from zero on every run, so "cluster 41" in this week's run has nothing to do with "cluster 41" last week. Topic identity matches the clusters of a run to the clusters of the previous run, assigns each cluster a stable uid that is carried forward, and reports what happened to it: new, continued, merged, split, or gone.
Biblicus supplies identity and statistics only. Naming a cluster, accepting it, and deciding what to publish belong to the consumer, which keys its own state on the uid.
Run it
Run topic modeling first, then identify the run. The first run has no previous run:
biblicus analyze topics --corpus corpora/AI-ML-research ... # produces <run_a>
biblicus analyze topic-identity --corpus corpora/AI-ML-research --current <run_a>
Every later run is matched to the run before it:
biblicus analyze topic-identity \
--corpus corpora/AI-ML-research \
--current <run_b> \
--previous <run_a> \
--threshold 0.5
--currentis the topic modeling snapshot identifier to identify (thesnapshot.snapshot_idofbiblicus analyze topics).--previousis the previous topic modeling snapshot. It must already have been identified, because stable identifiers are carried forward from its identity report. A missing report is an error that tells you which command to run first.--thresholdis the minimum Jaccard overlap of member item identifiers for two clusters to match. It must be greater than 0 and at most 1, and defaults to 0.5.--trends-snapshotoptionally names atopic-trendssnapshot built with--bin weekly --weeks 8or more for the same--currentrun. Each cluster then carries its 8-week series. See Topic trends.--format json(default) ormarkdownselects the standard output rendering. The JSON file is written either way.
How clusters are matched
Jaccard overlap is the number of shared item identifiers divided by the number of distinct item identifiers in either cluster. Every pair of a previous cluster and a current cluster whose overlap reaches the threshold is a match. The outlier topic (-1) is not a cluster and is never matched.
Each previous uid is carried to at most one current cluster: matches are taken in order of highest overlap, then previous uid, then current topic identifier, and a previous cluster or a current cluster is used once. Current clusters left without a carried uid get a new one, cl- followed by the first 12 characters of the SHA-256 of <run id>:<topic id> for the run in which they first appear. The same two runs and threshold always give the same identifiers and events.
| Event | When | uid |
|---|---|---|
new |
no previous cluster overlaps it | new, derived from this run and topic |
continued |
exactly one previous cluster overlaps it and that cluster overlaps nothing else | the previous uid |
merged |
two or more previous clusters overlap it | the uid of its best-overlapping predecessor if still free, otherwise new |
split |
exactly one previous cluster overlaps it, and that cluster also overlaps other current clusters | the best-overlapping successor keeps the previous uid; the others get new ones |
gone |
a previous cluster no current cluster overlaps | listed under gone, not under clusters |
predecessor_uids lists every overlapping previous cluster. For a split, successor_uids lists the uid of every current cluster that descends from the same predecessor, including the cluster itself.
first_seen is the date of the first run that contained the cluster. It is the date part of that run's generated_at, carried forward through continued, merged (the earliest predecessor) and split. earliest_published_at is the earliest dates.published_at among the member items, so a consumer can tell a cluster that is new to the pipeline from one that is new to the world.
Output
The report is written to:
analysis/topic-identity/<current run id>/output.json
and is the standard output of the command. It is a TopicIdentityOutput; the published schema is docs/api-schemas/analyze-topic-identity.response.schema.json and the request schema is analyze-topic-identity.request.schema.json. Re-running the same command overwrites the file with identical content.
{
"schema_version": 1,
"analysis_id": "topic-identity",
"run_id": "<current run id>",
"previous_run_id": "<previous run id>",
"generated_at": "2026-09-08T00:00:00+00:00",
"threshold": 0.5,
"trends_snapshot_id": null,
"series_iso_weeks": null,
"document_count": 26,
"outlier_document_count": 6,
"cluster_count": 1,
"event_counts": {"new": 0, "continued": 1, "merged": 0, "split": 0, "gone": 1},
"clusters": [
{
"uid": "cl-1f3a9c0b27d4",
"topic_id": 4,
"event": "continued",
"predecessor_uids": ["cl-1f3a9c0b27d4"],
"successor_uids": [],
"label": "Agent Reliability",
"top_terms": ["agent", "reliability"],
"member_item_count": 20,
"member_item_ids": ["..."],
"first_seen": "2026-09-01",
"earliest_published_at": "2026-03-02",
"weekly_series": null
}
],
"gone": [
{
"uid": "cl-77aa01c3be90",
"label": "Old Topic",
"top_terms": ["omega"],
"member_item_count": 11,
"first_seen": "2026-09-01",
"last_seen_run_id": "<previous run id>"
}
],
"artifact_paths": {"output": "/path/to/analysis/topic-identity/<run>/output.json"}
}
Field notes:
top_termsholds up to 10 topic keywords in rank order;member_item_idsis sorted;member_item_countequals its length.weekly_seriesisnullunless--trends-snapshotis given. When given, it is the last 8 weekly counts of the cluster's topic, oldest to newest, andseries_iso_weekslists the 8 matching ISO week labels. The trends snapshot must have been built from the same--currentrun with--bin weeklyand at least 8 weeks, otherwise the command fails with a message saying so.document_count,outlier_document_countandcluster_countrecord the size of the run. Growing, cooling and new-cluster labels are a front-end rule applied only after four or more weeks of series data; Biblicus does not compute them.eventis one ofnew,continued,merged,split.
Corpus size
BERTopic needs a reasonable corpus before its clusters stay put between runs. With too few documents clusters appear and vanish, which shows up here as spurious new and gone events. Record, for each corpus you track, the smallest document_count at which events were stable, and treat runs below it as unreliable. See Topic modeling.
Behavior specifications
features/topic_identity.feature specifies a continuing cluster, the first run, a new cluster, a merge, a split, a disappearing cluster, determinism, the threshold, carry-forward across three runs, reported statistics and corpus size, the failure cases, Markdown output, and the weekly series.