Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

Topic identity

BERTopic numbers its clusters from zero on every run, so "cluster 41" in this week's run has nothing to do with "cluster 41" last week. Topic identity matches the clusters of a run to the clusters of the previous run, assigns each cluster a stable uid that is carried forward, and reports what happened to it: new, continued, merged, split, or gone.

Biblicus supplies identity and statistics only. Naming a cluster, accepting it, and deciding what to publish belong to the consumer, which keys its own state on the uid.

Run it

Run topic modeling first, then identify the run. The first run has no previous run:

biblicus analyze topics --corpus corpora/AI-ML-research ...                    # produces <run_a>
biblicus analyze topic-identity --corpus corpora/AI-ML-research --current <run_a>

Every later run is matched to the run before it:

biblicus analyze topic-identity \
  --corpus corpora/AI-ML-research \
  --current <run_b> \
  --previous <run_a> \
  --threshold 0.5

How clusters are matched

Jaccard overlap is the number of shared item identifiers divided by the number of distinct item identifiers in either cluster. Every pair of a previous cluster and a current cluster whose overlap reaches the threshold is a match. The outlier topic (-1) is not a cluster and is never matched.

Each previous uid is carried to at most one current cluster: matches are taken in order of highest overlap, then previous uid, then current topic identifier, and a previous cluster or a current cluster is used once. Current clusters left without a carried uid get a new one, cl- followed by the first 12 characters of the SHA-256 of <run id>:<topic id> for the run in which they first appear. The same two runs and threshold always give the same identifiers and events.

Event When uid
new no previous cluster overlaps it new, derived from this run and topic
continued exactly one previous cluster overlaps it and that cluster overlaps nothing else the previous uid
merged two or more previous clusters overlap it the uid of its best-overlapping predecessor if still free, otherwise new
split exactly one previous cluster overlaps it, and that cluster also overlaps other current clusters the best-overlapping successor keeps the previous uid; the others get new ones
gone a previous cluster no current cluster overlaps listed under gone, not under clusters

predecessor_uids lists every overlapping previous cluster. For a split, successor_uids lists the uid of every current cluster that descends from the same predecessor, including the cluster itself.

first_seen is the date of the first run that contained the cluster. It is the date part of that run's generated_at, carried forward through continued, merged (the earliest predecessor) and split. earliest_published_at is the earliest dates.published_at among the member items, so a consumer can tell a cluster that is new to the pipeline from one that is new to the world.

Output

The report is written to:

analysis/topic-identity/<current run id>/output.json

and is the standard output of the command. It is a TopicIdentityOutput; the published schema is docs/api-schemas/analyze-topic-identity.response.schema.json and the request schema is analyze-topic-identity.request.schema.json. Re-running the same command overwrites the file with identical content.

{
  "schema_version": 1,
  "analysis_id": "topic-identity",
  "run_id": "<current run id>",
  "previous_run_id": "<previous run id>",
  "generated_at": "2026-09-08T00:00:00+00:00",
  "threshold": 0.5,
  "trends_snapshot_id": null,
  "series_iso_weeks": null,
  "document_count": 26,
  "outlier_document_count": 6,
  "cluster_count": 1,
  "event_counts": {"new": 0, "continued": 1, "merged": 0, "split": 0, "gone": 1},
  "clusters": [
    {
      "uid": "cl-1f3a9c0b27d4",
      "topic_id": 4,
      "event": "continued",
      "predecessor_uids": ["cl-1f3a9c0b27d4"],
      "successor_uids": [],
      "label": "Agent Reliability",
      "top_terms": ["agent", "reliability"],
      "member_item_count": 20,
      "member_item_ids": ["..."],
      "first_seen": "2026-09-01",
      "earliest_published_at": "2026-03-02",
      "weekly_series": null
    }
  ],
  "gone": [
    {
      "uid": "cl-77aa01c3be90",
      "label": "Old Topic",
      "top_terms": ["omega"],
      "member_item_count": 11,
      "first_seen": "2026-09-01",
      "last_seen_run_id": "<previous run id>"
    }
  ],
  "artifact_paths": {"output": "/path/to/analysis/topic-identity/<run>/output.json"}
}

Field notes:

Corpus size

BERTopic needs a reasonable corpus before its clusters stay put between runs. With too few documents clusters appear and vanish, which shows up here as spurious new and gone events. Record, for each corpus you track, the smallest document_count at which events were stable, and treat runs below it as unreliable. See Topic modeling.

Behavior specifications

features/topic_identity.feature specifies a continuing cluster, the first run, a new cluster, a merge, a split, a disappearing cluster, determinism, the threshold, carry-forward across three runs, reported statistics and corpus size, the failure cases, Markdown output, and the weekly series.