Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

Corpus profiling analysis

Biblicus provides a profiling analysis backend that summarizes corpus contents using deterministic counts and coverage metrics. Profiling is intended as a fast, local baseline before heavier analysis such as topic modeling.

What profiling does

The profiling analysis reports:

The output is structured JSON that can be stored, versioned, and compared across runs.

Run profiling from the CLI

biblicus analyze profile --corpus corpora/example --extraction-run pipeline:RUN_ID

If you omit --extraction-run, Biblicus uses the latest extraction snapshot and emits a reproducibility warning.

To customize profiling metrics, pass a configuration file:

biblicus analyze profile --corpus corpora/example --configuration configurations/profiling.yml --extraction-run pipeline:RUN_ID

Profiling configurations support cascading composition. Pass multiple --configuration files; later configurations override earlier configurations via a deep merge:

biblicus analyze profile \
  --corpus corpora/example \
  --configuration configurations/profiling/base.yml \
  --configuration configurations/profiling/strict.yml \
  --extraction-run pipeline:RUN_ID

To override the composed configuration view from the command line, use --config key=value with dotted keys:

biblicus analyze profile \
  --corpus corpora/example \
  --configuration configurations/profiling/base.yml \
  --config sample_size=200 \
  --extraction-run pipeline:RUN_ID

Profiling configuration configuration

Profiling configurations use the analysis schema version and accept these fields:

Example configuration:

schema_version: 1
sample_size: 500
min_text_characters: 50
percentiles: [50, 90, 99]
top_tag_count: 10
tag_filters: ["ag_news", "label:World"]

Run profiling from Python

from pathlib import Path

from biblicus.analysis import get_analysis_backend
from biblicus.corpus import Corpus
from biblicus.models import ExtractionRunReference

corpus = Corpus.open(Path("corpora/example"))
backend = get_analysis_backend("profiling")
output = backend.run_analysis(
    corpus,
    configuration_name="default",
    config={
        "schema_version": 1,
        "sample_size": 500,
        "min_text_characters": 50,
        "percentiles": [50, 90, 99],
        "top_tag_count": 10,
        "tag_filters": ["ag_news"],
    },
    extraction_snapshot=ExtractionRunReference(
        extractor_id="pipeline",
        snapshot_id="RUN_ID",
    ),
)
print(output.model_dump())

Output location

Profiling output is stored under:

analysis/profiling/<snapshot_id>/output.json

Reading the report

Profiling output is a structured report with separate sections for raw items and extracted text. A shortened example:

{
  "analysis_id": "profiling",
  "generated_at": "2024-01-01T00:00:00Z",
  "report": {
    "raw_items": {
      "total_items": 3,
      "media_type_counts": {
        "text/markdown": 3
      }
    },
    "extracted_text": {
      "extracted_nonempty_items": 3,
      "extracted_empty_items": 0,
      "extracted_missing_items": 0
    }
  }
}

The raw_items section summarizes corpus composition. The extracted_text section tells you how much content made it through extraction and how much was missing or empty.

Comparing profiling runs

Use the same extraction snapshot and configuration configuration whenever you compare profiling outputs:

  1. Run profiling on two corpus snapshots.
  2. Compare raw_items.total_items, media type counts, and tag coverage.
  3. Compare extracted_text coverage to spot extraction regressions.

Record the snapshot identifiers and catalog timestamps so you can trace differences later.

Common pitfalls

Working demo

A runnable demo is provided in scripts/profiling_demo.py. It downloads a corpus, runs extraction, and executes the profiling analysis so you can inspect the output:

python scripts/profiling_demo.py --corpus corpora/profiling_demo --force