Corpus profiling analysis
Biblicus provides a profiling analysis backend that summarizes corpus contents using deterministic counts and coverage metrics. Profiling is intended as a fast, local baseline before heavier analysis such as topic modeling.
What profiling does
The profiling analysis reports:
- Total item count and media type distribution
- Extracted text coverage (present, empty, missing)
- Size and length distributions with percentiles
- Tag coverage and top tags
The output is structured JSON that can be stored, versioned, and compared across runs.
Run profiling from the CLI
biblicus analyze profile --corpus corpora/example --extraction-run pipeline:RUN_ID
If you omit --extraction-run, Biblicus uses the latest extraction snapshot and emits a reproducibility warning.
To customize profiling metrics, pass a configuration file:
biblicus analyze profile --corpus corpora/example --configuration configurations/profiling.yml --extraction-run pipeline:RUN_ID
Profiling configurations support cascading composition. Pass multiple --configuration files; later configurations override earlier configurations
via a deep merge:
biblicus analyze profile \
--corpus corpora/example \
--configuration configurations/profiling/base.yml \
--configuration configurations/profiling/strict.yml \
--extraction-run pipeline:RUN_ID
To override the composed configuration view from the command line, use --config key=value with dotted keys:
biblicus analyze profile \
--corpus corpora/example \
--configuration configurations/profiling/base.yml \
--config sample_size=200 \
--extraction-run pipeline:RUN_ID
Profiling configuration configuration
Profiling configurations use the analysis schema version and accept these fields:
schema_version: analysis schema version, currently1sample_size: optional cap for distribution calculationsmin_text_characters: minimum extracted text length for inclusionpercentiles: percentiles to compute for size and length distributionstop_tag_count: maximum number of tags to list intop_tagstag_filters: optional list of tags to include in tag coverage metrics
Example configuration:
schema_version: 1
sample_size: 500
min_text_characters: 50
percentiles: [50, 90, 99]
top_tag_count: 10
tag_filters: ["ag_news", "label:World"]
Run profiling from Python
from pathlib import Path
from biblicus.analysis import get_analysis_backend
from biblicus.corpus import Corpus
from biblicus.models import ExtractionRunReference
corpus = Corpus.open(Path("corpora/example"))
backend = get_analysis_backend("profiling")
output = backend.run_analysis(
corpus,
configuration_name="default",
config={
"schema_version": 1,
"sample_size": 500,
"min_text_characters": 50,
"percentiles": [50, 90, 99],
"top_tag_count": 10,
"tag_filters": ["ag_news"],
},
extraction_snapshot=ExtractionRunReference(
extractor_id="pipeline",
snapshot_id="RUN_ID",
),
)
print(output.model_dump())
Output location
Profiling output is stored under:
analysis/profiling/<snapshot_id>/output.json
Reading the report
Profiling output is a structured report with separate sections for raw items and extracted text. A shortened example:
{
"analysis_id": "profiling",
"generated_at": "2024-01-01T00:00:00Z",
"report": {
"raw_items": {
"total_items": 3,
"media_type_counts": {
"text/markdown": 3
}
},
"extracted_text": {
"extracted_nonempty_items": 3,
"extracted_empty_items": 0,
"extracted_missing_items": 0
}
}
}
The raw_items section summarizes corpus composition. The extracted_text section tells you how much content made it
through extraction and how much was missing or empty.
Comparing profiling runs
Use the same extraction snapshot and configuration configuration whenever you compare profiling outputs:
- Run profiling on two corpus snapshots.
- Compare
raw_items.total_items, media type counts, and tag coverage. - Compare
extracted_textcoverage to spot extraction regressions.
Record the snapshot identifiers and catalog timestamps so you can trace differences later.
Common pitfalls
- Profiling without specifying an extraction snapshot, which makes comparisons harder to reproduce.
- Comparing runs with different
sample_sizeormin_text_characterssettings. - Interpreting tag counts without noting the
tag_filtersapplied.
Working demo
A runnable demo is provided in scripts/profiling_demo.py. It downloads a corpus, runs extraction, and executes the
profiling analysis so you can inspect the output:
python scripts/profiling_demo.py --corpus corpora/profiling_demo --force