Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

Benchmarking Quickstart Guide

Get started with benchmarking Biblicus extraction pipelines in under 10 minutes.

Prerequisites

Quick Start (5 minutes)

Step 1: Download Benchmark Dataset

python scripts/download_funsd_samples.py

This downloads 20 FUNSD form images into corpora/funsd_benchmark/ with ground truth text files.

What is FUNSD?

Step 2: Run Quick Benchmark

Run a quick benchmark on a subset of pipelines:

python scripts/benchmark_all_pipelines.py \
  --corpus-path corpora/funsd_benchmark \
  --config configs/benchmark/quick.yaml \
  --output results/quick_benchmark.json

What this does:

Step 3: View Results

Console summary:

cat results/quick_benchmark.json | jq '.pipelines[] | {name, f1: .metrics.set_based.avg_f1, recall: .metrics.set_based.avg_recall}'

Full report:

cat results/quick_benchmark.json | jq '.'

Example output:

{
  "name": "paddleocr",
  "f1": 0.787,
  "recall": 0.782
}
{
  "name": "docling-smol",
  "f1": 0.728,
  "recall": 0.675
}

Interpretation:

See Metrics Reference for detailed explanations.


Standard Benchmark (30-60 minutes)

For more thorough validation before release:

python scripts/benchmark_all_pipelines.py \
  --corpus-path corpora/funsd_benchmark \
  --config configs/benchmark/standard.yaml \
  --output results/standard_benchmark.json

Differences from quick:


Full Benchmark (2-4 hours)

For comprehensive evaluation:

python scripts/benchmark_all_pipelines.py \
  --corpus-path corpora/funsd_benchmark \
  --config configs/benchmark/full.yaml \
  --output results/full_benchmark.json

Differences:


Benchmark Specific Pipelines

Single Pipeline Benchmark

Test just one pipeline:

python scripts/evaluate_ocr_pipeline.py \
  --corpus corpora/funsd_benchmark \
  --config configs/ocr-paddleocr.yaml \
  --output results/paddleocr_results.json

Compare Two Pipelines

Direct comparison between two approaches:

python scripts/benchmark_heron_vs_paddleocr.py \
  --corpus corpora/funsd_benchmark \
  --output results/heron_vs_paddleocr.json

This script specifically compares:

Quick Layout-Aware Validation

Test layout-aware pipeline quickly:

python scripts/quick_benchmark_layout_aware.py

Validates the layout-aware Tesseract pipeline on a small subset.


Understanding Results

Key Metrics to Check

For Forms (FUNSD):

  1. F1 Score - Overall accuracy (target: ≥0.75)
  2. Recall - Completeness (target: ≥0.70)
  3. WER - Reading order quality (target: ≤0.60)

For Receipts:

  1. F1 Score - Entity extraction accuracy (target: ≥0.80)
  2. Precision - Clean output (target: ≥0.75)

For Academic Papers:

  1. LCS Ratio - Reading order preservation (target: ≥0.75)
  2. Bigram Overlap - Column mixing detection (target: ≥0.60)

See Metrics Reference for detailed explanations.

Result Structure

{
  "benchmark_timestamp": "2026-02-13T12:00:00Z",
  "corpus_path": "corpora/funsd_benchmark",
  "total_pipelines": 8,
  "successful_pipelines": 8,
  "failed_pipelines": 0,
  "pipelines": [
    {
      "name": "paddleocr",
      "snapshot_id": "abc123...",
      "success": true,
      "metrics": {
        "set_based": {
          "avg_f1": 0.787,
          "avg_precision": 0.792,
          "avg_recall": 0.782
        },
        "order_aware": {
          "avg_wer": 0.533,
          "avg_sequence_accuracy": 0.031,
          "avg_lcs_ratio": 0.621
        },
        "ngram": {
          "avg_bigram_overlap": 0.521,
          "avg_trigram_overlap": 0.412
        }
      },
      "total_documents": 20,
      "processing_time": 145.3
    }
  ],
  "best_performers": {
    "best_f1": "paddleocr",
    "best_sequence_accuracy": "docling-smol",
    "lowest_wer": "paddleocr",
    "best_bigram": "heron-tesseract"
  }
}

Exporting Results

To CSV for spreadsheet analysis:

from pathlib import Path
from biblicus.evaluation.ocr_benchmark import OCRBenchmark
import json

# Load benchmark results
with open("results/quick_benchmark.json") as f:
    results = json.load(f)

# Export to CSV
for pipeline in results["pipelines"]:
    name = pipeline["name"]
    # Access per-document results if needed
    # report.to_csv(Path(f"results/{name}_per_document.csv"))

To JSON for programmatic access:

Results are already in JSON format. Use jq for filtering:

# Get F1 scores only
jq '.pipelines[] | {name, f1: .metrics.set_based.avg_f1}' results/quick_benchmark.json

# Filter by F1 > 0.70
jq '.pipelines[] | select(.metrics.set_based.avg_f1 > 0.70)' results/quick_benchmark.json

# Best performer
jq '.best_performers' results/quick_benchmark.json

Customizing Benchmarks

Create Custom Benchmark Config

Create configs/benchmark/my-benchmark.yaml:

# Benchmark configuration
mode: custom
categories:
  - name: forms
    corpus_path: corpora/funsd_benchmark
    ground_truth_dir: metadata/funsd_ground_truth
    primary_metric: f1
    document_count: 30  # Subset of documents

pipelines:
  - configs/baseline-ocr.yaml
  - configs/ocr-paddleocr.yaml
  - configs/my-custom-pipeline.yaml

Run with:

python scripts/benchmark_all_pipelines.py \
  --config configs/benchmark/my-benchmark.yaml \
  --output results/my_benchmark.json

Add Custom Pipeline

  1. Create pipeline config in configs/my-custom-pipeline.yaml:
extractor_id: pipeline
config:
  stages:
    - extractor_id: heron-layout
      config:
        model_variant: "101"
        confidence_threshold: 0.7
    - extractor_id: ocr-paddleocr-vl
      config:
        use_layout_metadata: true
        lang: en
  1. Test pipeline manually:
from pathlib import Path
from biblicus import Corpus
from biblicus.extraction import build_extraction_snapshot
from biblicus.evaluation.ocr_benchmark import OCRBenchmark
import yaml

# Load config
with open("configs/my-custom-pipeline.yaml") as f:
    config = yaml.safe_load(f)

# Build extraction
corpus = Corpus(Path("corpora/funsd_benchmark"))
snapshot = build_extraction_snapshot(
    corpus,
    extractor_id=config["extractor_id"],
    configuration_name="my-custom-pipeline",
    configuration=config["config"]
)

# Evaluate
benchmark = OCRBenchmark(corpus)
report = benchmark.evaluate_extraction(
    snapshot_reference=snapshot.snapshot_id,
    pipeline_config=config
)

# Print results
report.print_summary()
  1. Add to benchmark suite:

Edit scripts/benchmark_all_pipelines.py:

PIPELINE_CONFIGS = [
    "configs/baseline-ocr.yaml",
    "configs/ocr-paddleocr.yaml",
    # ... existing configs ...
    "configs/my-custom-pipeline.yaml",  # Add yours
]

Benchmarking Custom Datasets

Prepare Your Dataset

  1. Create corpus directory:
mkdir -p my_corpus/metadata/ground_truth
  1. Add documents:
# Copy your images/PDFs
cp /path/to/documents/* my_corpus/
  1. Create ground truth files:

For each document, create a text file with the expected extracted text:

# my_corpus/metadata/ground_truth/<document-id>.txt
echo "Expected text from document" > my_corpus/metadata/ground_truth/doc1.txt
  1. Ingest into corpus:
from pathlib import Path
from biblicus import Corpus

corpus = Corpus.init(Path("my_corpus"))

for doc_path in Path("my_corpus").glob("*.png"):
    if not doc_path.stem.startswith("."):
        corpus.ingest_file(doc_path, tags=["benchmark"])

Run Benchmark

python scripts/benchmark_all_pipelines.py \
  --corpus-path my_corpus \
  --ground-truth-dir my_corpus/metadata/ground_truth \
  --config configs/benchmark/quick.yaml \
  --output results/my_corpus_benchmark.json

Troubleshooting

Common Issues

Error: "FUNSD dataset not found"

# Solution: Download the dataset first
python scripts/download_funsd_samples.py

Error: "Pipeline failed: paddleocr not installed"

# Solution: Install the required extra
pip install "biblicus[paddleocr]"

Error: "Ground truth file not found"

# Solution: Ensure ground truth files match document IDs
ls corpora/funsd_benchmark/metadata/funsd_ground_truth/

Slow benchmark performance:

# Solution: Use quick config for development
python scripts/benchmark_all_pipelines.py \
  --config configs/benchmark/quick.yaml  # Not standard or full

Debugging Failed Pipelines

Check the detailed error:

import json

with open("results/quick_benchmark.json") as f:
    results = json.load(f)

for pipeline in results["pipelines"]:
    if not pipeline["success"]:
        print(f"Failed: {pipeline['name']}")
        print(f"Error: {pipeline.get('error', 'Unknown error')}")

Memory Issues

If benchmarking runs out of memory:

  1. Reduce batch size in benchmark config
  2. Use quick config with fewer documents
  3. Test one pipeline at a time
# Instead of all pipelines:
python scripts/evaluate_ocr_pipeline.py \
  --corpus corpora/funsd_benchmark \
  --config configs/ocr-paddleocr.yaml

Next Steps

Now that you've run your first benchmark:

  1. Understand metrics - Learn what each metric means
  2. Explore pipelines - See all available pipelines and their trade-offs
  3. View current results - Compare your results to latest benchmarks
  4. Deep dive: OCR Benchmarking - Comprehensive guide to OCR evaluation
  5. Deep dive: Multi-Category Framework - Architecture and design

Benchmark Modes Summary

Mode Duration Documents Use Case Config
Quick 5-10 min 20 forms Development iteration configs/benchmark/quick.yaml
Standard 30-60 min 50 forms Release validation configs/benchmark/standard.yaml
Full 2-4 hours All (199 forms) Research/publication configs/benchmark/full.yaml

CLI Reference

Benchmark All Pipelines

python scripts/benchmark_all_pipelines.py \
  --corpus-path CORPUS_PATH \
  --config CONFIG_FILE \
  --output OUTPUT_FILE

Evaluate Single Pipeline

python scripts/evaluate_ocr_pipeline.py \
  --corpus CORPUS_PATH \
  --config PIPELINE_CONFIG \
  --output OUTPUT_FILE

Compare Two Pipelines

python scripts/benchmark_heron_vs_paddleocr.py \
  --corpus CORPUS_PATH \
  --output OUTPUT_FILE

Quick Layout-Aware Test

python scripts/quick_benchmark_layout_aware.py

Resources