Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

Biblicus Document Understanding Benchmark

Multi-category benchmark framework architecture and design documentation.

Looking for practical instructions? See the Quickstart Guide or OCR Benchmarking Guide. This document covers the architectural design of the multi-category framework.

The Biblicus Document Understanding Benchmark evaluates OCR and document extraction pipelines across diverse document types. Rather than testing on a single dataset, the benchmark measures performance across three distinct categories—forms, academic papers, and receipts—each presenting unique challenges for document processing systems.

Related Documentation:

Why a Multi-Category Benchmark?

Document extraction pipelines often excel at one document type while struggling with others. A pipeline optimized for clean academic PDFs may fail on noisy scanned forms. A receipt parser tuned for dense text may miss content in multi-column layouts.

The Biblicus benchmark reveals these trade-offs by testing pipelines across:

Category Dataset Documents Challenge
Forms FUNSD 199 Noise, handwriting, field extraction
Academic Scanned ArXiv 100+ Multi-column layout, reading order
Receipts SROIE 626 Dense text, entity extraction

Quick Start

# Download benchmark datasets
biblicus benchmark download --datasets funsd,sroie,scanned-arxiv

# Run quick benchmark (~5-10 minutes)
biblicus benchmark run --config configs/benchmark/quick.yaml

# Run standard benchmark (~30-60 minutes)
biblicus benchmark run

# Generate markdown report
biblicus benchmark report --output docs/guides/benchmark-results.md

Document Categories

Forms (FUNSD)

Dataset: Form Understanding in Noisy Scanned Documents (FUNSD)

FUNSD contains 199 real scanned forms from the 1980s-1990s with word-level ground truth annotations. These documents test a pipeline's ability to handle:

Primary Metric: F1 Score (balanced word finding)

Source: https://guillaumejaume.github.io/FUNSD/

Academic Papers (Scanned ArXiv)

Dataset: Scanned ArXiv Papers

Academic papers rendered as images (not born-digital PDFs) test layout-aware extraction:

Primary Metric: LCS Ratio (Longest Common Subsequence ratio measuring reading order preservation)

Source: HuggingFace IAMJB/scanned-arxiv-papers

Receipts (SROIE)

Dataset: Scanned Receipts OCR and Information Extraction (SROIE)

626 receipt images from ICDAR 2019 with both OCR text and structured entity annotations:

Primary Metrics: F1 Score + Entity F1 (per-entity-type accuracy)

Source: ICDAR 2019 Competition (https://rrc.cvc.uab.es/?ch=13)

Metrics

The benchmark uses three categories of metrics to evaluate extraction quality. For complete details on each metric including formulas, interpretations, and use case recommendations, see the Metrics Reference.

Quick summary:

Set-Based Metrics (Word Finding):

Order-Aware Metrics (Sequence Quality):

Entity Metrics (Semantic Extraction):

Scoring Strategy

Per-Category Scores (Primary)

Each category reports its primary metric independently:

This allows comparing pipelines within each category without conflating different document types.

Weighted Aggregate (Optional)

For quick overall comparison, an optional weighted aggregate combines category scores:

Aggregate = 0.40 × Forms F1 + 0.35 × Academic LCS + 0.25 × Receipts F1

Weights are configurable in benchmark configuration files. Adjust based on your document mix.

Running Benchmarks

Benchmark Modes

Mode Forms Academic Receipts Runtime
quick 20 docs 20 docs 50 docs ~5-10 min
standard 50 docs 100 docs 100 docs ~30-60 min
full 199 docs All 626 docs ~2-4 hours

Command Examples

# Quick benchmark for development iteration
biblicus benchmark run --config configs/benchmark/quick.yaml

# Standard benchmark (default)
biblicus benchmark run

# Full benchmark for release validation
biblicus benchmark run --config configs/benchmark/full.yaml

# Single category
biblicus benchmark run --category forms

# Specific pipelines
biblicus benchmark run --pipelines paddleocr,heron-tesseract,baseline-ocr

# Check dataset status
biblicus benchmark status

Configuration Files

Benchmark configurations in configs/benchmark/:

# configs/benchmark/quick.yaml
benchmark_name: quick
categories:
  forms:
    dataset: funsd
    subset_size: 20
    primary_metric: f1_score
  academic:
    dataset: scanned-arxiv
    subset_size: 20
    primary_metric: lcs_ratio
  receipts:
    dataset: sroie
    subset_size: 50
    primary_metric: f1_score
pipelines:
  - configs/baseline-ocr.yaml
  - configs/ocr-paddleocr.yaml
  - configs/heron-tesseract.yaml
aggregate_weights:
  forms: 0.40
  academic: 0.35
  receipts: 0.25

Understanding Results

JSON Output Structure

{
  "benchmark_version": "1.0.0",
  "timestamp": "2026-02-05T12:00:00Z",
  "categories": {
    "forms": {
      "dataset": "funsd",
      "documents_evaluated": 50,
      "pipelines": [
        {
          "name": "paddleocr",
          "metrics": {
            "f1": 0.787,
            "recall": 0.782,
            "precision": 0.792,
            "wer": 0.533
          }
        }
      ],
      "best_pipeline": "paddleocr"
    },
    "academic": {
      "dataset": "scanned-arxiv",
      "pipelines": [...],
      "best_pipeline": "heron-tesseract"
    },
    "receipts": {
      "dataset": "sroie",
      "pipelines": [...],
      "best_pipeline": "paddleocr"
    }
  },
  "aggregate": {
    "weighted_score": 0.72,
    "weights": {"forms": 0.40, "academic": 0.35, "receipts": 0.25}
  },
  "recommendations": {
    "best_overall": "paddleocr",
    "best_for_layout": "heron-tesseract",
    "best_for_speed": "rapidocr"
  }
}

Interpreting Trade-offs

High Recall, Lower Precision (e.g., Heron + Tesseract)

High Precision, Lower Recall (e.g., Docling-Smol)

High LCS Ratio (e.g., Layout-Aware Pipelines)

Pipeline Recommendations

Based on benchmark results:

Use Case Recommended Pipeline Why
General accuracy PaddleOCR Highest F1 across document types
Multi-column documents Heron + Tesseract Best reading order preservation
Receipts/forms PaddleOCR Strong entity extraction
Speed priority RapidOCR Fastest inference, acceptable accuracy
Completeness critical Heron + Tesseract Highest recall (0.810)
Low noise tolerance Docling-Smol Highest precision

Adding Custom Pipelines

To benchmark your own pipeline:

  1. Create a pipeline configuration in configs/:
# configs/my-custom-pipeline.yaml
extractor_id: pipeline
config:
  stages:
    - extractor_id: my-layout-detector
      config:
        threshold: 0.7
    - extractor_id: ocr-tesseract
      config:
        use_layout_metadata: true
  1. Run benchmark with your pipeline:
biblicus benchmark run --pipelines my-custom-pipeline
  1. Compare results:
biblicus benchmark report --input results/benchmark_*.json

Dataset Downloads

Automatic Download

# Download all datasets
biblicus benchmark download --datasets funsd,sroie,scanned-arxiv

# Download specific dataset
biblicus benchmark download --datasets sroie

Manual Download

If automatic download fails:

FUNSD:

  1. Visit https://guillaumejaume.github.io/FUNSD/
  2. Download dataset.zip
  3. Extract to corpora/funsd_benchmark/

SROIE:

  1. Register at https://rrc.cvc.uab.es/?ch=13
  2. Download task files
  3. Run python scripts/download_sroie_samples.py --from-local /path/to/sroie

Scanned ArXiv:

from datasets import load_dataset
dataset = load_dataset("IAMJB/scanned-arxiv-papers")

Licensing

Dataset License Usage
FUNSD CC BY-NC-SA 4.0 Non-commercial research
SROIE Research only ICDAR competition terms
Scanned ArXiv Varies Check per-paper license

See Also