Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

Benchmarking Overview

Biblicus is designed as a retrieval augmented generation platform where you can experiment with different extraction pipelines, retrieval backends, and configurations—then benchmark them against each other to find the best approach for your use case.

Why Benchmark?

Different documents require different approaches:

Biblicus lets you:

  1. Compare extraction pipelines (Tesseract, PaddleOCR, Docling VLMs, etc.)
  2. Evaluate retrieval backends (scan, SQLite FTS, TF-vector)
  3. Measure with comprehensive metrics (F1 score, WER, sequence accuracy, n-gram overlap)
  4. Reproduce results with snapshot-based evaluation

Benchmarking as a Platform

Rather than providing a single "best" configuration, Biblicus provides:

You can:

Current Benchmarks

Document Extraction (OCR) Benchmarks

Biblicus includes multi-category benchmarks for document extraction:

1. Forms (FUNSD Dataset)

2. Receipts (SROIE Dataset)

3. Academic Papers (Scanned ArXiv) (dataset pending)

Speech-to-Text (STT) Benchmarks

Biblicus includes STT provider benchmarks:

1. LibriSpeech test-clean

Evaluated Pipelines

We benchmark 8+ extraction pipelines including:

See the Pipeline Catalog for detailed descriptions and configurations.

Getting Started

Quick Start (5-10 minutes)

Run a quick benchmark on a subset of documents:

python scripts/benchmark_all_pipelines.py \
  --corpus-path corpora/funsd \
  --config configs/benchmark/quick.yaml \
  --output results/quick_benchmark.json

See Quickstart Guide for step-by-step instructions.

Understanding Metrics

Biblicus measures extraction quality using three categories of metrics:

See Metrics Reference for detailed explanations.

Current Results

For the latest benchmark results and recommendations by use case:

→ Current Benchmark Results

Documentation Structure

This documentation follows a hub-and-spoke model:

Core Guides:

Deep Dives:

Benchmark Modes

Biblicus supports three benchmark modes to balance speed vs. thoroughness:

OCR Benchmarks

Mode Duration Use Case Configuration
Quick 5-10 min Development iteration configs/benchmark/quick.yaml
Standard 30-60 min Release validation configs/benchmark/standard.yaml
Full 2-4 hours Comprehensive evaluation configs/benchmark/full.yaml

STT Benchmarks

Mode Duration Audio Files Use Case Configuration
Quick 2-5 min 20 Development iteration configs/benchmark/stt-quick.yaml
Standard 10-20 min 100 Release validation configs/benchmark/stt-standard.yaml
Full 2-4 hours ~2600 Comprehensive evaluation configs/benchmark/stt-full.yaml

Customization

Adding Custom Pipelines

You can add your own extraction pipelines to benchmark:

  1. Create a pipeline configuration in configs/
  2. Add it to the benchmark runner
  3. Run the benchmark to compare against existing pipelines

See Pipeline Catalog for examples.

Adding Custom Benchmarks

The benchmark framework is extensible:

  1. Create a new CategoryConfig with your dataset
  2. Define the primary metric for your document type
  3. Add it to the BenchmarkRunner

See Multi-Category Benchmark Framework for details.

Architecture

Biblicus uses a three-tier benchmarking system:

Tier 1: Individual Document Evaluation

Tier 2: Multi-Category Orchestration

Tier 3: User-Facing Scripts

See Multi-Category Benchmark Framework for architectural details.

Next Steps

  1. Run your first benchmark - 5-minute quickstart
  2. Explore pipelines - See what's available
  3. Understand metrics - Learn how quality is measured
  4. Review current results - See how pipelines compare

Contributing

To add new benchmarks or pipelines:

  1. Follow the existing patterns in src/biblicus/evaluation/
  2. Add documentation to the appropriate guide
  3. Update this overview to link to your additions
  4. Submit a pull request

For questions or issues, see the main repository.