Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

Heron Layout Detection Implementation - COMPLETE

Date: 2026-02-03 Status: โœ… Fully Implemented and Benchmarked

Summary

Successfully implemented IBM Research's Heron layout detection models - the ACTUAL tool was described in their OCR workflow. The complete pipeline (Heron + Tesseract) has been implemented, tested, and benchmarked.


What is Heron?

Models:


Implementation

Files Created/Modified

New Files:

  1. src/biblicus/extractors/heron_layout.py - Heron layout detection extractor
  2. configs/heron-tesseract.yaml - Pipeline configuration
  3. scripts/test_heron_pipeline.py - Test script
  4. scripts/benchmark_heron_vs_paddleocr.py - Benchmark script
  5. results/heron_tesseract.json - Benchmark results
  6. results/heron_vs_paddleocr_comparison.json - Comparison
  7. docs/guides/ocr-benchmarking.md - Complete benchmarking guide

Modified Files:

  1. src/biblicus/extractors/__init__.py - Registered HeronLayoutExtractor
  2. benchmark-results.md - Added Heron results
  3. GITHUB_ISSUES.md - Updated Issue #4 with completion status

workflow-based Complete Workflow Status

Part 1: Docling for Tables/Formulas โœ… WORKING

Part 2: Gibberish Filtering ๐Ÿ“‹ DOCUMENTED

Part 3: Heron + Tesseract โœ… WORKING


Benchmark Results

Tested on 5 FUNSD scanned form documents with ground truth annotations.

Heron-101 + Tesseract Performance

Metric Score Comparison
F1 Score 0.519 -13.7% vs PaddleOCR layout
Recall 0.810 +10.6% vs PaddleOCR layout โญ HIGHEST
Precision 0.384 -26.6% vs PaddleOCR layout
Word Error Rate 5.324 +74.2% vs PaddleOCR layout
Sequence Accuracy 0.012 -71.9% vs PaddleOCR layout
Bigram Overlap 0.561 +14.4% vs PaddleOCR layout โญ BEST

Comparison: All Layout-Aware Approaches

Pipeline F1 Recall Precision Bigram
Heron + Tesseract 0.519 0.810 0.384 0.561
PaddleOCR + Tesseract 0.601 0.732 0.523 0.491
Baseline Tesseract 0.607 0.599 0.626 0.350

Key Findings

Heron's Strengths

โœ… Highest Recall (0.810)

โœ… Best Local Ordering (Bigram: 0.561)

โœ… Aggressive Layout Detection

Heron's Trade-offs

โš ๏ธ Lower Precision (0.384)

โš ๏ธ Higher Word Error Rate (5.324)

โš ๏ธ Lower F1 Score (0.519)


When to Use Heron

โœ… Use Heron When:

โš ๏ธ Use PaddleOCR When:

โœ… Use Direct PaddleOCR When:


Architecture

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚                 Heron + Tesseract Pipeline              โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

Step 1: Heron Layout Detection
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ IBM Heron-101 (RT-DETR V2)           โ”‚
โ”‚ - Input: Document image              โ”‚
โ”‚ - Output: 24 regions with:           โ”‚
โ”‚   * Bounding boxes [x1,y1,x2,y2]     โ”‚
โ”‚   * Region types (17 classes)        โ”‚
โ”‚   * Confidence scores                โ”‚
โ”‚   * Reading order                    โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
            โ†“
    Layout Metadata
            โ†“
Step 2: Region-Based OCR
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ Tesseract OCR                        โ”‚
โ”‚ - For each region in order:          โ”‚
โ”‚   * Crop image to bbox               โ”‚
โ”‚   * Run OCR on region                โ”‚
โ”‚   * Collect text                     โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
            โ†“
Step 3: Text Reconstruction
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ Combine in Reading Order             โ”‚
โ”‚ - Join text from all regions         โ”‚
โ”‚ - Preserve layout-detected order     โ”‚
โ”‚ - Output: Complete document text     โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

Usage

Test Pipeline

python scripts/test_heron_pipeline.py

Expected output:

โœ“ Heron layout detection stage exists
โœ“ Found 5 layout metadata files
โœ“ Tesseract OCR stage exists
โœ“ Found 5 text output files
SUCCESS: Heron + Tesseract pipeline is working!

Benchmark

python scripts/benchmark_heron_vs_paddleocr.py

Use in Pipeline

# configs/my-heron-pipeline.yaml
extractor_id: pipeline
config:
  stages:
    - extractor_id: heron-layout
      config:
        model_variant: "101"  # or "base" for faster/lighter
        confidence_threshold: 0.6
    - extractor_id: ocr-tesseract
      config:
        use_layout_metadata: true
        lang: eng
from biblicus import Corpus
from biblicus.extraction import build_extraction_snapshot
from pathlib import Path

corpus = Corpus(Path("my_corpus").resolve())
snapshot = build_extraction_snapshot(
    corpus,
    extractor_id="pipeline",
    configuration_name="heron-tesseract",
    configuration=config["config"]
)

Dependencies

# Install Heron dependencies
pip install "transformers>=4.40.0" "torch>=2.0.0"

# First run downloads model (~150MB)
python scripts/test_heron_pipeline.py

Model download locations:


Comparison with Original Research

From the Heron paper (arXiv:2509.11720):

Our implementation:


Future Work

Multi-Column Documents

Current benchmarks use FUNSD (single-column forms). Layout detection should show even larger benefits on:

Gibberish Filtering

Implement workflow-based Part 2 to clean up false positives from aggressive layout detection. This would improve Heron's precision while maintaining high recall.

Heron-base Benchmark

Test the lighter Heron-base model (42.9M params) for speed/accuracy trade-off.


References


Conclusion

โœ… workflow-based Heron + Tesseract workflow is fully implemented and working

The implementation successfully replicates workflow-based production workflow for handling non-selectable files with complex layouts. Heron achieves the highest recall of any tested method (0.810), making it ideal for use cases where finding all text is more important than perfect accuracy.

The trade-off between recall and precision is well-understood and documented. Combined with gibberish filtering (Part 2 of two-stage layout-aware workflow), this approach provides a robust solution for challenging OCR tasks.

Bottom line: If you need to find ALL the text and can tolerate some noise, use Heron. If you need the best overall accuracy, use direct PaddleOCR (F1: 0.787).