Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

Metrics Reference

Complete guide to understanding evaluation metrics used in Biblicus benchmarks.

Overview

Biblicus uses three categories of metrics to evaluate extraction quality:

  1. Set-Based Metrics: Measure word-finding ability (position-agnostic)
  2. Order-Aware Metrics: Measure reading order preservation (sequence quality)
  3. N-gram Overlap: Measure local ordering quality (word pair/triple accuracy)

Each metric tells you something different about extraction quality. A good extraction pipeline needs:

Set-Based Metrics (Position-Agnostic)

Set-based metrics measure how well the extraction finds words, regardless of their order. These are the primary accuracy metrics.

F1 Score

Harmonic mean of precision and recall.

Formula:

F1 = 2 × (Precision × Recall) / (Precision + Recall)

Interpretation:

Example:

When to prioritize:


Precision

Percentage of extracted words that are correct.

Formula:

Precision = TP / (TP + FP)

Where:
  TP = True Positives (correct words found)
  FP = False Positives (incorrect words extracted)

Interpretation:

Example:

When to prioritize:


Recall

Percentage of ground truth words that were found.

Formula:

Recall = TP / (TP + FN)

Where:
  TP = True Positives (correct words found)
  FN = False Negatives (ground truth words missed)

Interpretation:

Example:

When to prioritize:


Character Accuracy

Character-level correctness metric.

When to use:


Order-Aware Metrics (Sequence Quality)

Order-aware metrics measure whether words appear in the correct sequence. These are critical for layout-aware OCR evaluation.

Word Error Rate (WER)

Edit distance normalized by ground truth length.

Formula:

WER = (Insertions + Deletions + Substitutions) / Total_Ground_Truth_Words

Interpretation:

Example:

Why WER can exceed 1.0: If you have 100 ground truth words but extract 200 words (100 correct + 100 insertions), WER = 100/100 = 1.0. More insertions push it higher.

When to prioritize:


Sequence Accuracy

Percentage of words in correct sequential position.

Formula:

Sequence Accuracy = Correct_Position_Words / Total_Ground_Truth_Words

Example:

Interpretation:

When to use:


LCS Ratio (Longest Common Subsequence)

Ratio of longest ordered subsequence to total.

Formula:

LCS Ratio = Length(LCS(ground_truth, extracted)) / Length(ground_truth)

Example:

Interpretation:

When to use:


N-gram Overlap (Local Ordering)

N-gram metrics measure whether adjacent words (bigrams) or word triples (trigrams) appear in the correct order.

Bigram Overlap

Percentage of word pairs in correct order.

Formula:

Bigram Overlap = Matching_Bigrams / Total_Bigrams

Where a bigram is a pair of adjacent words: ("word1", "word2")

Example:

Interpretation:

When to prioritize:


Trigram Overlap

Percentage of word triples in correct order.

Formula:

Trigram Overlap = Matching_Trigrams / Total_Trigrams

Where a trigram is three adjacent words: ("word1", "word2", "word3")

Example:

Interpretation:

When to use:


Metric Trade-offs

Precision vs. Recall

Different pipelines optimize for different trade-offs:

High Precision, Lower Recall (Conservative):

Lower Precision, High Recall (Aggressive):

Balanced:

Accuracy vs. Reading Order

You can have high F1 (finds words) but poor WER (wrong order):

Good F1, Poor WER:

Good F1, Good WER:

Poor F1, Good WER:


Choosing Metrics for Your Use Case

Forms (FUNSD-like)

Primary Metric: F1 Score

Secondary Metrics:

Target:


Receipts (Dense Text)

Primary Metric: F1 Score

Secondary Metrics:

Target:


Academic Papers (Multi-Column)

Primary Metric: LCS Ratio

Secondary Metrics:

Target:


Legal/Compliance Documents

Primary Metric: Recall

Secondary Metrics:

Target:


Interpreting Benchmark Results

Example Report

{
  "pipeline_name": "paddleocr",
  "metrics": {
    "set_based": {
      "avg_precision": 0.792,
      "avg_recall": 0.782,
      "avg_f1": 0.787
    },
    "order_aware": {
      "avg_wer": 0.533,
      "avg_sequence_accuracy": 0.031,
      "avg_lcs_ratio": 0.621
    },
    "ngram": {
      "avg_bigram_overlap": 0.521,
      "avg_trigram_overlap": 0.412
    }
  }
}

Interpretation

Set-Based Metrics:

Order-Aware Metrics:

N-gram Metrics:

Overall: Strong all-around pipeline. High F1 for accuracy, acceptable reading order for forms.


Comparing Two Pipelines

Scenario: PaddleOCR vs. Heron+Tesseract

Metric PaddleOCR Heron+Tesseract Winner
F1 0.787 0.519 PaddleOCR
Recall 0.782 0.810 Heron
Precision 0.792 0.384 PaddleOCR
WER 0.533 0.612 PaddleOCR
Bigram 0.521 0.561 Heron

Analysis:

Choose PaddleOCR if: You need clean, accurate extraction Choose Heron if: You need maximum extraction, can tolerate noise


Metric Calculation Details

Set-Based Calculation

Words are normalized before comparison:

Example:

Order-Aware Calculation

Word sequences are compared position-by-position:

Example:

N-gram Calculation

Sliding window over word sequences:

Example bigram calculation:


Tools and Libraries

Biblicus uses:

Without editdistance, WER calculation falls back to difflib (slower but works).


Next Steps


References