Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

STT Provider Benchmarking Guide

Comprehensive guide for benchmarking Speech-to-Text providers using Biblicus with real-world results.

New to benchmarking? Start with the Benchmarking Overview for a platform introduction.

Table of Contents

  1. Overview
  2. Benchmark Results Summary
  3. Supported STT Providers
  4. Quick Start
  5. Available Datasets
  6. Understanding Metrics
  7. Running Custom Benchmarks
  8. Best Practices
  9. Troubleshooting

Overview

Biblicus provides a robust framework for evaluating and comparing Speech-to-Text providers using standardized metrics on labeled audio datasets. This guide provides practical instructions and real benchmark results.

Key Features:


Benchmark Results Summary

120-Sample LibriSpeech test-clean Evaluation

Comprehensive benchmark results from 120 professionally recorded audiobook samples:

Provider WER CER F1 Score Speed Cost Verdict
AWS Transcribe 3.57% 1.01% 0.963 ~12s/file $$ ๐Ÿฅ‡ Best accuracy, slower
Aldea 3.60% 1.27% 0.962 ~1.5s/file $ ๐Ÿฅˆ Excellent balance
Deepgram Nova-3 3.76% 1.15% 0.962 ~1s/file $$ ๐Ÿฅ‰ Fastest, great accuracy
OpenAI Whisper 4.30% 1.32% 0.964 ~1.5s/file $ โœ… Good, widely available
Faster-Whisper 4.33% 1.29% 0.964 Local Free โœ… Best for offline/free
GPT-4o Audio 45.11% 31.75% 0.847 ~1.3s/file $$$ โŒ Not suitable for STT

Key Findings:

Recommendation by Use Case:


Supported STT Providers

Currently Integrated (6 providers)

1. AWS Transcribe

2. Aldea

3. Deepgram Nova-3

4. OpenAI Whisper (API)

5. Faster-Whisper (Local)


Quick Start

1. Download Benchmark Dataset

# Download 100 LibriSpeech test-clean samples
python scripts/download_librispeech_samples.py \
  --corpus corpora/librispeech_benchmark \
  --count 100

# For larger benchmarks (500 samples for statistical significance)
python scripts/download_librispeech_samples.py \
  --corpus corpora/librispeech_benchmark \
  --count 500 \
  --force

# Download challenging test-other subset
python scripts/download_openslr_samples.py \
  --corpus corpora/librispeech_test_other \
  --dataset SLR12 \
  --subset test-other \
  --count 100

# 3. View results
cat results/stt_benchmark.json | jq '.providers[] | {name, wer: .metrics.wer.avg}'

Benchmark Dataset

LibriSpeech Test-Clean

What is LibriSpeech?

Download:

python scripts/download_librispeech_samples.py \
  --corpus corpora/librispeech_benchmark \
  --count 100

This will:

  1. Download LibriSpeech test-clean (~346 MB)
  2. Extract FLAC audio files
  3. Parse ground truth transcriptions from .trans.txt files
  4. Create corpus at corpora/librispeech_benchmark/
  5. Store ground truth in metadata/ground_truth/

Dataset structure:

corpora/librispeech_benchmark/
โ”œโ”€โ”€ metadata/
โ”‚   โ”œโ”€โ”€ config.json
โ”‚   โ”œโ”€โ”€ catalog.json
โ”‚   โ””โ”€โ”€ ground_truth/
โ”‚       โ”œโ”€โ”€ <audio-id>.txt        # Ground truth transcription
โ”‚       โ””โ”€โ”€ ...
โ”œโ”€โ”€ <audio-id>.flac                # Audio file
โ””โ”€โ”€ ...

Using Your Own Dataset

To benchmark on custom audio:

  1. Prepare ground truth files:

    corpus_dir/metadata/ground_truth/
    โ”œโ”€โ”€ <audio-id>.txt
    โ””โ”€โ”€ ...
    
  2. Ingest audio:

    from biblicus import Corpus
    from pathlib import Path
    
    corpus = Corpus(Path("my_corpus"))
    corpus.ingest_file("audio.flac", tags=["benchmark", "speech"])
    
  3. Run evaluation:

    from biblicus.evaluation.stt_benchmark import STTBenchmark
    
    benchmark = STTBenchmark(corpus)
    report = benchmark.evaluate_extraction(
        snapshot_reference="<snapshot-id>",
        ground_truth_dir=corpus.root / "metadata" / "ground_truth"
    )
    

Available STT Providers

Biblicus includes 3 STT provider integrations:

Provider Model Strengths API Required
OpenAI Whisper whisper-1 General-purpose, multilingual OPENAI_API_KEY
Deepgram Nova-3 nova-3 Fast, accurate, features DEEPGRAM_API_KEY
Aldea (default) Custom STT service ALDEA_API_KEY

Understanding Metrics

Biblicus STT benchmarks use three categories of metrics:

Word Error Rate (WER)

Primary metric for STT accuracy.

WER = (Substitutions + Deletions + Insertions) / Total Words

Interpretation:

Character Error Rate (CER)

Character-level accuracy metric.

CER = (Character Substitutions + Deletions + Insertions) / Total Characters

Word-Level Metrics

Precision, Recall, F1 Score (bag-of-words)

These ignore word order and focus on vocabulary accuracy.

For detailed explanations, see the Metrics Reference.


Running Benchmarks

Benchmark All Providers

python scripts/benchmark_all_stt_providers.py \
  --corpus corpora/librispeech_benchmark \
  --output results/stt_comparison.json

What it does:

  1. Tests all 3 STT providers (OpenAI, Deepgram, Aldea)
  2. Transcribes all audio files in the corpus
  3. Calculates WER, CER, and word-level metrics
  4. Generates comprehensive comparison report

Output:

Example output:

================================================================================
COMPREHENSIVE STT PROVIDER COMPARISON
================================================================================

Provider                       WER      CER      Precision  Recall     F1       Status
------------------------------------------------------------------------------------------------
OpenAI Whisper                0.045    0.023    0.982      0.975      0.978    โœ“ OK
Deepgram Nova-3               0.052    0.028    0.976      0.968      0.972    โœ“ OK
Aldea                         0.068    0.035    0.965      0.952      0.958    โœ“ OK
------------------------------------------------------------------------------------------------

๐Ÿ† Best Word Accuracy (Lowest WER): OpenAI Whisper
   WER: 0.045, Median: 0.042

๐Ÿ† Best Character Accuracy (Lowest CER): OpenAI Whisper
   CER: 0.023, Median: 0.021

๐Ÿ† Best Word Finding (F1): OpenAI Whisper
   F1: 0.978, Precision: 0.982, Recall: 0.975

Benchmark Specific Providers

# Test only OpenAI and Deepgram
python scripts/benchmark_all_stt_providers.py \
  --corpus corpora/librispeech_benchmark \
  --providers stt-openai stt-deepgram \
  --output results/openai_vs_deepgram.json

Quick Test with Fewer Files

python scripts/benchmark_all_stt_providers.py \
  --corpus corpora/librispeech_benchmark \
  --limit 5 \
  --output results/quick_test.json

Results Analysis

JSON Output Structure

{
  "benchmark_timestamp": "2026-02-13T20:00:00",
  "corpus_path": "corpora/librispeech_benchmark",
  "total_providers": 3,
  "successful_providers": 3,
  "failed_providers": 0,
  "providers": [
    {
      "name": "OpenAI Whisper",
      "snapshot_id": "abc123...",
      "success": true,
      "metrics": {
        "wer": {
          "avg": 0.045,
          "median": 0.042,
          "avg_substitutions": 2.3,
          "avg_deletions": 0.8,
          "avg_insertions": 0.5
        },
        "cer": {
          "avg": 0.023,
          "median": 0.021
        },
        "word_level": {
          "avg_precision": 0.982,
          "avg_recall": 0.975,
          "avg_f1": 0.978,
          "median_f1": 0.980
        }
      },
      "provider_configuration": { ... },
      "total_audio_files": 100
    }
  ],
  "best_performers": {
    "lowest_wer": "OpenAI Whisper",
    "lowest_cer": "OpenAI Whisper",
    "best_f1": "OpenAI Whisper"
  }
}

Analyzing Results

Find best provider:

cat results/stt_comparison.json | jq '.providers | sort_by(.metrics.wer.avg) | .[0] | {name, wer: .metrics.wer.avg}'

Compare WER across providers:

cat results/stt_comparison.json | jq '.providers[] | {name, wer: .metrics.wer.avg, cer: .metrics.cer.avg}' | jq -s 'sort_by(.wer)'

Find audio files with high WER:

import json

with open('results/stt_comparison.json') as f:
    data = json.load(f)

# Assuming per-audio results are stored (implementation detail)
# You can analyze which utterances have highest error rates

Dependencies

Installing STT Provider Dependencies

Different providers require different dependencies:

OpenAI Whisper:

pip install "biblicus[openai]"
export OPENAI_API_KEY="sk-..."

Deepgram:

pip install "biblicus[deepgram]"
export DEEPGRAM_API_KEY="..."

Aldea:

pip install "biblicus[aldea]"
export ALDEA_API_KEY="..."

All providers:

pip install "biblicus[openai,deepgram,aldea]"

Checking Dependencies

# Test OpenAI
python -c "from openai import OpenAI; print('OpenAI OK')"

# Test Deepgram
python -c "from deepgram import DeepgramClient; print('Deepgram OK')"

# Test Aldea
python -c "import httpx; print('Aldea OK')"

Troubleshooting

Common Issues

Issue: "API key not found"

Solution: Set environment variable or configure in ~/.biblicus/config.yml:
  openai:
    api_key: "sk-..."
  deepgram:
    api_key: "..."
  aldea:
    api_key: "..."

Issue: "Ground truth directory not found"

Solution: Run python scripts/download_librispeech_samples.py first

Issue: "Audio format not supported"

Solution: LibriSpeech uses FLAC format. Ensure audio files are valid FLAC.

Issue: "API rate limit exceeded"

Solution:
- Reduce --limit parameter
- Add delays between API calls
- Use provider with higher rate limits

Issue: "Results don't match expected WER"

Solution: Check:
- Correct ground truth files loaded
- Audio quality is good
- Provider configuration is appropriate
- Punctuation/formatting settings

Issue: "High API costs"

Solution:
- Start with --limit 5 or --limit 10 for testing
- Use quick.yaml config (20 files) instead of full (2600 files)
- Calculate costs: ~$0.006 per minute of audio (varies by provider)

Benchmark Modes

Use configuration files for different benchmark scales:

Quick (20 audio files, ~2-5 minutes):

# Manually specify count
python scripts/download_librispeech_samples.py --corpus corpora/librispeech_benchmark --count 20
python scripts/benchmark_all_stt_providers.py --corpus corpora/librispeech_benchmark --output results/quick.json

Standard (100 audio files, ~10-20 minutes):

python scripts/download_librispeech_samples.py --corpus corpora/librispeech_benchmark --count 100
python scripts/benchmark_all_stt_providers.py --corpus corpora/librispeech_benchmark --output results/standard.json

Full (all audio files, ~2-4 hours):

# Download full test-clean dataset
python scripts/download_librispeech_samples.py --corpus corpora/librispeech_benchmark --count 2600
python scripts/benchmark_all_stt_providers.py --corpus corpora/librispeech_benchmark --output results/full.json

Cost Estimation

STT benchmarking involves API costs:

Provider Cost per Hour 20 Files (~10 min) 100 Files (~50 min) Full (~5.4 hours)
OpenAI Whisper $0.36/hr ~$0.06 ~$0.30 ~$1.94
Deepgram Nova-3 $0.36/hr ~$0.06 ~$0.30 ~$1.94
Aldea (varies) (varies) (varies) (varies)

Cost-saving tips:


Additional Datasets

LibriSpeech test-clean is the only dataset that has been benchmarked end to end. The repository ships download scripts for other speech corpora, with the following status:

Dataset Script Status
LibriSpeech test-clean scripts/download_librispeech_samples.py Working, 500 samples ingested and tested
LibriSpeech test-other and other OpenSLR sets scripts/download_openslr_samples.py Working
Mozilla Common Voice scripts/download_common_voice_samples.py Requires a manual download from Mozilla
Google FLEURS (multilingual) scripts/download_fleurs_samples.py Requires the datasets library
TED-LIUM scripts/download_tedlium_samples.py Upstream download URL is broken
VoxForge scripts/download_voxforge_samples.py Upstream download URL is broken
CMU AN4 scripts/download_an4_samples.py Upstream download URL is broken

Provider Configuration Notes

Accuracy tiers on LibriSpeech test-clean (120 samples):


Planned Improvements


See Also

Benchmarking Documentation:

STT Documentation:

External References: