STT Provider Benchmarking Guide
Comprehensive guide for benchmarking Speech-to-Text providers using Biblicus with real-world results.
New to benchmarking? Start with the Benchmarking Overview for a platform introduction.
Table of Contents
- Overview
- Benchmark Results Summary
- Supported STT Providers
- Quick Start
- Available Datasets
- Understanding Metrics
- Running Custom Benchmarks
- Best Practices
- Troubleshooting
Overview
Biblicus provides a robust framework for evaluating and comparing Speech-to-Text providers using standardized metrics on labeled audio datasets. This guide provides practical instructions and real benchmark results.
Key Features:
- 6 integrated STT providers (AWS, Aldea, Deepgram, OpenAI, Faster-Whisper, GPT-4o Audio)
- Multiple evaluation metrics (WER, CER, precision, recall, F1)
- 500+ LibriSpeech samples ready for testing
- Automated ground truth comparison
- JSON export for detailed analysis
Benchmark Results Summary
120-Sample LibriSpeech test-clean Evaluation
Comprehensive benchmark results from 120 professionally recorded audiobook samples:
| Provider | WER | CER | F1 Score | Speed | Cost | Verdict |
|---|---|---|---|---|---|---|
| AWS Transcribe | 3.57% | 1.01% | 0.963 | ~12s/file | $$ | ๐ฅ Best accuracy, slower |
| Aldea | 3.60% | 1.27% | 0.962 | ~1.5s/file | $ | ๐ฅ Excellent balance |
| Deepgram Nova-3 | 3.76% | 1.15% | 0.962 | ~1s/file | $$ | ๐ฅ Fastest, great accuracy |
| OpenAI Whisper | 4.30% | 1.32% | 0.964 | ~1.5s/file | $ | โ Good, widely available |
| Faster-Whisper | 4.33% | 1.29% | 0.964 | Local | Free | โ Best for offline/free |
| GPT-4o Audio | 45.11% | 31.75% | 0.847 | ~1.3s/file | $$$ | โ Not suitable for STT |
Key Findings:
- Top tier providers (AWS, Aldea, Deepgram) achieve ~3.6-3.8% WER on clean speech
- Differences between top 3 are statistically insignificant at this sample size
- Deepgram offers best speed-accuracy tradeoff for production use
- Faster-Whisper matches OpenAI Whisper accuracy at zero cost
- GPT-4o Audio is 10x worse - use specialized STT models instead
Recommendation by Use Case:
- Production (balanced): Deepgram Nova-3 - fastest with top-tier accuracy
- Production (max accuracy): AWS Transcribe - slightly better WER, slower
- Cost-sensitive: Faster-Whisper large-v3 - local, free, matches paid APIs
- General purpose: OpenAI Whisper - widely available, good performance
- Avoid: GPT-4o Audio - not optimized for transcription
Supported STT Providers
Currently Integrated (6 providers)
1. AWS Transcribe
- Extractor ID:
stt-aws-transcribe - Accuracy: โญ 3.57% WER (Best)
- Speed: Slow (~12s per file due to S3 upload)
- Cost: $$ (~$1.50 per 100 files)
- Requirements: AWS credentials, S3 bucket
- Best for: Maximum accuracy requirements
2. Aldea
- Extractor ID:
stt-aldea - Accuracy: โญ 3.60% WER
- Speed: Fast (~1.5s per file)
- Cost: $ (~$0.50 per 100 files)
- Requirements: Aldea API key
- Best for: Excellent balance of speed, cost, accuracy
3. Deepgram Nova-3
- Extractor ID:
stt-deepgram - Accuracy: โญ 3.76% WER
- Speed: Fastest (~1s per file)
- Cost: $$ (~$0.80 per 100 files)
- Requirements: Deepgram API key
- Best for: Production workloads requiring speed
4. OpenAI Whisper (API)
- Extractor ID:
stt-openai - Accuracy: โ 4.30% WER
- Speed: Fast (~1.5s per file)
- Cost: $ (~$0.60 per 100 files)
- Requirements: OpenAI API key
- Best for: General purpose, widely available
5. Faster-Whisper (Local)
- Extractor ID:
stt-faster-whisper - Model: large-v3 with CTranslate2
- Accuracy: โ 4.33% WER
- Speed: Slow (local CPU/GPU processing)
- Cost: Free (local inference)
- Requirements: Local compute, ~3GB model download
- Best for: Offline, privacy-sensitive, or cost-sensitive applications
6. GPT-4o Audio (Not Recommended)
- Extractor ID:
stt-openai-audio - Accuracy: โ 45.11% WER (Poor)
- Speed: Fast (~1.3s per file)
- Cost: $$$ (~$2.00 per 100 files)
- Note: Multimodal model not optimized for transcription
- Recommendation: Use specialized STT models instead
Quick Start
1. Download Benchmark Dataset
# Download 100 LibriSpeech test-clean samples
python scripts/download_librispeech_samples.py \
--corpus corpora/librispeech_benchmark \
--count 100
# For larger benchmarks (500 samples for statistical significance)
python scripts/download_librispeech_samples.py \
--corpus corpora/librispeech_benchmark \
--count 500 \
--force
# Download challenging test-other subset
python scripts/download_openslr_samples.py \
--corpus corpora/librispeech_test_other \
--dataset SLR12 \
--subset test-other \
--count 100
# 3. View results
cat results/stt_benchmark.json | jq '.providers[] | {name, wer: .metrics.wer.avg}'
Benchmark Dataset
LibriSpeech Test-Clean
What is LibriSpeech?
- Corpus of ~1000 hours of 16kHz read English speech
- Derived from LibriVox audiobook recordings
- Test-clean subset: 5.4 hours of high-quality audio
- ~2600 utterances with ground truth transcriptions
Download:
python scripts/download_librispeech_samples.py \
--corpus corpora/librispeech_benchmark \
--count 100
This will:
- Download LibriSpeech test-clean (~346 MB)
- Extract FLAC audio files
- Parse ground truth transcriptions from .trans.txt files
- Create corpus at
corpora/librispeech_benchmark/ - Store ground truth in
metadata/ground_truth/
Dataset structure:
corpora/librispeech_benchmark/
โโโ metadata/
โ โโโ config.json
โ โโโ catalog.json
โ โโโ ground_truth/
โ โโโ <audio-id>.txt # Ground truth transcription
โ โโโ ...
โโโ <audio-id>.flac # Audio file
โโโ ...
Using Your Own Dataset
To benchmark on custom audio:
-
Prepare ground truth files:
corpus_dir/metadata/ground_truth/ โโโ <audio-id>.txt โโโ ... -
Ingest audio:
from biblicus import Corpus from pathlib import Path corpus = Corpus(Path("my_corpus")) corpus.ingest_file("audio.flac", tags=["benchmark", "speech"]) -
Run evaluation:
from biblicus.evaluation.stt_benchmark import STTBenchmark benchmark = STTBenchmark(corpus) report = benchmark.evaluate_extraction( snapshot_reference="<snapshot-id>", ground_truth_dir=corpus.root / "metadata" / "ground_truth" )
Available STT Providers
Biblicus includes 3 STT provider integrations:
| Provider | Model | Strengths | API Required |
|---|---|---|---|
| OpenAI Whisper | whisper-1 | General-purpose, multilingual | OPENAI_API_KEY |
| Deepgram Nova-3 | nova-3 | Fast, accurate, features | DEEPGRAM_API_KEY |
| Aldea | (default) | Custom STT service | ALDEA_API_KEY |
Understanding Metrics
Biblicus STT benchmarks use three categories of metrics:
Word Error Rate (WER)
Primary metric for STT accuracy.
WER = (Substitutions + Deletions + Insertions) / Total Words
- Substitutions: Wrong word transcribed
- Deletions: Word missed
- Insertions: Extra word added
- Lower is better (0.0 = perfect, 1.0 = all words wrong)
Interpretation:
- WER < 0.05: Excellent (human-level)
- WER < 0.10: Very good (production-ready)
- WER < 0.20: Good (acceptable for many uses)
- WER > 0.30: Poor (needs improvement)
Character Error Rate (CER)
Character-level accuracy metric.
CER = (Character Substitutions + Deletions + Insertions) / Total Characters
- More granular than WER
- Better for languages without clear word boundaries
- Lower is better
Word-Level Metrics
Precision, Recall, F1 Score (bag-of-words)
- Precision: What % of transcribed words are correct
- Recall: What % of actual words were transcribed
- F1 Score: Harmonic mean of precision and recall
These ignore word order and focus on vocabulary accuracy.
For detailed explanations, see the Metrics Reference.
Running Benchmarks
Benchmark All Providers
python scripts/benchmark_all_stt_providers.py \
--corpus corpora/librispeech_benchmark \
--output results/stt_comparison.json
What it does:
- Tests all 3 STT providers (OpenAI, Deepgram, Aldea)
- Transcribes all audio files in the corpus
- Calculates WER, CER, and word-level metrics
- Generates comprehensive comparison report
Output:
results/stt_comparison.json- Full results with all metrics- Console table showing provider comparison
Example output:
================================================================================
COMPREHENSIVE STT PROVIDER COMPARISON
================================================================================
Provider WER CER Precision Recall F1 Status
------------------------------------------------------------------------------------------------
OpenAI Whisper 0.045 0.023 0.982 0.975 0.978 โ OK
Deepgram Nova-3 0.052 0.028 0.976 0.968 0.972 โ OK
Aldea 0.068 0.035 0.965 0.952 0.958 โ OK
------------------------------------------------------------------------------------------------
๐ Best Word Accuracy (Lowest WER): OpenAI Whisper
WER: 0.045, Median: 0.042
๐ Best Character Accuracy (Lowest CER): OpenAI Whisper
CER: 0.023, Median: 0.021
๐ Best Word Finding (F1): OpenAI Whisper
F1: 0.978, Precision: 0.982, Recall: 0.975
Benchmark Specific Providers
# Test only OpenAI and Deepgram
python scripts/benchmark_all_stt_providers.py \
--corpus corpora/librispeech_benchmark \
--providers stt-openai stt-deepgram \
--output results/openai_vs_deepgram.json
Quick Test with Fewer Files
python scripts/benchmark_all_stt_providers.py \
--corpus corpora/librispeech_benchmark \
--limit 5 \
--output results/quick_test.json
Results Analysis
JSON Output Structure
{
"benchmark_timestamp": "2026-02-13T20:00:00",
"corpus_path": "corpora/librispeech_benchmark",
"total_providers": 3,
"successful_providers": 3,
"failed_providers": 0,
"providers": [
{
"name": "OpenAI Whisper",
"snapshot_id": "abc123...",
"success": true,
"metrics": {
"wer": {
"avg": 0.045,
"median": 0.042,
"avg_substitutions": 2.3,
"avg_deletions": 0.8,
"avg_insertions": 0.5
},
"cer": {
"avg": 0.023,
"median": 0.021
},
"word_level": {
"avg_precision": 0.982,
"avg_recall": 0.975,
"avg_f1": 0.978,
"median_f1": 0.980
}
},
"provider_configuration": { ... },
"total_audio_files": 100
}
],
"best_performers": {
"lowest_wer": "OpenAI Whisper",
"lowest_cer": "OpenAI Whisper",
"best_f1": "OpenAI Whisper"
}
}
Analyzing Results
Find best provider:
cat results/stt_comparison.json | jq '.providers | sort_by(.metrics.wer.avg) | .[0] | {name, wer: .metrics.wer.avg}'
Compare WER across providers:
cat results/stt_comparison.json | jq '.providers[] | {name, wer: .metrics.wer.avg, cer: .metrics.cer.avg}' | jq -s 'sort_by(.wer)'
Find audio files with high WER:
import json
with open('results/stt_comparison.json') as f:
data = json.load(f)
# Assuming per-audio results are stored (implementation detail)
# You can analyze which utterances have highest error rates
Dependencies
Installing STT Provider Dependencies
Different providers require different dependencies:
OpenAI Whisper:
pip install "biblicus[openai]"
export OPENAI_API_KEY="sk-..."
Deepgram:
pip install "biblicus[deepgram]"
export DEEPGRAM_API_KEY="..."
Aldea:
pip install "biblicus[aldea]"
export ALDEA_API_KEY="..."
All providers:
pip install "biblicus[openai,deepgram,aldea]"
Checking Dependencies
# Test OpenAI
python -c "from openai import OpenAI; print('OpenAI OK')"
# Test Deepgram
python -c "from deepgram import DeepgramClient; print('Deepgram OK')"
# Test Aldea
python -c "import httpx; print('Aldea OK')"
Troubleshooting
Common Issues
Issue: "API key not found"
Solution: Set environment variable or configure in ~/.biblicus/config.yml:
openai:
api_key: "sk-..."
deepgram:
api_key: "..."
aldea:
api_key: "..."
Issue: "Ground truth directory not found"
Solution: Run python scripts/download_librispeech_samples.py first
Issue: "Audio format not supported"
Solution: LibriSpeech uses FLAC format. Ensure audio files are valid FLAC.
Issue: "API rate limit exceeded"
Solution:
- Reduce --limit parameter
- Add delays between API calls
- Use provider with higher rate limits
Issue: "Results don't match expected WER"
Solution: Check:
- Correct ground truth files loaded
- Audio quality is good
- Provider configuration is appropriate
- Punctuation/formatting settings
Issue: "High API costs"
Solution:
- Start with --limit 5 or --limit 10 for testing
- Use quick.yaml config (20 files) instead of full (2600 files)
- Calculate costs: ~$0.006 per minute of audio (varies by provider)
Benchmark Modes
Use configuration files for different benchmark scales:
Quick (20 audio files, ~2-5 minutes):
# Manually specify count
python scripts/download_librispeech_samples.py --corpus corpora/librispeech_benchmark --count 20
python scripts/benchmark_all_stt_providers.py --corpus corpora/librispeech_benchmark --output results/quick.json
Standard (100 audio files, ~10-20 minutes):
python scripts/download_librispeech_samples.py --corpus corpora/librispeech_benchmark --count 100
python scripts/benchmark_all_stt_providers.py --corpus corpora/librispeech_benchmark --output results/standard.json
Full (all audio files, ~2-4 hours):
# Download full test-clean dataset
python scripts/download_librispeech_samples.py --corpus corpora/librispeech_benchmark --count 2600
python scripts/benchmark_all_stt_providers.py --corpus corpora/librispeech_benchmark --output results/full.json
Cost Estimation
STT benchmarking involves API costs:
| Provider | Cost per Hour | 20 Files (~10 min) | 100 Files (~50 min) | Full (~5.4 hours) |
|---|---|---|---|---|
| OpenAI Whisper | $0.36/hr | ~$0.06 | ~$0.30 | ~$1.94 |
| Deepgram Nova-3 | $0.36/hr | ~$0.06 | ~$0.30 | ~$1.94 |
| Aldea | (varies) | (varies) | (varies) | (varies) |
Cost-saving tips:
- Start with
--limit 5for development - Use quick config (20 files) for iteration
- Run full benchmarks only for final results
Additional Datasets
LibriSpeech test-clean is the only dataset that has been benchmarked end to end. The repository ships download scripts for other speech corpora, with the following status:
| Dataset | Script | Status |
|---|---|---|
| LibriSpeech test-clean | scripts/download_librispeech_samples.py |
Working, 500 samples ingested and tested |
| LibriSpeech test-other and other OpenSLR sets | scripts/download_openslr_samples.py |
Working |
| Mozilla Common Voice | scripts/download_common_voice_samples.py |
Requires a manual download from Mozilla |
| Google FLEURS (multilingual) | scripts/download_fleurs_samples.py |
Requires the datasets library |
| TED-LIUM | scripts/download_tedlium_samples.py |
Upstream download URL is broken |
| VoxForge | scripts/download_voxforge_samples.py |
Upstream download URL is broken |
| CMU AN4 | scripts/download_an4_samples.py |
Upstream download URL is broken |
Provider Configuration Notes
- AWS Transcribe (
stt-aws-transcribe) uploads audio to an S3 bucket before transcription. Configurelanguage_code,region_nameands3_bucket. - Faster-Whisper (
stt-faster-whisper) runs locally with CTranslate2. The benchmark usedmodel_size: large-v3,device: autoandcompute_type: int8, at roughly 14 seconds per file on CPU. - GPT-4o Audio (
stt-openai-audio) does not accept FLAC directly; the extractor converts FLAC to WAV in memory withpydub, which needssoxorffmpeginstalled on the host. - Azure Speech (
stt-azure-speech) and Google Cloud Speech-to-Text (stt-google-speech) are integrated but have not been benchmarked yet.
Accuracy tiers on LibriSpeech test-clean (120 samples):
- Top tier (under 4% WER): AWS Transcribe, Aldea, Deepgram Nova-3
- Mid tier (4% to 5% WER): OpenAI Whisper, Faster-Whisper
- Poor (over 40% WER): GPT-4o Audio
Planned Improvements
- Expand the LibriSpeech sample to 500 or more for statistical significance.
- Add harder datasets (LibriSpeech test-other, Common Voice) and multilingual testing with FLEURS.
- Add domain-specific speech (medical, legal, technical).
- Measure real-time factor and streaming latency.
- Run providers in parallel, track cost in benchmark results, and visualize results.
See Also
Benchmarking Documentation:
- Benchmarking Overview - Platform introduction
- Metrics Reference - Understanding WER, CER, and other metrics
- OCR Benchmarking Guide - OCR pipeline benchmarking
STT Documentation:
- Speech to Text - STT extraction guide
External References: