STT Provider Comparison - Benchmark Results
Comprehensive benchmark results comparing 6 Speech-to-Text providers on LibriSpeech test-clean dataset.
Executive Summary
Test Date: February 2026 Dataset: LibriSpeech test-clean (clean audiobook speech) Sample Size: 120 professionally recorded utterances (~10 minutes total) Providers Tested: 6 (AWS Transcribe, Aldea, Deepgram Nova-3, OpenAI Whisper, Faster-Whisper large-v3, GPT-4o Audio)
Winner: AWS Transcribe (by 0.03%)
AWS Transcribe achieved the lowest Word Error Rate at 3.57%, narrowly beating Aldea (3.60%) and Deepgram (3.76%). However, the differences between the top 3 providers are statistically insignificant at this sample size.
Recommended Provider: Deepgram Nova-3 offers the best speed-accuracy tradeoff for production use.
Full Results Table
| Rank | Provider | WER โ | CER โ | Precision | Recall | F1 | Median WER | Speed | Cost |
|---|---|---|---|---|---|---|---|---|---|
| ๐ฅ | AWS Transcribe | 3.57% | 1.01% | 0.965 | 0.962 | 0.963 | 0.0% | ~12s/file | $$ |
| ๐ฅ | Aldea | 3.60% | 1.27% | 0.964 | 0.960 | 0.962 | 0.0% | ~1.5s/file | $ |
| ๐ฅ | Deepgram Nova-3 | 3.76% | 1.15% | 0.964 | 0.961 | 0.962 | 0.0% | ~1s/file | $$ |
| 4 | OpenAI Whisper | 4.30% | 1.32% | 0.965 | 0.964 | 0.964 | 0.0% | ~1.5s/file | $ |
| 5 | Faster-Whisper | 4.33% | 1.29% | 0.964 | 0.966 | 0.964 | 0.0% | Local | Free |
| 6 | GPT-4o Audio | 45.11% | 31.75% | 0.846 | 0.855 | 0.847 | 0.0% | ~1.3s/file | $$$ |
Key:
- WER/CER: Lower is better (0% = perfect)
- Precision/Recall/F1: Higher is better (1.0 = perfect)
- Median WER: Most samples had perfect transcription (0% error)
- Speed: Time per audio file (averaged across 120 samples)
- Cost: Relative pricing ($ = cheapest, $$$ = most expensive)
Detailed Analysis
Top Tier: Professional Grade (WER < 4%)
๐ฅ AWS Transcribe - 3.57% WER
Strengths:
- Best overall word accuracy
- Lowest character error rate (1.01%)
- Excellent for production where accuracy is critical
Weaknesses:
- Slowest processing (~12s per file)
- Requires S3 bucket setup
- Higher cost
Use Cases:
- Medical transcription
- Legal documentation
- Applications requiring maximum accuracy
๐ฅ Aldea - 3.60% WER
Strengths:
- Virtually tied with AWS Transcribe for accuracy
- Fast processing (~1.5s per file)
- Lower cost than AWS
Weaknesses:
- None significant
Use Cases:
- Production applications needing great balance
- High-volume transcription workloads
- Cost-sensitive projects requiring accuracy
๐ฅ Deepgram Nova-3 - 3.76% WER
Strengths:
- Fastest processing (~1s per file)
- Top-tier accuracy
- Excellent API reliability
Weaknesses:
- Slightly higher WER than AWS/Aldea (difference likely not significant)
Use Cases:
- Recommended for most production deployments
- Real-time transcription
- High-throughput applications
- Applications prioritizing speed without sacrificing accuracy
Mid Tier: Good Performance (WER 4-5%)
OpenAI Whisper (API) - 4.30% WER
Strengths:
- Widely available and trusted
- Good accuracy for general use
- Fast processing
- Well-documented API
Weaknesses:
- Slightly worse than top tier providers
- Not as fast as Deepgram
Use Cases:
- General purpose transcription
- Applications already using OpenAI services
- Prototyping and development
Faster-Whisper (Local) - 4.33% WER
Strengths:
- Free (local inference)
- Matches OpenAI Whisper API accuracy
- Privacy-preserving (no data sent externally)
- Offline capable
Weaknesses:
- Slower processing (local compute required)
- Requires ~3GB model download
- GPU recommended for reasonable speed
Use Cases:
- Cost-sensitive applications
- Privacy-sensitive data (HIPAA, etc.)
- Offline transcription
- High-volume processing where API costs add up
Poor Performance: Not Recommended
GPT-4o Audio - 45.11% WER โ
Findings:
- 10x worse than specialized STT models
- Multimodal model not optimized for transcription
- Most expensive option
- Should NOT be used for pure transcription tasks
Why Poor Performance: GPT-4o Audio is designed for conversation and audio understanding, not accurate verbatim transcription. It often paraphrases or summarizes rather than transcribing exactly.
Recommendation: Use specialized STT models (any of the top 5) instead.
Statistical Significance
Top 3 Providers
The differences between AWS Transcribe (3.57%), Aldea (3.60%), and Deepgram (3.76%) are within 0.2 percentage points. At 120 samples:
- Confidence: These providers are essentially equivalent for clean speech
- Recommendation: Choose based on speed, cost, and API preferences rather than accuracy alone
- Note: Larger benchmark (500+ samples) needed to establish statistical significance
Mid Tier Providers
OpenAI Whisper (4.30%) and Faster-Whisper (4.33%) are statistically tied and about 0.5-0.7 percentage points worse than the top tier. This difference is noticeable but still represents excellent performance.
Error Analysis
Median WER: 0.0% for All Providers
All providers achieved perfect transcription (0% WER) on the majority of samples. Errors were concentrated in:
- Proper names
- Technical terminology
- Unusual pronunciations
- Very short utterances
Common Error Types
Substitutions (most common):
- Proper names: "Catherine" โ "Katherine"
- Homophones: "their" โ "there"
- Contractions: "it's" โ "its"
Deletions (less common):
- Filler words: "um", "uh"
- Repeated words
Insertions (least common):
- Added articles: "the"
- Punctuation differences
Speed Comparison
Processing time for 120 samples (~10 minutes of audio):
| Provider | Total Time | Avg per File | Notes |
|---|---|---|---|
| Deepgram Nova-3 | ~116s | ~1s | Fastest |
| GPT-4o Audio | ~159s | ~1.3s | Fast |
| OpenAI Whisper | ~173s | ~1.5s | Fast |
| Aldea | ~176s | ~1.5s | Fast |
| AWS Transcribe | ~1170s | ~12s | Slow (S3 overhead) |
| Faster-Whisper | ~1693s | ~28min total | Local (single CPU) |
Note: Faster-Whisper speed depends heavily on hardware. GPU acceleration significantly improves performance.
Cost Comparison
Estimated cost per 100 audio files (~10 minutes total):
| Provider | Cost | Notes |
|---|---|---|
| Faster-Whisper | $0 | Local inference, free |
| Aldea | ~$0.50 | Most economical API |
| OpenAI Whisper | ~$0.60 | Good value |
| Deepgram Nova-3 | ~$0.80 | Premium speed |
| AWS Transcribe | ~$1.50 | Premium accuracy |
| GPT-4o Audio | ~$2.00 | Expensive, poor results |
Note: Costs are approximate and based on standard pricing. Volume discounts may apply.
Recommendations by Use Case
Maximum Accuracy
Choice: AWS Transcribe Why: Lowest WER (3.57%), best CER (1.01%) Trade-off: Slower and more expensive
Best Balance (Recommended)
Choice: Deepgram Nova-3 Why: Top-tier accuracy (3.76%) with fastest speed Trade-off: Slightly higher cost than alternatives
Cost-Sensitive
Choice: Faster-Whisper large-v3 Why: Free, matches OpenAI Whisper accuracy Trade-off: Requires local compute, slower
High-Volume Processing
Choice: Aldea or Deepgram Why: Fast processing, excellent accuracy, reasonable cost Trade-off: None significant
Privacy-Sensitive
Choice: Faster-Whisper large-v3 Why: Local processing, no data leaves your infrastructure Trade-off: Requires local compute
General Purpose
Choice: OpenAI Whisper (API) Why: Widely available, well-documented, good performance Trade-off: Slightly worse than top tier
Methodology
Dataset
- Source: LibriSpeech test-clean
- Samples: 120 audio files
- Duration: ~10 minutes total
- Quality: Professional audiobook recordings
- Speaker: Multiple speakers, clear pronunciation
- Domain: General English, narrative text
Metrics
- WER (Word Error Rate): Primary metric for accuracy
- CER (Character Error Rate): Character-level accuracy
- Precision: Percentage of predicted words that are correct
- Recall: Percentage of actual words that were predicted
- F1 Score: Harmonic mean of precision and recall
Benchmark Process
- Download LibriSpeech test-clean dataset
- Ingest 120 samples into Biblicus corpus
- Store ground truth transcriptions
- Run each STT provider extraction
- Compare transcriptions against ground truth
- Calculate all metrics
- Generate comprehensive report
Next Steps
Expand Sample Size
Current 120-sample benchmark provides initial comparison. Recommend:
- 500+ samples for statistical significance
- Multiple datasets (test-other, Common Voice) for robustness
- Domain-specific tests (medical, legal, technical speech)
Test Additional Conditions
- Noisy environments
- Accented speech
- Spontaneous conversation
- Technical terminology
- Multiple languages
Measure Additional Metrics
- Real-time factor (processing speed vs audio duration)
- Latency (time to first word)
- Streaming vs batch performance
- API reliability and uptime
Conclusion
For most applications: Use Deepgram Nova-3 for the best speed-accuracy tradeoff.
For maximum accuracy: Use AWS Transcribe, though the improvement over Deepgram is marginal.
For cost-sensitive or privacy-sensitive applications: Use Faster-Whisper, which matches paid APIs at zero cost.
Avoid: GPT-4o Audio for transcription - it's designed for conversation understanding, not accurate transcription.
All top 5 providers achieve excellent results on clean speech. Your choice should be based on:
- Speed requirements
- Cost constraints
- Privacy considerations
- Existing infrastructure/preferences
Benchmark Reproducibility
All benchmarks are reproducible using Biblicus:
# Download dataset
python scripts/download_librispeech_samples.py \
--corpus corpora/librispeech_benchmark \
--count 120
# Run benchmark
python scripts/benchmark_all_stt_providers.py \
--corpus corpora/librispeech_benchmark \
--output results/stt_120_samples.json
Results: results/stt_100_samples.json