Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

STT Provider Comparison - Benchmark Results

Comprehensive benchmark results comparing 6 Speech-to-Text providers on LibriSpeech test-clean dataset.

Executive Summary

Test Date: February 2026 Dataset: LibriSpeech test-clean (clean audiobook speech) Sample Size: 120 professionally recorded utterances (~10 minutes total) Providers Tested: 6 (AWS Transcribe, Aldea, Deepgram Nova-3, OpenAI Whisper, Faster-Whisper large-v3, GPT-4o Audio)

Winner: AWS Transcribe (by 0.03%)

AWS Transcribe achieved the lowest Word Error Rate at 3.57%, narrowly beating Aldea (3.60%) and Deepgram (3.76%). However, the differences between the top 3 providers are statistically insignificant at this sample size.

Recommended Provider: Deepgram Nova-3 offers the best speed-accuracy tradeoff for production use.


Full Results Table

Rank Provider WER โ†“ CER โ†“ Precision Recall F1 Median WER Speed Cost
๐Ÿฅ‡ AWS Transcribe 3.57% 1.01% 0.965 0.962 0.963 0.0% ~12s/file $$
๐Ÿฅˆ Aldea 3.60% 1.27% 0.964 0.960 0.962 0.0% ~1.5s/file $
๐Ÿฅ‰ Deepgram Nova-3 3.76% 1.15% 0.964 0.961 0.962 0.0% ~1s/file $$
4 OpenAI Whisper 4.30% 1.32% 0.965 0.964 0.964 0.0% ~1.5s/file $
5 Faster-Whisper 4.33% 1.29% 0.964 0.966 0.964 0.0% Local Free
6 GPT-4o Audio 45.11% 31.75% 0.846 0.855 0.847 0.0% ~1.3s/file $$$

Key:


Detailed Analysis

Top Tier: Professional Grade (WER < 4%)

๐Ÿฅ‡ AWS Transcribe - 3.57% WER

Strengths:

Weaknesses:

Use Cases:


๐Ÿฅˆ Aldea - 3.60% WER

Strengths:

Weaknesses:

Use Cases:


๐Ÿฅ‰ Deepgram Nova-3 - 3.76% WER

Strengths:

Weaknesses:

Use Cases:


Mid Tier: Good Performance (WER 4-5%)

OpenAI Whisper (API) - 4.30% WER

Strengths:

Weaknesses:

Use Cases:


Faster-Whisper (Local) - 4.33% WER

Strengths:

Weaknesses:

Use Cases:


GPT-4o Audio - 45.11% WER โŒ

Findings:

Why Poor Performance: GPT-4o Audio is designed for conversation and audio understanding, not accurate verbatim transcription. It often paraphrases or summarizes rather than transcribing exactly.

Recommendation: Use specialized STT models (any of the top 5) instead.


Statistical Significance

Top 3 Providers

The differences between AWS Transcribe (3.57%), Aldea (3.60%), and Deepgram (3.76%) are within 0.2 percentage points. At 120 samples:

Mid Tier Providers

OpenAI Whisper (4.30%) and Faster-Whisper (4.33%) are statistically tied and about 0.5-0.7 percentage points worse than the top tier. This difference is noticeable but still represents excellent performance.


Error Analysis

Median WER: 0.0% for All Providers

All providers achieved perfect transcription (0% WER) on the majority of samples. Errors were concentrated in:

Common Error Types

Substitutions (most common):

Deletions (less common):

Insertions (least common):


Speed Comparison

Processing time for 120 samples (~10 minutes of audio):

Provider Total Time Avg per File Notes
Deepgram Nova-3 ~116s ~1s Fastest
GPT-4o Audio ~159s ~1.3s Fast
OpenAI Whisper ~173s ~1.5s Fast
Aldea ~176s ~1.5s Fast
AWS Transcribe ~1170s ~12s Slow (S3 overhead)
Faster-Whisper ~1693s ~28min total Local (single CPU)

Note: Faster-Whisper speed depends heavily on hardware. GPU acceleration significantly improves performance.


Cost Comparison

Estimated cost per 100 audio files (~10 minutes total):

Provider Cost Notes
Faster-Whisper $0 Local inference, free
Aldea ~$0.50 Most economical API
OpenAI Whisper ~$0.60 Good value
Deepgram Nova-3 ~$0.80 Premium speed
AWS Transcribe ~$1.50 Premium accuracy
GPT-4o Audio ~$2.00 Expensive, poor results

Note: Costs are approximate and based on standard pricing. Volume discounts may apply.


Recommendations by Use Case

Maximum Accuracy

Choice: AWS Transcribe Why: Lowest WER (3.57%), best CER (1.01%) Trade-off: Slower and more expensive

Choice: Deepgram Nova-3 Why: Top-tier accuracy (3.76%) with fastest speed Trade-off: Slightly higher cost than alternatives

Cost-Sensitive

Choice: Faster-Whisper large-v3 Why: Free, matches OpenAI Whisper accuracy Trade-off: Requires local compute, slower

High-Volume Processing

Choice: Aldea or Deepgram Why: Fast processing, excellent accuracy, reasonable cost Trade-off: None significant

Privacy-Sensitive

Choice: Faster-Whisper large-v3 Why: Local processing, no data leaves your infrastructure Trade-off: Requires local compute

General Purpose

Choice: OpenAI Whisper (API) Why: Widely available, well-documented, good performance Trade-off: Slightly worse than top tier


Methodology

Dataset

Metrics

Benchmark Process

  1. Download LibriSpeech test-clean dataset
  2. Ingest 120 samples into Biblicus corpus
  3. Store ground truth transcriptions
  4. Run each STT provider extraction
  5. Compare transcriptions against ground truth
  6. Calculate all metrics
  7. Generate comprehensive report

Next Steps

Expand Sample Size

Current 120-sample benchmark provides initial comparison. Recommend:

Test Additional Conditions

Measure Additional Metrics


Conclusion

For most applications: Use Deepgram Nova-3 for the best speed-accuracy tradeoff.

For maximum accuracy: Use AWS Transcribe, though the improvement over Deepgram is marginal.

For cost-sensitive or privacy-sensitive applications: Use Faster-Whisper, which matches paid APIs at zero cost.

Avoid: GPT-4o Audio for transcription - it's designed for conversation understanding, not accurate transcription.

All top 5 providers achieve excellent results on clean speech. Your choice should be based on:

  1. Speed requirements
  2. Cost constraints
  3. Privacy considerations
  4. Existing infrastructure/preferences

Benchmark Reproducibility

All benchmarks are reproducible using Biblicus:

# Download dataset
python scripts/download_librispeech_samples.py \
  --corpus corpora/librispeech_benchmark \
  --count 120

# Run benchmark
python scripts/benchmark_all_stt_providers.py \
  --corpus corpora/librispeech_benchmark \
  --output results/stt_120_samples.json

Results: results/stt_100_samples.json