Whisper Medium на английском YouTube-аудио — бенчмарк 2026-03-30 (WER, CER, RTF)
2026-03-30Test
Eric King
Author
В этом материале зафиксирован прогон с фиксированной конфигурацией на английском YouTube-аудио с Whisper medium. В result.json строгий результат: WER 68,23% и Accuracy 31,77%, при ярко выраженном профиле удалений (D=8718, S=131, I=0). Это чаще указывает на проблему покрытия/выравнивания между референсом и гипотезой, а не только на локальные ошибки распознавания.
Video and reference text. Source video: https://www.youtube.com/watch?v=EatCzpKNTMs. The reference (
ref.vtt) comes from the platform caption track, while model.vtt is the model output. So this benchmark measures agreement with platform captions (practical baseline), not a manually curated linguistic gold standard.1. Why This Benchmark Matters
Long-form YouTube audio is a realistic ASR stress case: changing pace, edits, names, topic shifts, and mixed speaking styles. For subtitle QA, retrieval indexing, and content repurposing, this setup is more representative than short clean demos.
Using platform captions as reference also answers a practical product question: how far does our ASR output drift from what end users already see as subtitles? Even if captions are not perfect gold labels, this comparison is operationally useful and reproducible.
2. Testing Setup
Values below are taken from
other.yaml and result.json in case 20260330.| Field | Value |
|---|---|
| Source | YouTube video |
| Date (processing window) | 2026-03-30 (processtime-at → completed-at) |
| Language | English |
| Whisper model | medium |
| Audio duration (YAML label) | 22:44 |
| Audio duration (scorer / YAML parsed) | 1364 s (≈ 22.73 minutes) |
| STT processing time | 365 s |
| RTF | 0.2676 |
Wall-clock timestamps: 2026-03-30 19:49:57 → 2026-03-30 19:56:02, consistent with 365 seconds processing time.
3. Evaluation Methodology
Evaluation script used:
scripts/evaluate-vtt-metrics.js.The script reads
ref.vtt and model.vtt, extracts plain cue text, normalizes tokens, and aligns reference/hypothesis with Levenshtein dynamic programming.[
\mathrm{WER} = rac{S + D + I}{N}, \qquad \mathrm{Accuracy} = 1 - \mathrm{WER}
]
[
\mathrm{CER} = rac{ ext{Character edit distance}}{ ext{Reference character count (no spaces)}}
]
[
\mathrm{RTF} = rac{ ext{STT processing time}}{ ext{Audio duration}}
]
The script outputs two views:
strictMetrics: default normalizationrelaxedMetrics: additional normalization (quotes/number formatting)
This helps distinguish formatting noise from true lexical/coverage mismatch.
4. Model Overview
Whisper medium is a common speed/quality trade-off checkpoint in practical transcription stacks. It is often suitable for draft transcripts, indexing, and downstream NLP preprocessing, but still requires review for verbatim publishing or compliance-sensitive workflows.
This benchmark tests one fixed setup only (no decoder sweep, no custom post-correction, no domain lexicon boosting).
5. Results (From result.json)
Strict metrics (
metrics / strictMetrics)- Reference word count (N): 12,970
- Substitutions (S): 131
- Deletions (D): 8,718
- Insertions (I): 0
- WER: 0.6822667694680031
- Accuracy: 0.31773323053199687
- Reference character count: 51,745
- Character edit distance: 34,683
- CER: 0.6702676587109866
- Audio duration (seconds): 1,364
- STT processing time (seconds): 365
- RTF: 0.26759530791788855
- Eval script runtime (seconds): 149.07
Relaxed metrics (
relaxedMetrics)- WER: 0.682112567463377
- Accuracy: 0.317887432536623
- CER: 0.6700148518721175
- Character edit distance: 34,286
- Reference character count: 51,172
Rounded interpretation
- Strict WER ≈ 68.23%, Accuracy ≈ 31.77%, CER ≈ 67.03%
- Relaxed WER ≈ 68.21%, Accuracy ≈ 31.79%, CER ≈ 67.00%
- Small strict/relaxed gap indicates mismatch is not mainly punctuation/formatting noise.
- RTF ≈ 0.268 (about 3.7× faster than real time)
6. Error Pattern Analysis
Two signals stand out:
- Insertion = 0
- Deletion >> substitution (8,718 vs 131)
This pattern usually means many reference words are not aligned to hypothesis tokens. Typical causes include segmentation mismatch, truncated hypothesis coverage, or reference captions containing spans not reflected in model output.
Because strict and relaxed results are almost identical, normalization tweaks are not the main driver; coverage/alignment is likely dominant.
7. Key Insights
- Speed: RTF is comfortably below 1, so throughput is practical for batch processing.
- Accuracy: ~68% WER is too high for quote-level publication without review.
- Error mode: Deletion-heavy profile suggests checking pairing/coverage before hyperparameter tuning.
- Method robustness: strict and relaxed metrics are close, improving interpretability.
- Representativeness: ~22.7 minutes is meaningful long-form input, but still only one clip/one setup.
8. Best Model for This Scenario
Under the narrow scope “Whisper medium + this exact clip + this exact reference source,” the run is a transparent baseline for future A/B comparisons. It does not claim universal superiority across all English YouTube transcription scenarios.
9. Neutral Final Verdict
For draft, indexing, and topic extraction workflows, this setup can be operationally useful. For verbatim publishing, compliance records, or accessibility-critical subtitles, current agreement levels still imply mandatory human correction or a stronger setup.
Keep the evaluation method fixed (
scripts/evaluate-vtt-metrics.js) when iterating models so improvements remain comparable.Source Materials
- Original audio (video): https://www.youtube.com/watch?v=EatCzpKNTMs
- Reference transcript (VTT):
test-transcripts/{case-name}/ref.vtt - Model transcript (VTT):
test-transcripts/{case-name}/model.vtt - Run metadata:
test-transcripts/{case-name}/other.yaml - Precomputed evaluation metrics:
test-transcripts/{case-name}/result.json
{case-name} = 20260330. Evaluation script: scripts/evaluate-vtt-metrics.js.