Whisper Medium sur audio YouTube en anglais — Benchmark du 30/03/2026 (WER, CER, RTF)

Cet article présente une exécution à configuration fixe sur un audio YouTube en anglais avec Whisper medium. Dans result.json, le score strict est WER 67,75% et Accuracy 32,25%, avec un profil d’erreur dominé par les suppressions (D=5722, S=68, I=0). Cela indique plutôt un écart de couverture/alignment entre la référence et l’hypothèse qu’un simple bruit lexical.

Video and reference text. Source video: https://www.youtube.com/watch?v=7J96ESznKMQ. The reference (ref.vtt) comes from the platform caption track, while model.vtt is the model output. So this benchmark measures agreement with platform captions (practical baseline), not a manually curated linguistic gold standard.

1. Why This Benchmark Matters

Long-form YouTube audio is a realistic ASR stress case: changing pace, edits, names, topic shifts, and mixed speaking styles. For subtitle QA, retrieval indexing, and content repurposing, this setup is more representative than short clean demos.

Using platform captions as reference also answers a practical product question: how far does our ASR output drift from what end users already see as subtitles? Even if captions are not perfect gold labels, this comparison is operationally useful and reproducible.

2. Testing Setup

Values below are taken from other.yaml and result.json in case 20260331.

Field	Value
Source	YouTube video
Date (processing window)	2026-03-31 (`processtime-at` → `completed-at`)
Language	English
Whisper model	medium
Audio duration (YAML label)	17:20
Audio duration (scorer / YAML parsed)	1040 s (≈ 17.33 minutes)
STT processing time	133 s
RTF	0.1279

Wall-clock timestamps: 2026-03-31 19:20:56 → 2026-03-31 19:23:09, consistent with 133 seconds processing time.

3. Evaluation Methodology

Evaluation script used: scripts/evaluate-vtt-metrics.js.

The script reads ref.vtt and model.vtt, extracts plain cue text, normalizes tokens, and aligns reference/hypothesis with Levenshtein dynamic programming.

[ \mathrm{WER} = \frac{S + D + I}{N}, \qquad \mathrm{Accuracy} = 1 - \mathrm{WER} ]

[ \mathrm{CER} = \frac{\text{Character edit distance}}{\text{Reference character count (no spaces)}} ]

[ \mathrm{RTF} = \frac{\text{STT processing time}}{\text{Audio duration}} ]

The script outputs two views:

strictMetrics: default normalization
relaxedMetrics: additional normalization (quotes/number formatting)

This helps distinguish formatting noise from true lexical/coverage mismatch.

4. Model Overview

Whisper medium is a common speed/quality trade-off checkpoint in practical transcription stacks. It is often suitable for draft transcripts, indexing, and downstream NLP preprocessing, but still requires review for verbatim publishing or compliance-sensitive workflows.

This benchmark tests one fixed setup only (no decoder sweep, no custom post-correction, no domain lexicon boosting).

5. Results (From `result.json`)

Strict metrics (metrics / strictMetrics)

Reference word count (N): 8,546
Substitutions (S): 68
Deletions (D): 5,722
Insertions (I): 0
WER: 0.6775099461736485
Accuracy: 0.32249005382635154
Reference character count: 32,329
Character edit distance: 21,566
CER: 0.6670790930743296
Audio duration (seconds): 1,040
STT processing time (seconds): 133
RTF: 0.12788461538461537
Eval script runtime (seconds): 56.703

Relaxed metrics (relaxedMetrics)

WER: 0.6775099461736485
Accuracy: 0.32249005382635154
CER: 0.666760334707683
Character edit distance: 21,355
Reference character count: 32,028

Rounded interpretation

Strict WER ≈ 67.75%, Accuracy ≈ 32.25%, CER ≈ 66.71%
Relaxed WER ≈ 67.75%, Accuracy ≈ 32.25%, CER ≈ 66.68%
Small strict/relaxed gap indicates mismatch is not mainly punctuation/formatting noise.
RTF ≈ 0.128 (about 7.8× faster than real time)

6. Error Pattern Analysis

Two signals stand out:

Insertion = 0
Deletion >> substitution (5,722 vs 68)

This pattern usually means many reference words are not aligned to hypothesis tokens. Typical causes include segmentation mismatch, truncated hypothesis coverage, or reference captions containing spans not reflected in model output.

Because strict and relaxed results are almost identical, normalization tweaks are not the main driver; coverage/alignment is likely dominant.

7. Key Insights

Speed: RTF is comfortably below 1, so throughput is practical for batch processing.
Accuracy: ~68% WER is too high for quote-level publication without review.
Error mode: Deletion-heavy profile suggests checking pairing/coverage before hyperparameter tuning.
Method robustness: strict and relaxed metrics are close, improving interpretability.
Representativeness: ~17.3 minutes is meaningful long-form input, but still only one clip/one setup.

8. Best Model for This Scenario

Under the narrow scope “Whisper medium + this exact clip + this exact reference source,” the run is a transparent baseline for future A/B comparisons. It does not claim universal superiority across all English YouTube transcription scenarios.

9. Neutral Final Verdict

For draft, indexing, and topic extraction workflows, this setup can be operationally useful. For verbatim publishing, compliance records, or accessibility-critical subtitles, current agreement levels still imply mandatory human correction or a stronger setup.

Keep the evaluation method fixed (scripts/evaluate-vtt-metrics.js) when iterating models so improvements remain comparable.

Source Materials

Original audio (video): https://www.youtube.com/watch?v=7J96ESznKMQ
Reference transcript (VTT): test-transcripts/{case-name}/ref.vtt
Model transcript (VTT): test-transcripts/{case-name}/model.vtt
Run metadata: test-transcripts/{case-name}/other.yaml
Precomputed evaluation metrics: test-transcripts/{case-name}/result.json

{case-name} = 20260331. Evaluation script: scripts/evaluate-vtt-metrics.js.