Whisper Best Settings: Complete Guide to Optimal Configuration

---

Understanding Whisper Parameters

import whisper

model = whisper.load_model("base")

result = model.transcribe(
    audio="audio.mp3",
    verbose=False,
    temperature=0.0,
    compression_ratio_threshold=2.4,
    logprob_threshold=-1.0,
    no_speech_threshold=0.6,
    condition_on_previous_text=True,
    initial_prompt=None,
    word_timestamps=False,
    prepend_punctuations="\"'"¿([{-",
    append_punctuations="\"'.。,，!！?？:：")]}、",
    decode_options=None,
    best_of=5,
    beam_size=5,
    patience=1.0,
    length_penalty=1.0,
    suppress_tokens="-1",
    suppress_blank=True,
    without_timestamps=False,
    max_initial_timestamp=1.0,
    word_timestamps=False,
    language=None,
    task="transcribe",
    fp16=True,
    temperature_increment_on_fallback=0.2,
    compression_ratio_threshold=2.4,
    logprob_threshold=-1.0,
    no_speech_threshold=0.6
)

---

Core Parameters

1. Model Size (model)

# Available models (from smallest to largest):
model = whisper.load_model("tiny")    # Fastest, lowest accuracy
model = whisper.load_model("base")    # Balanced
model = whisper.load_model("small")   # Good accuracy
model = whisper.load_model("medium")  # High accuracy
model = whisper.load_model("large")   # Best accuracy, slowest

Model	Accuracy	Speed	VRAM	Best For
tiny	⭐⭐	⭐⭐⭐⭐⭐	~1GB	Quick testing
base	⭐⭐⭐	⭐⭐⭐⭐	~1GB	General use
small	⭐⭐⭐⭐	⭐⭐⭐	~2GB	Good balance
medium	⭐⭐⭐⭐⭐	⭐⭐	~5GB	Recommended for most
large	⭐⭐⭐⭐⭐⭐	⭐	~10GB	Maximum accuracy

• For most use cases: Use medium model
• For speed-critical: Use small or base
• For maximum accuracy: Use large
• For testing: Use tiny or base

---

2. Temperature

result = model.transcribe(
    audio,
    temperature=0.0  # Most deterministic, best for accuracy
)

Value	Behavior	Use Case
0.0	Most deterministic	✅ Best for accuracy
0.2	Slight randomness	Good balance
0.6	Default, balanced	General use
1.0+	More creative	Not recommended for transcription

# For maximum accuracy
temperature=0.0

# Whisper can use multiple temperatures
temperature=(0.0, 0.2, 0.4, 0.6, 0.8)  # Tries each, picks best

• Reduces random variations
• More consistent results
• Better for technical terms
• Improves accuracy by 2-5%

---

3. Best Of

result = model.transcribe(
    audio,
    best_of=5  # Try 5 decodings, pick best
)

Value	Accuracy Gain	Speed Impact	Recommendation
1 (default)	Baseline	Fastest	General use
3	+2-4%	3x slower	Good balance
5	+3-8%	5x slower	✅ Best for accuracy
10	+5-10%	10x slower	Overkill for most

# For critical transcriptions
best_of=5  # Good balance of accuracy and speed

# For general use
best_of=1  # Fastest, still accurate

---

4. Beam Size

result = model.transcribe(
    audio,
    beam_size=5  # Explore 5 candidate sequences
)

Value	Accuracy	Speed	Use Case
1	Fastest	Fastest	Real-time (greedy)
5	High	Medium	✅ Recommended
10	Very High	Slow	Maximum accuracy

# Standard configuration
beam_size=5  # Good balance

# For maximum accuracy
beam_size=5  # Works well with best_of=5

---

5. Patience

result = model.transcribe(
    audio,
    patience=1.0  # Default, good balance
)

Value	Behavior	Use Case
0.0	Aggressive pruning	Fast, less accurate
1.0	Default	✅ Recommended
2.0	More patient	Slower, slightly better

---

6. Condition on Previous Text

result = model.transcribe(
    audio,
    condition_on_previous_text=True  # Use context
)

Value	Behavior	Impact
True	Uses previous context	✅ Better accuracy
False	Independent segments	Faster, less accurate

# Always enable for better accuracy
condition_on_previous_text=True

• Maintains context across segments
• Better handling of pronouns and references
• Improves accuracy by 2-5%

---

7. Language

result = model.transcribe(
    audio,
    language="en"  # English
)

language="en"  # English
language="zh"  # Chinese
language="es"  # Spanish
language="fr"  # French
language="de"  # German
language="ja"  # Japanese
# ... and 90+ more languages

# If language is known, always specify
language="en"  # Improves accuracy by 5-10%

# For auto-detection
language=None  # Slower, may misdetect

• Skips language detection step (faster)
• Prevents misdetection errors
• Better handling of accents
• Improves accuracy by 5-10%

---

8. Initial Prompt

result = model.transcribe(
    audio,
    initial_prompt="This is a medical consultation discussing patient symptoms and treatment options."
)

# Include:
# - Topic/domain
# - Speaker names
# - Technical terms
# - Context

initial_prompt="""
This is a business meeting about Q4 product planning.
Participants: Sarah (Product Manager), John (Engineer).
Topics: API endpoints, microservices, Kubernetes.
"""

• Better recognition of proper nouns
• Improved technical terminology
• Context-aware transcription
• Improves accuracy by 5-15% for domain-specific content

---

9. Word Timestamps

result = model.transcribe(
    audio,
    word_timestamps=True  # Get word-level timestamps
)

# Access word timestamps
for segment in result["segments"]:
    for word in segment.get("words", []):
        print(f"{word['word']}: {word['start']:.2f}s - {word['end']:.2f}s")

Value	Output	Use Case
True	Word-level timestamps	Subtitles, search, analysis
False	Segment-level only	General transcription

# Enable for subtitles, search, or detailed analysis
word_timestamps=True

# Disable for simple transcription (faster)
word_timestamps=False

---

10. Task

# Transcribe (default)
result = model.transcribe(audio, task="transcribe")

# Translate to English
result = model.transcribe(audio, task="translate")

Value	Behavior	Use Case
"transcribe"	Original language	Standard transcription
"translate"	English translation	Multilingual content

# For transcription
task="transcribe"

# For translation to English
task="translate"  # Useful for multilingual meetings

---

Threshold Parameters

11. No Speech Threshold

result = model.transcribe(
    audio,
    no_speech_threshold=0.6  # Default
)

Value	Behavior	Use Case
0.2	Very sensitive	Noisy audio
0.6	Default	✅ Recommended
0.8	Less sensitive	Clean audio

---

12. Log Probability Threshold

result = model.transcribe(
    audio,
    logprob_threshold=-1.0  # Default
)

Value	Behavior	Use Case
-2.0	More permissive	Noisy audio
-1.0	Default	✅ Recommended
0.0	Strict	Very clean audio

---

13. Compression Ratio Threshold

result = model.transcribe(
    audio,
    compression_ratio_threshold=2.4  # Default
)

Value	Behavior	Use Case
2.0	More strict	Very clean audio
2.4	Default	✅ Recommended
3.0	More permissive	Noisy audio

---

Optimal Settings by Use Case

Maximum Accuracy Configuration

import whisper

model = whisper.load_model("large")  # or "medium"

result = model.transcribe(
    audio,
    language="en",  # Specify if known
    temperature=0.0,  # Most deterministic
    best_of=5,  # Try multiple decodings
    beam_size=5,  # Beam search
    patience=1.0,  # Beam search patience
    condition_on_previous_text=True,  # Use context
    word_timestamps=True,  # Detailed output
    initial_prompt="Context about your audio here...",
    fp16=True  # Use GPU if available
)

---

Balanced Speed/Accuracy Configuration

import whisper

model = whisper.load_model("medium")  # or "small"

result = model.transcribe(
    audio,
    language="en",  # Specify if known
    temperature=0.0,  # Deterministic
    best_of=1,  # Single decoding (faster)
    beam_size=5,  # Standard beam search
    condition_on_previous_text=True,  # Use context
    initial_prompt="Brief context if helpful...",
    fp16=True
)

---

Fast Configuration

import whisper

model = whisper.load_model("base")  # or "small"

result = model.transcribe(
    audio,
    language="en",  # Still specify for speed
    temperature=0.0,
    best_of=1,  # Single decoding
    beam_size=1,  # Greedy decoding (fastest)
    condition_on_previous_text=False,  # Skip context
    fp16=True
)

---

Noisy Audio Configuration

import whisper

model = whisper.load_model("large")  # Larger model for noise

result = model.transcribe(
    audio,
    language="en",
    temperature=0.0,
    best_of=5,  # Multiple attempts help with noise
    beam_size=5,
    condition_on_previous_text=True,
    initial_prompt="Phone call with background noise...",
    no_speech_threshold=0.4,  # More sensitive to speech
    logprob_threshold=-1.5,  # More permissive
    fp16=True
)

---

Multilingual Configuration

import whisper

model = whisper.load_model("medium")

# For transcription (keep original languages)
result = model.transcribe(
    audio,
    language=None,  # Auto-detect
    temperature=0.0,
    best_of=3,  # Helpful for language detection
    condition_on_previous_text=True,
    fp16=True
)

# For translation (to English)
result = model.transcribe(
    audio,
    task="translate",  # Translate to English
    temperature=0.0,
    best_of=3,
    fp16=True
)

---

Long-Form Audio Configuration

import whisper

model = whisper.load_model("medium")

# Process in chunks with context
def transcribe_long_audio(audio_path, chunk_length=60):
    # Split audio into chunks
    # Process each with context from previous
    result = model.transcribe(
        chunk_audio,
        temperature=0.0,
        best_of=1,  # Use 1 for speed on long files
        beam_size=5,
        condition_on_previous_text=True,  # Critical for long audio
        initial_prompt=previous_context,  # Pass previous text
        fp16=True
    )
    return result

---

Complete Optimal Settings Template

For Most Use Cases (Recommended)

import whisper

def transcribe_optimal(audio_path, model_size="medium", language="en", context=None):
    """
    Optimal Whisper transcription settings for most use cases.
    
    Args:
        audio_path: Path to audio file
        model_size: Model size ("small", "medium", "large")
        language: Language code or None for auto-detect
        context: Optional context string for initial_prompt
    """
    # Load model
    model = whisper.load_model(model_size)
    
    # Prepare parameters
    transcribe_params = {
        "language": language,
        "temperature": 0.0,  # Most deterministic
        "best_of": 5,  # Try multiple decodings
        "beam_size": 5,  # Beam search
        "patience": 1.0,  # Beam search patience
        "condition_on_previous_text": True,  # Use context
        "word_timestamps": True,  # Detailed output
        "fp16": True,  # Use GPU if available
    }
    
    # Add context if provided
    if context:
        transcribe_params["initial_prompt"] = context
    
    # Transcribe
    result = model.transcribe(audio_path, **transcribe_params)
    
    return result

# Usage
result = transcribe_optimal(
    "meeting.mp3",
    model_size="medium",
    language="en",
    context="Business meeting discussing Q4 product planning."
)

---

Settings Comparison Table

Setting	Default	Optimal (Accuracy)	Optimal (Speed)
Model	base	large/medium	base/small
Temperature	0.6	0.0	0.0
Best Of	1	5	1
Beam Size	5	5	1
Patience	1.0	1.0	1.0
Condition on Previous	True	True	False
Language	Auto	Specify	Specify
Initial Prompt	None	Provide	Optional
Word Timestamps	False	True	False

---

Performance Impact Summary

Setting Change	Accuracy Impact	Speed Impact
base → medium	+15-20%	-50%
base → large	+25-30%	-75%
best_of: 1 → 5	+3-8%	-80%
temperature: 0.6 → 0.0	+2-5%	No change
Specify language	+5-10%	+10% (faster)
Add initial_prompt	+5-15%	No change
condition_on_previous: True	+2-5%	-5%

---

Common Mistakes to Avoid

❌ Using Default Temperature

❌ Not Specifying Language

❌ Using Tiny Model for Important Content

❌ Disabling Context

❌ Not Using Initial Prompt

---

Best Practices Summary

1. ✅ Use medium model for best balance (or large for maximum accuracy)
2. ✅ Set temperature=0.0 for deterministic results
3. ✅ Use best_of=5 for critical content (or 1 for speed)
4. ✅ Set beam_size=5 for good accuracy
5. ✅ Enable condition_on_previous_text=True for context
6. ✅ Specify language when known (improves accuracy and speed)
7. ✅ Provide initial_prompt with domain context
8. ✅ Enable word_timestamps=True for detailed output
9. ✅ Use fp16=True if GPU available (faster)
10. ✅ Chunk long audio and use context between chunks

---

Quick Reference: Copy-Paste Configurations

Maximum Accuracy

model.transcribe(audio, 
    language="en", temperature=0.0, best_of=5, beam_size=5,
    condition_on_previous_text=True, word_timestamps=True,
    initial_prompt="Your context here")

Balanced

model.transcribe(audio,
    language="en", temperature=0.0, best_of=1, beam_size=5,
    condition_on_previous_text=True)

Fast

model.transcribe(audio,
    language="en", temperature=0.0, beam_size=1,
    condition_on_previous_text=False)

---

Conclusion

1. Model size (biggest impact)
2. Language specification (easy win)
3. Temperature (always use 0.0)
4. Best of (for critical content)
5. Initial prompt (for domain-specific content)

---