OpenAI Whisper - Robust Speech Recognition via Large-Scale Weak Supervision

This course examines OpenAI Whisper through its research paper and implementation. It covers weakly supervised multilingual training, evaluation, model architecture, mel spectrograms, language detection, transcription, voice activity detection, and decoding heuristics.

Syllabus

Intro
Paper overview
Collecting a large scale weakly supervised dataset
Evaluation metric issues WER
Effective robustness
Scaling laws in progress
Decoding is hacky
Code walk-through
Model architecture diagram vs code
Transcription task
Loading the audio, mel spectrograms
Language detection
Transcription task continued
Suppressing token logits
Voice activity detection
Decoding and heuristics
Outro