Whisper

Whisper is an open-source automatic speech recognition (ASR) system developed by OpenAI, trained on 680,000 hours of multilingual and multitask supervised data collected from the web. Released in September 2022, Whisper approaches human-level robustness and accuracy on English speech recognition and demonstrates strong performance across many languages, accents, and audio conditions without the need for fine-tuning. The model is designed as a general-purpose speech recognition system capable of multilingual speech recognition, speech translation to English, and language identification. Whisper employs a transformer-based encoder-decoder architecture and was trained using a large-scale weak supervision approach, learning from a diverse dataset of audio paired with transcripts sourced from the internet. This training methodology enables Whisper to handle a wide variety of audio conditions including background noise, technical jargon, accented speech, and low-quality recordings that challenge traditional ASR systems. The model is available in multiple sizes to accommodate different computational requirements: Tiny (39M parameters), Base (74M), Small (244M), Medium (769M), and Large (1.55B parameters), with larger models providing higher accuracy at the cost of increased inference time and memory usage. Whisper supports transcription and translation across approximately 100 languages, with particularly strong performance in English, Spanish, French, German, Italian, Portuguese, Japanese, Korean, Chinese, and many other widely spoken languages. The model includes automatic punctuation, capitalization, and timestamp generation at both the segment and word level. As open-source software released under the MIT license, Whisper can be run locally on consumer hardware with a compatible GPU, integrated into applications through Python, or accessed through OpenAI's commercial API for managed inference. The open-source release has spawned a large ecosystem of tools, integrations, and optimized variants including faster-whisper, whisper.cpp, and WhisperX that offer improved inference speed and additional features.