Speech to text works by turning your voice into numbers, turning those numbers into a picture of sound called a spectrogram, and running that picture through a neural network that predicts the most likely words. It's what runs every time you dictate a text, switch on captions, or ask Siri something.
The best models now get about 1 word in 100 wrong on clear audio. Meetings, background noise, strong accents, and jargon still trip them up, and that's where it gets interesting.
Here's how speech to text works, step by step, in plain English. It's one corner of natural language processing, the wider field of AI that deals with human language, so start there if you want the bigger picture.
What is speech to text?
Speech to text is software that turns spoken words into written text. It's also called speech recognition, voice to text, or automatic speech recognition (ASR). They all mean the same thing.
Speech to text only works out what was said. Working out who said it is a separate job called speaker recognition, which is what a voice-unlock feature does.
What is the difference between speech to text and text to speech?
Speech to text turns your voice into written words, and text to speech does the reverse: it reads written text out loud in a synthetic voice. Dictation, captions, and meeting transcripts use speech to text. Screen readers, sat nav directions, and AI voiceovers use text to speech. Voice assistants use both: speech to text to hear you, and text to speech to answer.
How does speech to text work, step by step?
Speech to text works in five steps: record the sound, turn it into a spectrogram, work out the speech sounds, pick the words that make sense, and tidy up the text. Older systems used a separate model for each step. Modern AI models like OpenAI's Whisper roll most of it into one neural network, but the steps are still the clearest way to see what's going on.
1. The microphone turns sound into numbers
A microphone turns changes in air pressure into an electrical signal. A chip called an analogue-to-digital converter then measures that signal thousands of times a second. Whisper, for example, works on audio sampled 16,000 times a second (Radford et al.). Some systems also trim silence and filter out steady background hum at this stage.
2. The audio becomes a spectrogram
A spectrogram is a picture of sound. Time runs left to right, pitch runs bottom to top, and brightness shows how loud each pitch is at each moment. The software builds it by chopping the audio into tiny overlapping slices and measuring the energy in each frequency band.
Whisper uses 25-millisecond slices taken every 10 milliseconds, with 80 frequency bands on the Mel scale, which spaces pitches the way human hearing does (Radford et al.). Older guides call this step "feature extraction". The model never hears your voice the way you do. It reads this picture.
3. An acoustic model works out the speech sounds
The acoustic model matches slices of the spectrogram to speech sounds, called phonemes. A phoneme is the smallest sound that changes a word: "pat" and "bat" differ by one phoneme.
Acoustic models are neural networks trained on thousands of hours of recorded speech with matching human transcripts. They output probabilities, not certainties. This slice is probably a "b", but it might be a "p".
4. A language model picks the words that make sense
The language model decides which string of words is most likely, given the sounds. "Recognise speech" and "wreck a nice beach" sound almost the same out loud. The language model knows the first is far more common, so it wins unless the rest of the sentence says otherwise.
The language model also picks between "their", "there", and "they're", which sound identical. A decoder searches through the possible combinations of sounds and words and keeps the most probable transcript.
5. Clean-up adds punctuation, numbers, and speaker labels
The last step turns raw words into readable text. It adds punctuation and capital letters, and turns "twenty dollars" into "$20". It can also add timestamps for captions and labels for who said what. Labelling speakers is called diarisation, and it's what lets a meeting transcript say "Speaker 2" instead of one long blob.
How do modern AI speech recognition models like Whisper work?
Modern AI speech recognition models like OpenAI's Whisper are end-to-end: one neural network takes the spectrogram in and writes the text out. There's no separate acoustic model, language model, and clean-up step.
Whisper is an encoder-decoder transformer, the same family of neural network behind ChatGPT. The encoder reads 30 seconds of spectrogram at a time. The decoder then writes the transcript a few letters at a time, punctuation and timestamps included (Radford et al.).
The big change is scale. OpenAI trained Whisper on 680,000 hours of audio paired with transcripts from the internet, including 117,000 hours in 96 languages other than English. That's why it copes with accents, background noise, and technical words better than older systems trained on tidy recordings. Whisper's code and model weights are free to use under the MIT licence (GitHub).
Not every model waits for a 30-second chunk. Streaming models type words as you speak. Google's on-device recogniser for Gboard, for example, uses a recurrent neural network transducer that outputs characters one by one as you talk (Google Research).
The team behind Hugging Face's Open ASR Leaderboard tested 86 speech recognition systems. The most accurate ones paired a Conformer encoder with a transformer decoder, while simpler decoders traded some accuracy for much more speed (Srivastav et al.).
How accurate is speech to text?
Speech to text is very accurate on clear audio and much less accurate on messy audio. On Hugging Face's Open ASR Leaderboard, the best models get about 1 word in 100 wrong on clean audiobook recordings and about 6 in 100 on recorded meetings. OpenAI's popular Whisper large-v3 gets nearly 14 in 100 wrong on meetings.
| Audio | OpenAI Whisper large-v3 | ElevenLabs Scribe v2 | AssemblyAI Universal-3.5 Pro |
|---|---|---|---|
| Clean audiobooks (LibriSpeech) | 1.56% | 1.06% | 1.1% |
| Earnings calls with many accents (Earnings-22) | 7.99% | 4.8% | 6.05% |
| Recorded meetings (AMI) | 13.63% | 8.19% | 10.63% |
Word error rates from the Open ASR Leaderboard. Lower is better.
The pattern matters more than any one number. The same model makes several times more mistakes on a meeting than on an audiobook. LibriSpeech is about 1,000 hours of people reading audiobooks aloud. Earnings-22 is 119 hours of company earnings calls from around the world, built to test accents. The AMI corpus is 100 hours of meetings recorded on close-talking and far-off microphones.
What is word error rate in speech recognition?
Word error rate (WER) is the standard measure of speech recognition accuracy. It counts the words the software swapped, dropped, or added, then divides by the number of words actually spoken (Koenecke et al.).
Say 100 words and get back a transcript with 3 wrong words, 1 missing word, and 1 extra word. That's 5 errors, or a 5% WER. A vendor's "95% accuracy" usually means the same thing, so always ask what audio it was measured on.
Is speech to text as accurate as a human?
On some tests, yes. In 2017 Microsoft's system hit a 5.1% WER on Switchboard, a set of recorded phone calls between strangers, matching professional human transcribers (Microsoft Research). Even then, Microsoft said noisy rooms, distant microphones, and accented speech were still unsolved.
What makes speech to text get words wrong?
Speech to text gets words wrong when the audio doesn't look like what the model was trained on. The usual culprits are accents, background noise, people talking over each other, rare words, and long silences.
Accents and dialects
Accents and dialects change how speech sounds, and speech to text copes worst with the ones it heard least in training. A Stanford study tested systems from Amazon, Apple, Google, IBM, and Microsoft and found an average WER of 35% for Black American speakers against 19% for white speakers (Koenecke et al.). The researchers traced the gap to the acoustic models, because it held even when both groups said identical phrases.
I'm Australian, so here's the local version. A Macquarie University study tested Google's American English and Australian English recognisers and found each was more accurate on speakers of its own accent (Szalay et al.). The American model stumbled because Australian English has more vowel sounds, and because we drop the "r" after a vowel, so "car" and "park" don't sound the way it expects. If your phone or app offers English (Australia) as a language, pick it.
Background noise and crosstalk
Background noise and crosstalk smear extra sound across the spectrogram, so the model has to pull one voice out of the mess. That's a big part of why the best models on the Open ASR Leaderboard make several times more mistakes on recorded meetings than on audiobooks. When two people talk at once, their voices overlap in the same picture, and the transcript often drops words or gives them to the wrong speaker. A microphone close to the speaker's mouth fixes more than any software setting.
Jargon, names, and slang
Rare words trip up the language model, because it favours words it has seen often. Medical terms, product names, surnames, and Aussie slang like "arvo" or "servo" are the kind of words that get swapped for something more common. Many services now let you give the model a list of words to expect. ElevenLabs Scribe, for example, accepts up to 1,000 key terms (ElevenLabs).
Silence and made-up words
Models that write text like a chatbot can also invent words nobody said. In a Cornell-led study of an earlier version of OpenAI's Whisper, roughly 1% of transcriptions contained whole made-up phrases or sentences, and 38% of those hallucinations included explicit harms like violent language (Koenecke et al.). They turned up most for speakers with long pauses.
That's one reason AI medical scribes, which run speech to text before an AI writes up the doctor's notes, still need a human to check every note.
Does speech to text run on your device or in the cloud?
Speech to text runs both ways: phone dictation increasingly runs on the device, while meeting apps and transcription services send your audio to their servers.
- On the device. Apple says iPhone dictation is processed on the phone in many languages, with no internet connection needed (Apple). Google moved Gboard voice typing on Pixel phones onto the device in 2019, with a model of about 80MB that works offline (Google Research). Whisper runs on your own computer, so the audio never leaves it.
- In the cloud. Otter, Deepgram, AssemblyAI, ElevenLabs, and Google Cloud run bigger models on their own servers, with extras like speaker labels and summaries. Windows voice typing (Windows key + H) is cloud-based too: Microsoft says it needs an internet connection and uses its Azure speech service (Microsoft).
On-device speech to text is the private option, because your voice stays on your phone or computer. With a cloud service, check what it keeps, for how long, and whether it trains its AI on your recordings. Otter's privacy policy, for example, says it trains its AI on de-identified audio recordings (Otter). For health, legal, or HR conversations, that's worth knowing before you hit record.
Where is speech to text used?
Speech to text is used anywhere people talk and someone needs the words in writing. The big ones:
- Dictation. The microphone key on iPhone and Android keyboards, and voice typing on Windows and Mac.
- Captions and subtitles. YouTube's automatic captions and the AI caption generators creators use for TikTok and Reels.
- AI medical scribes. They transcribe a consult, then an AI drafts the clinical note for the doctor to check.
- Voice assistants. Siri, Alexa, and Google's assistant turn your request into text before working out what you want.
- Meeting notes. Otter and the note-takers built into Zoom and Teams transcribe calls and summarise them.
- Accessibility. Live captions help people who are deaf or hard of hearing, and dictation helps people who can't easily type.
Which speech to text tools can you use?
For everyday dictation, use the free tool built into your phone or computer, and for transcribing recordings or building an app, use a paid service. Here are the main ones.
| Tool | What it's for | Where it runs | Price |
|---|---|---|---|
| Apple Dictation | Dictating on iPhone | On the phone in many languages | Free |
| Gboard voice typing | Dictating on Android | On the phone (Pixel) or Google's servers | Free |
| OpenAI Whisper | Free open-source model | Your own computer, or OpenAI's API | Free to run yourself; API US$0.006 a minute (OpenAI) |
| Otter | Meeting notes | Cloud | Free for 300 minutes a month; Pro US$16.99 a month, or US$8.33 a month billed yearly |
| ElevenLabs Scribe v2 | Transcribing recordings and live audio | Cloud | US$0.22 an hour of audio (US$0.39 for Realtime) |
| Deepgram Nova-3 | Developer API | Cloud | US$0.0043 a minute for recordings |
| AssemblyAI Universal-3.5 Pro | Developer API | Cloud | US$0.21 an hour for recordings |
| Google Cloud Speech-to-Text | Developer API | Cloud | US$0.016 a minute |
Prices come from each company's own pricing page.
