1 Speech-to-text (STT / ASR) pipelines
Imagine a super-fast typist with magic ears. You talk, and they instantly type down every word you say. They don't understand what you mean — they just turn the sounds coming out of your mouth into written words on a page. That magic-eared typist is what we call speech-to-text. It listens to noise and hands back a sentence you can read.
The first job in any voice system is turning sound into words. This is called Speech-to-Text (STT), also known as Automatic Speech Recognition (ASR) — the two terms mean the same thing. Your microphone captures sound as a stream of numbers (an audio waveform), and an ASR model converts that waveform into a string of text.
What an ASR model actually does
Sound is just air pressure wobbling over time. A microphone records those wobbles thousands of times per second as numbers. An ASR model takes that raw audio and works out which words were spoken. Modern ASR models — like OpenAI's Whisper, or services such as Deepgram, Google Speech-to-Text, and AssemblyAI — are themselves neural networks (often transformers, the same family of engine you met in Session 1). The rough internal steps are:
| # | Step | What happens |
|---|---|---|
| 1 | Capture audio | The mic records sound as a waveform — a long list of numbers sampled (e.g.) 16,000 times per second. |
| 2 | Feature extraction | The raw waveform is converted into a spectrogram — a picture of which sound frequencies are present over time. This is what the model "looks at." |
| 3 | Acoustic modelling | A neural network maps chunks of sound to likely speech units (sounds/letters/sub-words). |
| 4 | Decoding to text | Those units are assembled into the most likely words and sentence, often using a language model to choose between sound-alikes. |
| 5 | Formatting | Capitalisation, punctuation, and spacing are added so you get a clean sentence, not i love peanut butter. |
ASR is translation, not understanding. It converts the form of language (sounds) into another form (text). It has no idea what you mean — that's the LLM's job later. ASR just asks one question, very well: "Which words were spoken?"
Why ASR is hard — the real-world challenges
- Accents & dialects. The same word sounds different across speakers and regions. A model trained mostly on one accent struggles with others.
- Background noise. Traffic, music, a barking dog, or other people talking all blur the signal the model is trying to read.
- Homophones. "their / there / they're" or "to / two / too" sound identical — the model must use surrounding context to choose the right one.
- Punctuation & casing. You don't speak commas and full stops, so the model has to infer them. Bad punctuation can change meaning entirely.
- Domain words. Names, brands, medical or technical jargon ("Anthropic", "Kubernetes") are rare in training data and easily misheard.
- Crosstalk & overlap. When two people speak at once, the model must figure out who said what (called speaker diarization).
ASR quality is usually measured by Word Error Rate (WER) — the percentage of words it gets wrong (insertions, deletions, and substitutions). A WER of 5% means 1 in 20 words is off. Clean studio audio can hit very low WER; noisy phone calls with thick accents push it much higher.
You speak into your phone: "Book a table for two at eight."
- A good ASR returns:
Book a table for two at eight. - A weaker ASR in a noisy café might return:
book a table for to at ate— same sounds, wrong homophones, no punctuation.
Notice the second version is still readable, but if you feed it to the LLM brain, "for to at ate" could confuse the booking. This is why ASR accuracy directly affects how smart your whole voice agent feels — garbage in, garbage out.
2 The voice agent workflow
Talking to a voice assistant is like passing a note around a circle of friends. You say something (your mouth). One friend writes it down (ears → text). The next friend is the clever one who thinks of an answer (the brain). The last friend reads that answer out loud in a nice voice (mouth → speaker), and you hear it. The note goes all the way around the circle and comes back to you — out loud!
A voice agent is really just three machines wired together in a loop. The clever "brain" in the middle is exactly the LLM agent you met in Session 3 and Session 7 — it can still call tools, look things up, and reason. We've simply bolted ears (STT) on the front and a mouth (TTS) on the back. This is often called the STT → LLM → TTS pipeline (sometimes "the cascaded" or "pipeline" approach).
The full loop, end to end
Then the loop repeats: you reply, and round it goes again. Let's look at each box.
The three machines
| Stage | Turns… | …into | Job |
|---|---|---|---|
| STT (ears) | Audio | Text | Transcribe what you said. |
| LLM (brain) | Text | Text | Understand, reason, optionally call tools, and decide what to say back. |
| TTS (mouth) | Text | Audio | Speak the reply in a natural voice. |
What is TTS?
Text-to-Speech (TTS) is the mirror image of STT: it takes written text and produces audio of a human-sounding voice reading it. Old TTS sounded robotic and flat ("the… robot… voice"). Modern neural TTS (from providers like ElevenLabs, OpenAI, Cartesia, and others) is generated by neural networks and sounds remarkably natural — with correct rhythm, emphasis, and emotion. Key things TTS systems control:
- Voice / timbre. Which "person" is speaking — male, female, deep, bright. You can pick a preset voice or even clone a specific voice from a short sample.
- Prosody. The melody of speech — intonation, stress, and pacing. Good prosody is the difference between "are you OK?" (concerned) and "are you OK." (flat).
- Pronunciation. Handling tricky words, numbers ("$5" → "five dollars"), dates, and acronyms correctly.
- Emotion & style. Cheerful, calm, apologetic — useful for matching the situation.
The classic design is the cascaded pipeline above (three separate STT, LLM, TTS models). A newer approach is the speech-to-speech (or "realtime / multimodal") model that takes audio in and produces audio out directly, with no separate text step — for example OpenAI's Realtime API or Google's Gemini Live. Speech-to-speech can be faster and preserve tone better, but the cascaded pipeline is easier to debug, swap parts, and add tools to. Both are common in 2026.
You call a restaurant's AI line and say "Hi, can I book a table for two tomorrow at eight?"
- 🎤 → 📝 STT transcribes:
"Hi, can I book a table for two tomorrow at eight?" - 🧠 LLM understands the request, calls a
check_availabilitytool (the same tool-use idea from Session 7), finds a slot, and writes:"Sure! A table for two at 8 PM tomorrow is available. Shall I book it?" - 🔊 TTS reads that reply aloud in a warm, friendly voice.
- 📢 Speaker plays it down the phone line — and you hear it as if talking to a person.
The "intelligence" is all in the LLM brain; STT and TTS are the ears and mouth that let you reach it by voice instead of typing.
3 Real-time conversation: turn-taking, VAD & barge-in
When two people talk, there's an invisible game of "your turn / my turn." You wait for your friend to stop, then you speak. And if they're rambling and you suddenly remember something important, you cut in: "wait, wait —". A good voice assistant has to play this same game: know when you've stopped talking so it can answer, and politely stop talking itself if you interrupt it.
Transcribing words is the easy part. The thing that makes a voice agent feel alive is handling the natural rhythm of a real conversation — the back-and-forth, the pauses, the interruptions. This is called turn-taking, and it relies on a few key pieces.
Voice Activity Detection (VAD)
Voice Activity Detection (VAD) is a small, fast model whose only job is to answer: "Is someone speaking right now, yes or no?" It separates speech from silence and background noise. VAD is lightweight and runs constantly, so the system knows when audio is worth sending to the (heavier) ASR model and when it's just silence or room hum.
Endpointing — knowing when you've finished
The trickier question is endpointing: deciding the moment you've finished your turn so the agent can start replying. This is harder than it sounds, because a pause doesn't always mean you're done — you might just be thinking:
- Wait too short a time → the agent cuts you off mid-thought ("My order number is one two three… —Got it!— …four five six"). Annoying.
- Wait too long → there's an awkward silent gap before it replies, and it feels slow and unnatural.
Good endpointing balances these, often using both the length of silence and linguistic cues (a sentence that sounds grammatically complete is more likely finished than one ending in "and… um…").
VAD asks "is anyone talking?"; endpointing asks "has this person finished talking?". Getting endpointing right is one of the biggest factors in whether a voice agent feels responsive and human versus clumsy and frustrating.
Barge-in / interruptions
Barge-in is letting the user interrupt the agent while it is speaking — just like cutting into a human who's going on too long. When the system detects you've started talking over the agent (via VAD), it should immediately:
- Stop the TTS audio (go quiet at once, don't talk over you).
- Start listening to your new input.
- Update its memory to reflect that it was cut off — it should not assume you heard the whole sentence it was halfway through.
Without barge-in, the agent ploughs on talking while you're trying to speak — the classic "please listen carefully as our menu has recently changed" frustration. With it, the conversation feels respectful and natural.
If the agent hears its own voice from the speaker through the microphone, naive barge-in makes it interrupt itself. Real systems use acoustic echo cancellation (AEC) to subtract the agent's own audio so it only reacts to you.
Agent (reading aloud): "Sure, your options for delivery are standard, which takes three to five
business days, or express, which—"
You (cutting in): "Express, please."
- VAD detects you've started speaking.
- Barge-in instantly silences the TTS mid-word.
- Endpointing waits until you finish "please," then triggers the reply.
- The agent (now aware it was interrupted) responds: "Great, express it is."
That smooth cut-in is what separates a delightful voice agent from a robotic phone menu.
4 Streaming: don't wait for the whole thing
Imagine pouring water through a hose versus carrying it one full bucket at a time. With buckets, you wait and wait until a bucket is full before anything moves. With a hose, the water flows continuously — it starts coming out almost right away. Streaming is the hose: instead of waiting for the whole sentence to be ready, the system sends little pieces as soon as they exist, so things start happening immediately.
Remember from Session 1 that an LLM generates its answer one token at a time (autoregressive generation), which is why ChatGPT appears to "type out" its reply. Streaming takes that same idea and applies it to the whole voice pipeline — at every stage, work in small pieces instead of waiting for the full result.
Streaming the audio in (partial transcripts)
Instead of recording your entire sentence, stopping, and only then sending it to ASR, a streaming ASR receives your audio continuously as you speak. It emits partial transcripts — its best guess so far — that get refined as more words arrive:
As you say "What's the weather in Paris today?", a streaming ASR might emit:
what's thewhat's the weatherwhat's the weather in pears← early guess, wrong!what's the weather in paris today← corrected as context arrives
The early guesses let the system start working (and show live captions) before you've even finished. The final, stabilised transcript is what gets sent to the LLM.
Streaming the tokens/audio out
On the way back, two things stream:
- LLM tokens stream out as they're generated (exactly the token-by-token loop from Session 1) — you don't wait for the whole reply to be written.
- TTS streams too: as soon as the LLM has produced the first chunk of words (say, the first sentence or even the first few words), TTS can start speaking them while the LLM is still writing the rest. This is streamed TTS.
Streaming overlaps work that would otherwise happen one-after-another. Instead of listen-fully → transcribe-fully → think-fully → speak-fully (slow!), the stages run in a pipeline where each starts as soon as it has its first scrap of input. The user hears a reply beginning almost immediately, even though the agent hasn't finished "thinking" the whole thing.
In a chat window, waiting two seconds for a reply to appear is fine. In a spoken conversation, two seconds of total silence feels broken — humans expect a reply to start in well under a second. Streaming is what makes that possible: the agent can begin speaking before it has decided everything it's going to say, just like a person who starts a sentence before they've planned the end of it.
Starting TTS on the very first words is fast, but risky: if the LLM's later words change the meaning, or the first chunk is an awkward place to break a sentence, the speech can sound choppy. Systems often wait for a sensible boundary (a full clause or sentence) before speaking — a small delay for much smoother audio.
5 Latency optimization: making voice feel instant
If you say "hi!" to a friend and they take five whole seconds to say "hi" back, it feels weird and broken — even if they were just thinking. Talking only feels natural when the answer comes back almost right away. So a voice agent has to be fast — much faster than a chatbot you read. Every little delay along the way has to be trimmed down.
Latency is the delay between you finishing speaking and the agent starting to reply. For voice, this is make-or-break. Research and product experience put the comfortable target at roughly 500–800 milliseconds of response delay — humans in normal conversation reply in around 200ms, so even under a second already feels a touch slow. Anything over ~1.5 seconds feels broken. Hitting this budget is the central engineering challenge of voice agents.
Where does the delay come from?
The total delay is the sum of every stage in the pipeline. The big contributors:
- Endpointing delay — the wait to confirm you've actually stopped talking (Topic 3).
- STT time — transcribing your speech to text.
- LLM time-to-first-token (TTFT) — how long the brain takes to produce its first token. (Recall from Session 1 that "prefill" reads your prompt, then "decode" generates tokens; TTFT is the gap before the first decoded token appears.)
- TTS time — generating the first chunk of audio from the reply text.
- Network — round trips to the cloud servers and back, which add up across stages.
A sample latency budget
Here's an illustrative breakdown of where the milliseconds go in a cascaded pipeline, and a target for a snappy agent. (Exact numbers vary by model and network — these are realistic 2026 ballpark figures.)
| Stage | What's happening | Typical budget |
|---|---|---|
| Endpointing | Confirming you finished speaking | ~100–300 ms |
| STT (final) | Finalising the transcript | ~50–150 ms |
| Network hops | To/from the cloud between stages | ~50–150 ms |
| LLM time-to-first-token | Brain produces its first token | ~200–500 ms |
| TTS first audio | First chunk of speech generated | ~75–200 ms |
| Total to first sound | You hear the reply begin | ~500–800 ms |
Crucially, this is the time to the first sound, not the whole reply — thanks to streaming (Topic 4), the agent keeps generating the rest while it's already talking.
Techniques to cut latency
| Technique | How it helps |
|---|---|
| Streaming & pipelining | Overlap the stages (Topic 4) so STT, LLM, and TTS run at the same time instead of one-after-another. The single biggest win. |
| Smaller / faster models | Use a quicker, lighter LLM (or a distilled ASR/TTS model) for low-latency turns. Speed often matters more than a slightly smarter answer in conversation. |
| Optimise time-to-first-token | Shorter prompts, prompt caching, and fast inference servers reduce how long the LLM takes to emit its first token — what the user actually waits on. |
| Speculative / filler responses | Speak a quick acknowledgement ("Sure, let me check…") while the real answer or a tool call is still being computed, so the user is never met with silence. |
| Smart endpointing | Tighten the silence threshold using linguistic cues so the agent starts sooner without cutting people off. |
| Edge / co-location | Run models physically closer to the user (or co-locate STT, LLM, TTS) to shave network round-trips. |
Recall the speech-to-speech models from Topic 2: by skipping the separate text stages, they remove several hand-offs and network hops, which can cut latency noticeably and preserve tone. The trade-off is less control and harder debugging — so teams choose based on whether raw speed or flexibility matters more.
You ask the booking agent: "Is there anything available on Saturday night?" — which requires a slow database/tool lookup.
- Without optimisation: 2 seconds of dead silence, then the answer. Feels broken.
- With a speculative filler: the agent instantly says "Let me check Saturday for you…" (a quick, pre-computable phrase) while the tool call runs in the background, then continues "…yes, 9 PM is open." The total time is the same, but it feels responsive because there's no silence.
This is a favourite trick: humans tolerate a delay far better when something is happening than when it's silent.
★ Putting it all together
You just learned how to give an AI agent ears and a mouth. Here's the one-paragraph story that connects all five topics:
A voice agent is the LLM brain from Session 1 (still able to use tools, as in Sessions 3 & 7) wrapped in a microphone → STT → LLM → TTS → speaker loop. STT/ASR turns your audio into text (battling accents, noise, and homophones), and TTS turns the reply back into natural speech. To feel like a real conversation, the agent needs turn-taking: VAD hears whether you're speaking, endpointing knows when you've finished, and barge-in lets you interrupt it mid-sentence. To feel instant, everything streams — the same token-by-token idea from Session 1, now applied to the whole pipeline so partial transcripts come in and audio goes out before anything is fully finished. And because spoken silence feels broken, the agent fights latency with streaming, faster models, low time-to-first-token, and speculative fillers to start replying in well under a second.
Quick self-check
What's the difference between STT and TTS, and where do they sit in the pipeline?
STT (speech-to-text / ASR) turns your spoken audio into text and sits at the front (the ears). TTS (text-to-speech) turns the LLM's text reply back into spoken audio and sits at the back (the mouth). The LLM brain is in the middle.
VAD vs endpointing — what does each one decide?
VAD (voice activity detection) decides whether someone is speaking right now (speech vs silence). Endpointing decides when the speaker has finished their turn so the agent can reply. Endpointing too eager cuts people off; too patient feels laggy.
How does streaming here relate to the token-by-token generation from Session 1?
It's the same idea extended across the whole pipeline. The LLM still generates one token at a time, but now ASR also emits partial transcripts as you speak, and TTS starts speaking the first words while the LLM is still producing the rest — so stages overlap instead of waiting for each to fully finish.
What is barge-in, and why does it need echo cancellation?
Barge-in lets the user interrupt the agent while it's speaking — VAD detects the user talking and instantly stops the TTS so it listens. Echo cancellation (AEC) is needed so the agent doesn't hear its own voice through the mic and accidentally interrupt itself.
A user asks something that needs a slow tool call. How do you keep it from feeling broken?
Use a speculative filler — immediately speak a quick acknowledgement like "Let me check that for you…" while the tool call runs in the background, then continue with the answer. Humans tolerate delay far better when something is happening than during silence.
Why is low time-to-first-token (TTFT) so important for voice but less so for chat?
In voice, the user is waiting in real time and even a second of silence feels broken (target ~500–800ms to first sound). TTFT is the gap before the LLM's first token, which streaming turns into the first spoken word — so cutting TTFT directly cuts perceived delay. In a chat window, a short wait for text to appear is far more acceptable.
📚 References & Further Reading
Class material
- 📄 Original course notes / handout (source sheet) — open the shared GenAI class material for this session.
- Class handout: "Conversational & Voice-Based AI Agents".
Papers, docs & deep dives
- OpenAI Whisper — the open speech-to-text (ASR) model from Topic 1.
- OpenAI Realtime API guide — a speech-to-speech model for low-latency voice agents (Topic 2).
- OpenAI voice agents guide — building the STT → LLM → TTS pipeline end to end.
- WebRTC — the real-time audio transport that streams voice between browser and server.
- ElevenLabs documentation — neural text-to-speech (TTS) covering voices, prosody, and streaming.