ModelRefs / Build a Real-Time Voice AI — Tutorial
Build a Real-Time Voice AI — Tutorial
Wire a low-latency voice assistant by composing speech-to-text, an LLM, and text-to-speech behind a turn-taking pipeline. Covers What is a Voice AI Pipeline.
Overview
Wire a low-latency voice assistant by composing speech-to-text, an LLM, and text-to-speech behind a turn-taking pipeline.
Level: Expert. Estimated reading time: 55 minutes.
What is a Voice AI Pipeline?
A voice AI pipeline converts speech to text (ASR), processes it with an LLM, converts the response back to speech (TTS), and plays it back — all fast enough to feel like a real conversation.
The key engineering challenge: latency. Users expect to hear the first word of a response within 500–800 ms of finishing their sentence. Any delay above ~1.5 s feels broken.
The Three Components
1. ASR (Automatic Speech Recognition) • Converts audio → text transcript • Options: Whisper (OpenAI, open-source), Deepgram Nova-3, AssemblyAI • Key metrics: WER (Word Error Rate), latency to first token
2. LLM (Language Model) • Processes transcript → generates response text • Use streaming to minimize time-to-first-word • Keep context window small to reduce inference latency
3. TTS (Text-to-Speech) • Converts response text → audio • Options: ElevenLabs, OpenAI TTS, Cartesia (ultra-low latency) • Key metric: TTFA (Time To First Audio) — you want < 200 ms
Barge-In (Interruption Handling)
In a real conversation, users interrupt. Your voice AI must handle this.
Barge-in: when the user starts speaking while the assistant is still talking, stop playback immediately and process the new input.
Implementation: • Run ASR on the input stream continuously, even while TTS is playing • When ASR detects speech above a confidence threshold → cancel TTS playback → start new turn • Use a WebSocket connection to the client for low-latency bidirectional audio
Latency Budget
Target < 800 ms total latency per turn:
• ASR streaming: ~100–200 ms (first words start processing immediately) • LLM first token: ~200–400 ms (use a fast model; stream output) • TTS first audio: ~100–200 ms (stream from TTS as LLM tokens arrive) • Network overhead: ~50–100 ms
You hit sub-second TTFA by pipelining: start TTS as soon as the first sentence from the LLM arrives, don't wait for the full response.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Build a Real-Time Voice AI — Tutorial.