Articles / agentsvoicegenai-system-design
Design a real-time voice AI agent: latency, turn-taking, telephony and cost
A system design for real-time voice agents: cascade vs speech-to-speech, an 800 ms latency budget, barge-in, telephony, cost per minute and evals.
A voice agent is a chat agent with a stopwatch attached. The model, tools and prompts can be the same ones you use in a text agent, but the caller hears every millisecond you spend, talks over the agent when it rambles, and hangs up when it goes quiet. Most of the design work sits in the parts a text agent never needs: deciding when the user has finished, streaming audio through three services at once, stopping cleanly when interrupted, and getting audio in and out of a phone network.
The top search results for this topic are mostly vendor posts. They explain the STT, LLM and TTS pipeline well and some give a latency breakdown, but they rarely show cost per minute with the arithmetic, how speech-to-speech pricing grows with call length, what fails in production, or how to test any of it. This post covers those, with a small script whose numbers you can change.
Two architectures#
CASCADE (three models, text in the middle)
caller audio ──► VAD + turn detector ──► streaming STT ──► LLM (+ tools) ──► streaming TTS ──► caller
│ partial text tokens audio chunks
└──── barge-in signal cancels LLM and TTS, flushes playback ◄────────┘
SPEECH-TO-SPEECH (one model, audio in and audio out)
caller audio ──► realtime model (VAD, reasoning, voice, tool calls) ──► caller audio
│
└── tool call events ──► your backend ──► tool results back into the session
In the cascade, each stage is a separate service you can swap, log and test. You get a transcript for free, so guardrails, PII redaction and analytics work on text. The cost is latency: three network hops and three models that each need time to produce their first output.
Speech-to-speech models take audio and return audio. Kyutai's Moshi paper reports a theoretical latency of 160 ms and 200 ms in practice for its full-duplex model, which is below what a cascade can reach. Hosted options such as the OpenAI Realtime API also handle turn detection and interruption inside the session. You give up control: you can't pick a different STT for Hindi place names, you can't run a text guardrail before the agent speaks, and pricing works very differently, as the cost section shows.
I'd default to the cascade for anything that touches money, healthcare or regulated data, and consider speech-to-speech for open-ended conversation where tone and speed matter more than auditability.
The latency budget#
The target is time from the moment the user stops speaking to the moment they hear the first syllable of the reply. Human conversation sets the bar. In a study of turn-taking across ten languages, Stivers et al. found a mode of 0 ms and a mean of about 208 ms between one speaker ending and the next starting. No cascade gets close to that, so the common engineering target is about 800 ms, which still feels like a conversation rather than a walkie-talkie.
Here is the budget I use for a streaming cascade over WebRTC. Every figure is an estimate for a well-tuned setup with services in the same region, not a measurement:
| Stage | Estimate | Why this number |
|---|---|---|
| Uplink network + jitter buffer | 40 ms | Browser or app to a nearby media server |
| Endpointing | 300 ms | LiveKit's default minimum delay when its turn-detector model is on is 0.3 s (docs) |
| STT final transcript | 50 ms | Streaming STT has most words already; it only finalises the tail |
| LLM time to first token | 250 ms | Small model, cached prompt prefix; this is the stage with the widest spread |
| Tokens until the first clause | 40 ms | TTS needs a few words before it can start |
| TTS time to first audio | 90 ms | ElevenLabs states about 75 ms model latency for Flash (pricing page); I add network |
| Downlink network + playout buffer | 40 ms | |
| Total | 810 ms |
Two observations from this table. Endpointing is the biggest single item, bigger than the LLM, and it's pure waiting. And the LLM row is the one that blows the budget in practice: a large model, a long uncached prompt, or a tool call before the first token can each add hundreds of milliseconds.
The script at the end prints this budget and a second one for phone calls. Over PSTN, add carrier transit, a media gateway and G.711 transcoding; I assume 120 ms for that, which puts the same pipeline at about 930 ms. Phone calls start with less headroom.
Streaming at every stage#
The budget only works if no stage waits for the previous one to finish:
- STT streams partial transcripts while the user talks, so at end of turn it only finalises the last word or two.
- LLM streams tokens. You start the request the moment the turn ends, or earlier (see speculative starts below).
- Text to TTS. Cut the token stream at clause or sentence boundaries and send each piece to TTS as soon as it's complete. Sending single tokens makes prosody worse; waiting for the full reply adds the entire generation time.
- TTS streams audio chunks to the caller as they're produced.
Without streaming, the AssemblyAI example in its architecture post adds 500 ms of STT, 1 s of LLM and 400 ms of TTS end to end, for about 1.9 s before the caller hears anything.
Two tricks buy more headroom. A speculative start sends the transcript to the LLM when the turn detector is fairly sure the user is done, and throws the result away if they keep talking. It costs extra LLM calls. A filler ("Let me check that") played from a cache while a slow tool runs covers the gap without pretending the work is done.
Turn detection and barge-in#
Knowing when the user is done#
Voice activity detection tells you whether there is speech in a frame. Silero VAD, a common open-source choice, says a 30 ms chunk takes under 1 ms on one CPU thread and supports 8 kHz and 16 kHz audio, so it runs fine on telephone audio. VAD alone gives you a silence timer, and the silence timer is a bad trade: set it to 300 ms and the agent cuts in when someone pauses to read out a card number; set it to 1 s and every reply feels slow.
The fix is a turn model that looks at more than silence. OpenAI's Realtime API offers server_vad (silence-based, with silence_duration_ms) and semantic_vad, which judges whether the words form a complete utterance and has an eagerness setting (docs). LiveKit's turn detector does the same job on top of VAD; with it enabled, LiveKit drops its default minimum endpointing delay from 0.5 s to 0.3 s (docs). That 200 ms is the gain you are paying a model for.
Per-domain tuning matters more than the defaults. An agent that collects phone numbers, addresses or OTPs needs a lower eagerness during those slots than during chit-chat. I'd change the endpointing setting by conversation state rather than pick one value for the whole call.
Barge-in#
When the user starts talking over the agent, three things must happen fast, in this order:
user speech detected while agent audio is playing
│
├─ 1. stop playback: flush the client or telephony buffer (Twilio: send "clear")
├─ 2. cancel in-flight LLM and TTS requests
├─ 3. truncate the agent's last message in history to what was actually heard
└─ 4. go back to listening; the new user turn is the next input
Step 3 is the one people skip. If the agent said "Your balance is four thousand two hundred, and your next payment..." and was cut off, the history must hold only the heard part. Otherwise the LLM believes the user heard the payment date. Tracking what was heard needs playback markers: Twilio Media Streams has a mark message that reports when a given chunk finished playing, and clear discards queued audio (docs).
Not every sound is an interruption. Coughs, "mm-hmm" and TV noise trigger VAD. Require a minimum speech duration (a few hundred ms) or a couple of transcribed words before you cancel, and run echo cancellation so the agent's own voice coming back through a speakerphone doesn't interrupt itself.
Telephony: WebRTC vs SIP and PSTN#
| WebRTC (browser or app) | SIP trunk / PSTN (phone number) | |
|---|---|---|
| Audio | Opus, wideband (RFC 6716) | Usually G.711 at 8 kHz narrowband |
| Latency | Lower; you control the path | Higher; carrier and gateway hops |
| STT accuracy | Better on 16 kHz+ audio | Worse on 8 kHz, especially names and numbers |
| Echo and noise | Browser AEC built in | Speakerphones, cars, call-centre noise |
| Reach | Users who have your app or site | Anyone with a phone |
| Extra pieces | Media server (LiveKit, Daily, your own SFU) | Number provider, SIP trunk, media gateway, DTMF handling, call transfer |
Twilio's Media Streams sends and expects mu-law audio at 8 kHz, mono, base64 encoded over a WebSocket (docs). So the phone path means resampling for STT and encoding TTS output to mu-law at 8 kHz on the way back. Ask your TTS vendor for that format directly so you don't transcode twice.
Phone calls also bring features a web widget doesn't need: warm transfer to a human with a summary, DTMF input for PINs (better than speaking them), voicemail detection on outbound calls, and call recording with consent rules that differ by country.
Tool calls during a call#
Tool calls are where voice agents differ most from chat. A 2-second database lookup is fine in a chat window and painful on a call.
- Say something first. If a tool will take more than about 500 ms, have the agent speak a short acknowledgement while the call runs. Generate it from a fixed set, not from the LLM, so it costs no tokens and no latency.
- Keep tools fast or make them async. Pre-fetch what you can at call start (caller ID lookup, account summary) and put it in the prompt. For slow tasks (booking, refunds), confirm the request and send the result by SMS.
- Confirm before side effects. Read back the amount, date or address and wait for a yes. Speech errors turn "fifteen" into "fifty".
- Handle interruption during a tool call. If the user barges in while a refund API is running, you can cancel the speech but not always the API call. Make side-effecting tools idempotent with a request ID so a retried turn doesn't refund twice.
The guardrail and tool-design patterns from my agentic system design interview guide apply unchanged; only the time limits are tighter.
Cost per minute, with the math#
Prices change often, so treat these as worked examples with list prices I checked on 21 Nov 2026, and rerun the script with your own rates.
Assumptions for one minute of an inbound phone call: 4 exchanges per minute, the agent speaks 40% of the time, spoken English runs about 900 characters a minute, each LLM turn sends 2,500 input tokens (system prompt, tool schemas, average history) and gets 60 tokens back.
| Item | Price used | Math | $/min |
|---|---|---|---|
| Twilio US inbound local + Media Streams | $0.0085 + $0.0044 /min (pricing) | 0.0085 + 0.0044 | 0.0129 |
| STT, Deepgram Nova-3 streaming | $0.0048 /min, promo rate (pricing) | the whole minute of audio is streamed | 0.0048 |
| TTS, ElevenLabs Flash | $0.04 per 1k chars (pricing) | 0.4 × 900 = 360 chars → 0.36 × 0.04 | 0.0144 |
| LLM, gpt-4.1-mini | $0.40 in / $1.60 out per 1M (pricing) | 4 × (2,500 × 0.40 + 60 × 1.60) / 1M | 0.0044 |
| Total | ≈ 0.037 |
The LLM is the cheapest line. TTS and telephony dominate, which surprises people who come from text agents. A larger LLM changes that: at ten times the per-token price, the LLM becomes the biggest line at about $0.044 a minute.
Why speech-to-speech costs grow with call length#
OpenAI's Realtime API bills user audio at 1 token per 100 ms and agent audio at 1 token per 50 ms, and sends the entire conversation to the model for each response, so later turns cost more. With gpt-realtime list prices ($32 per 1M audio input, $0.40 cached, $64 output, $4 text input) and the same call shape, the script gives:
| Call length | No cache hits | 80% of history cached |
|---|---|---|
| 2 min | $0.18/min | $0.08/min |
| 5 min | $0.34/min | $0.12/min |
| 10 min | $0.61/min | $0.17/min |
These include the same $0.0129 telephony. Two things follow. Speech-to-speech here costs roughly 2 to 17 times the cascade per minute, depending on call length and caching. And the cost per minute rises the longer the call goes on, because each response re-reads a longer history. The Realtime docs note that caching is best-effort, that changing instructions or tools mid-session lowers the hit rate, and that truncation breaks the cache near the start of the conversation. If you choose speech-to-speech, keep the session prefix fixed and cap call length or history. The -mini realtime models cost about a third as much on audio.
Here is the script that produced both tables. It has no API calls, only arithmetic, so you can run it as is (tested with Python 3.15, standard library only):
# voice_budget.py: latency budget and cost-per-minute estimates for a voice agent.
# Every number below is an assumption or a list price checked on 2026-11-21. Swap in your own.
LATENCY_MS = { # user stops talking -> first agent audio at the caller's ear
"uplink network + jitter buffer": 40,
"endpointing (silence + turn model)": 300,
"STT final transcript after endpoint": 50,
"LLM time to first token": 250,
"LLM tokens until first clause": 40,
"TTS time to first audio byte": 90,
"downlink network + playout buffer": 40,
}
def latency_report(stages, extra=None):
stages = dict(stages, **(extra or {}))
total = 0
for name, ms in stages.items():
total += ms
print(f" {name:<40}{ms:>5} ms (running {total} ms)")
return total
# --- cascaded STT -> LLM -> TTS, one minute of call ---
TURNS_PER_MIN = 4 # assumption: ~15 s per user+agent exchange
AGENT_TALK_SHARE = 0.4 # agent speaks 40% of the minute
CHARS_PER_SPOKEN_MIN = 900 # ~150 words/min x ~6 chars incl. spaces
PROMPT_TOKENS = 2500 # system prompt + tool schemas + average history per turn
OUT_TOKENS = 60 # one or two short spoken sentences
PRICE = {
"twilio_inbound_min": 0.0085, "twilio_media_stream_min": 0.0044,
"stt_nova3_min": 0.0048, # Deepgram promo rate
"tts_per_1k_chars": 0.04, # ElevenLabs Flash/Turbo
"llm_in_per_m": 0.40, "llm_out_per_m": 1.60, # gpt-4.1-mini
"rt_audio_in_per_m": 32.0, "rt_audio_cached_per_m": 0.40, "rt_audio_out_per_m": 64.0,
"rt_text_in_per_m": 4.0, # gpt-realtime text input
}
def cascaded_per_min():
tel = PRICE["twilio_inbound_min"] + PRICE["twilio_media_stream_min"]
stt = PRICE["stt_nova3_min"] # streaming STT bills all audio sent
tts = AGENT_TALK_SHARE * CHARS_PER_SPOKEN_MIN / 1000 * PRICE["tts_per_1k_chars"]
llm = TURNS_PER_MIN * (PROMPT_TOKENS * PRICE["llm_in_per_m"] + OUT_TOKENS * PRICE["llm_out_per_m"]) / 1e6
return {"telephony": tel, "STT": stt, "TTS": tts, "LLM": llm}
def realtime_call_cost(minutes, cache_hit):
"""Speech-to-speech: the whole conversation is re-sent as input on every response."""
user_tok_per_turn = (1 - AGENT_TALK_SHARE) * 600 / TURNS_PER_MIN # 1 token / 100 ms of user audio
agent_tok_per_turn = AGENT_TALK_SHARE * 1200 / TURNS_PER_MIN # 1 token / 50 ms of agent audio
text_prompt = 2000
history, cost = 0.0, 0.0
for _ in range(int(minutes * TURNS_PER_MIN)):
new_in = user_tok_per_turn
cached = history * cache_hit
uncached = history * (1 - cache_hit) + new_in
cost += (uncached * PRICE["rt_audio_in_per_m"] + cached * PRICE["rt_audio_cached_per_m"]
+ text_prompt * PRICE["rt_text_in_per_m"] * (1 - cache_hit)
+ text_prompt * PRICE["rt_audio_cached_per_m"] * cache_hit
+ agent_tok_per_turn * PRICE["rt_audio_out_per_m"]) / 1e6
history += new_in + agent_tok_per_turn
tel = minutes * (PRICE["twilio_inbound_min"] + PRICE["twilio_media_stream_min"])
return (cost + tel) / minutes
if __name__ == "__main__":
print("WebRTC, streaming cascade:")
web = latency_report(LATENCY_MS)
print(f" TOTAL {web} ms\n")
print("Same pipeline over PSTN via a media gateway (assumed extra hops):")
pstn = latency_report(LATENCY_MS, {"carrier + gateway + G.711 transcoding": 120})
print(f" TOTAL {pstn} ms\n")
c = cascaded_per_min()
for k, v in c.items():
print(f" cascaded {k:<10} ${v:.4f}/min")
print(f" cascaded TOTAL ${sum(c.values()):.4f}/min\n")
for mins in (2, 5, 10):
for hit in (0.0, 0.8):
print(f" speech-to-speech, {mins:>2}-min call, cache hit {hit:.0%}: ${realtime_call_cost(mins, hit):.4f}/min")
Output (PSTN latency lines trimmed to the total):
WebRTC, streaming cascade:
uplink network + jitter buffer 40 ms (running 40 ms)
endpointing (silence + turn model) 300 ms (running 340 ms)
STT final transcript after endpoint 50 ms (running 390 ms)
LLM time to first token 250 ms (running 640 ms)
LLM tokens until first clause 40 ms (running 680 ms)
TTS time to first audio byte 90 ms (running 770 ms)
downlink network + playout buffer 40 ms (running 810 ms)
TOTAL 810 ms
Same pipeline over PSTN via a media gateway (assumed extra hops):
...
carrier + gateway + G.711 transcoding 120 ms (running 930 ms)
TOTAL 930 ms
cascaded telephony $0.0129/min
cascaded STT $0.0048/min
cascaded TTS $0.0144/min
cascaded LLM $0.0044/min
cascaded TOTAL $0.0365/min
speech-to-speech, 2-min call, cache hit 0%: $0.1812/min
speech-to-speech, 2-min call, cache hit 80%: $0.0839/min
speech-to-speech, 5-min call, cache hit 0%: $0.3425/min
speech-to-speech, 5-min call, cache hit 80%: $0.1177/min
speech-to-speech, 10-min call, cache hit 0%: $0.6113/min
speech-to-speech, 10-min call, cache hit 80%: $0.1742/min
The model is simple on purpose. It ignores silence (some vendors bill connection time, others only speech), retries, the cascade's own history growth, and volume discounts. For a cascade I'd also log real cost per call; my LLM cost and latency tracker shows the pattern.
Failure modes#
| Failure | What the caller experiences | Mitigation |
|---|---|---|
| Endpointing too eager | Agent cuts in mid-number or mid-thought | Semantic turn model; slower eagerness during slot filling |
| Endpointing too slow | Long silences, "hello?" | Turn model instead of a long silence timer; alert on p95 |
| Echo triggers barge-in | Agent stops itself mid-sentence on speakerphone | Echo cancellation; minimum speech duration before cancel |
| History not truncated on barge-in | Agent assumes the user heard things they didn't | Playback marks; store only the heard text |
| STT mishears names, numbers, codes | Wrong booking, wrong account | Keyword boosting, read-back confirmation, DTMF for PINs |
| Slow tool call | Dead air for seconds | Cached acknowledgement, pre-fetch, async with SMS follow-up |
| Duplicate side effects | Two refunds after a retried turn | Idempotency keys on every write tool |
| Vendor outage or latency spike | Call fails or stalls | Timeouts per stage, fallback STT/TTS vendor, transfer to human |
| Prompt injection by voice | Caller talks the agent into a forbidden action | Same tool permissions and policy checks as a text agent |
| Long-call cost creep (speech-to-speech) | Cost per minute rises through the call | Fixed prefix for caching, history caps, call time limit |
How to evaluate a voice agent#
Test the components and the whole call. A low WER with bad turn-taking still produces a bad call.
| What | Metric | How |
|---|---|---|
| STT | WER overall and on entities (names, numbers, addresses) | Recorded test calls with reference transcripts; jiwer computes WER. Use 8 kHz audio if you serve phone calls |
| Latency | p50, p95, p99 voice-to-voice, plus per stage | Timestamp end of user speech and first agent audio at the client; the median hides the turns people complain about |
| Turn-taking | False cut-in rate, mean time to respond after end of turn | Scripted callers who pause mid-sentence |
| Barge-in | Time from interruption to agent silence; was history truncated? | Scripted interruptions at random points in agent speech |
| Task success | Share of calls that complete the goal correctly | Simulated callers (an LLM playing a persona over TTS) against a checklist, plus human review of a sample of real calls |
| Safety | Wrong side effects, policy violations | Adversarial scripts; check tool calls, not only the words spoken |
| Cost | $ per call and per successful call | From billing logs, not estimates |
Measure latency at the caller's side. Server logs show when you sent audio, not when the caller heard it, and on phone calls the gap can be large. Simulated calls catch most regressions when you change the prompt, the turn settings or a vendor; the harness pattern in evaluating an AI agent in Python carries over once you add a TTS voice for the simulated user and an STT pass on the agent's replies.
Decision table#
| Situation | Choice | Reason |
|---|---|---|
| Regulated domain (banking, health, insurance) | Cascade | Transcript before speech, text guardrails, audit trail |
| Non-English or mixed-language callers, local names | Cascade | Pick the STT that wins on your audio; swap it alone |
| Lowest possible latency, open conversation, expressive voice | Speech-to-speech | One model, no text hop |
| High call volume, cost-sensitive | Cascade with a small LLM | Roughly $0.04 a minute in the example above |
| Long calls (10+ minutes) | Cascade, or speech-to-speech with strict history caps | Re-sent history makes long speech-to-speech calls expensive |
| Phone numbers, inbound support | SIP/PSTN with a media gateway | Reach, transfer to humans, DTMF |
| In-app assistant | WebRTC | Wideband audio, lower latency, browser echo cancellation |
| Many slow tool calls | Cascade, with acknowledgements and async results | Easier to control what is said while tools run |
For a first build I'd start with a cascade on an open-source orchestration framework such as Pipecat or LiveKit Agents, measure p95 voice-to-voice latency and task success on 50 scripted calls, and only then decide whether the latency gap justifies a speech-to-speech model. If the agent also needs to remember callers between calls, the read and write paths in my agent memory design apply, with the read path pre-fetched at call start so it adds nothing to turn latency.
Sources#
- Moshi: a speech-text foundation model for real-time dialogue, for full-duplex speech-to-speech and its 160 ms theoretical, 200 ms practical latency.
- Stivers et al., Universals and cultural variation in turn-taking in conversation, for human gap timing (mode 0 ms, mean about 208 ms).
- OpenAI Realtime API: voice activity detection, for
server_vad,semantic_vadand eagerness. - OpenAI Realtime API: managing costs, for audio tokenisation, history re-billing and caching behaviour.
- OpenAI API pricing, for gpt-realtime and gpt-4.1-mini prices.
- LiveKit turn detector docs, for endpointing defaults with and without the model.
- Silero VAD, for VAD speed and supported sample rates.
- Twilio Media Streams WebSocket messages, for mu-law 8 kHz audio and the
clearandmarkmessages. - Twilio US voice pricing, Deepgram pricing, ElevenLabs API pricing, for the cost table (checked 21 Nov 2026).
- RFC 6716: Definition of the Opus audio codec, the codec WebRTC uses for voice.
- AssemblyAI, voice agent architecture, for the non-streaming latency example.
- jiwer, Pipecat, LiveKit Agents, open-source tools mentioned above.