GyaanStack. Blog

Articles / agentsvoicegenai-system-design

Design a real-time voice AI agent: latency, turn-taking, telephony and cost

A system design for real-time voice agents: cascade vs speech-to-speech, an 800 ms latency budget, barge-in, telephony, cost per minute and evals.

By Raghunath · · 18 min read

A voice agent is a chat agent with a stopwatch attached. The model, tools and prompts can be the same ones you use in a text agent, but the caller hears every millisecond you spend, talks over the agent when it rambles, and hangs up when it goes quiet. Most of the design work sits in the parts a text agent never needs: deciding when the user has finished, streaming audio through three services at once, stopping cleanly when interrupted, and getting audio in and out of a phone network.

The top search results for this topic are mostly vendor posts. They explain the STT, LLM and TTS pipeline well and some give a latency breakdown, but they rarely show cost per minute with the arithmetic, how speech-to-speech pricing grows with call length, what fails in production, or how to test any of it. This post covers those, with a small script whose numbers you can change.

Two architectures#

CASCADE (three models, text in the middle)

 caller audio ──► VAD + turn detector ──► streaming STT ──► LLM (+ tools) ──► streaming TTS ──► caller
                       │                     partial text      tokens            audio chunks
                       └──── barge-in signal cancels LLM and TTS, flushes playback ◄────────┘

SPEECH-TO-SPEECH (one model, audio in and audio out)

 caller audio ──► realtime model (VAD, reasoning, voice, tool calls) ──► caller audio
                       │
                       └── tool call events ──► your backend ──► tool results back into the session

In the cascade, each stage is a separate service you can swap, log and test. You get a transcript for free, so guardrails, PII redaction and analytics work on text. The cost is latency: three network hops and three models that each need time to produce their first output.

Speech-to-speech models take audio and return audio. Kyutai's Moshi paper reports a theoretical latency of 160 ms and 200 ms in practice for its full-duplex model, which is below what a cascade can reach. Hosted options such as the OpenAI Realtime API also handle turn detection and interruption inside the session. You give up control: you can't pick a different STT for Hindi place names, you can't run a text guardrail before the agent speaks, and pricing works very differently, as the cost section shows.

I'd default to the cascade for anything that touches money, healthcare or regulated data, and consider speech-to-speech for open-ended conversation where tone and speed matter more than auditability.

The latency budget#

The target is time from the moment the user stops speaking to the moment they hear the first syllable of the reply. Human conversation sets the bar. In a study of turn-taking across ten languages, Stivers et al. found a mode of 0 ms and a mean of about 208 ms between one speaker ending and the next starting. No cascade gets close to that, so the common engineering target is about 800 ms, which still feels like a conversation rather than a walkie-talkie.

Here is the budget I use for a streaming cascade over WebRTC. Every figure is an estimate for a well-tuned setup with services in the same region, not a measurement:

Stage Estimate Why this number
Uplink network + jitter buffer 40 ms Browser or app to a nearby media server
Endpointing 300 ms LiveKit's default minimum delay when its turn-detector model is on is 0.3 s (docs)
STT final transcript 50 ms Streaming STT has most words already; it only finalises the tail
LLM time to first token 250 ms Small model, cached prompt prefix; this is the stage with the widest spread
Tokens until the first clause 40 ms TTS needs a few words before it can start
TTS time to first audio 90 ms ElevenLabs states about 75 ms model latency for Flash (pricing page); I add network
Downlink network + playout buffer 40 ms
Total 810 ms

Two observations from this table. Endpointing is the biggest single item, bigger than the LLM, and it's pure waiting. And the LLM row is the one that blows the budget in practice: a large model, a long uncached prompt, or a tool call before the first token can each add hundreds of milliseconds.

The script at the end prints this budget and a second one for phone calls. Over PSTN, add carrier transit, a media gateway and G.711 transcoding; I assume 120 ms for that, which puts the same pipeline at about 930 ms. Phone calls start with less headroom.

Streaming at every stage#

The budget only works if no stage waits for the previous one to finish:

  • STT streams partial transcripts while the user talks, so at end of turn it only finalises the last word or two.
  • LLM streams tokens. You start the request the moment the turn ends, or earlier (see speculative starts below).
  • Text to TTS. Cut the token stream at clause or sentence boundaries and send each piece to TTS as soon as it's complete. Sending single tokens makes prosody worse; waiting for the full reply adds the entire generation time.
  • TTS streams audio chunks to the caller as they're produced.

Without streaming, the AssemblyAI example in its architecture post adds 500 ms of STT, 1 s of LLM and 400 ms of TTS end to end, for about 1.9 s before the caller hears anything.

Two tricks buy more headroom. A speculative start sends the transcript to the LLM when the turn detector is fairly sure the user is done, and throws the result away if they keep talking. It costs extra LLM calls. A filler ("Let me check that") played from a cache while a slow tool runs covers the gap without pretending the work is done.

Turn detection and barge-in#

Knowing when the user is done#

Voice activity detection tells you whether there is speech in a frame. Silero VAD, a common open-source choice, says a 30 ms chunk takes under 1 ms on one CPU thread and supports 8 kHz and 16 kHz audio, so it runs fine on telephone audio. VAD alone gives you a silence timer, and the silence timer is a bad trade: set it to 300 ms and the agent cuts in when someone pauses to read out a card number; set it to 1 s and every reply feels slow.

The fix is a turn model that looks at more than silence. OpenAI's Realtime API offers server_vad (silence-based, with silence_duration_ms) and semantic_vad, which judges whether the words form a complete utterance and has an eagerness setting (docs). LiveKit's turn detector does the same job on top of VAD; with it enabled, LiveKit drops its default minimum endpointing delay from 0.5 s to 0.3 s (docs). That 200 ms is the gain you are paying a model for.

Per-domain tuning matters more than the defaults. An agent that collects phone numbers, addresses or OTPs needs a lower eagerness during those slots than during chit-chat. I'd change the endpointing setting by conversation state rather than pick one value for the whole call.

Barge-in#

When the user starts talking over the agent, three things must happen fast, in this order:

user speech detected while agent audio is playing
   │
   ├─ 1. stop playback: flush the client or telephony buffer (Twilio: send "clear")
   ├─ 2. cancel in-flight LLM and TTS requests
   ├─ 3. truncate the agent's last message in history to what was actually heard
   └─ 4. go back to listening; the new user turn is the next input

Step 3 is the one people skip. If the agent said "Your balance is four thousand two hundred, and your next payment..." and was cut off, the history must hold only the heard part. Otherwise the LLM believes the user heard the payment date. Tracking what was heard needs playback markers: Twilio Media Streams has a mark message that reports when a given chunk finished playing, and clear discards queued audio (docs).

Not every sound is an interruption. Coughs, "mm-hmm" and TV noise trigger VAD. Require a minimum speech duration (a few hundred ms) or a couple of transcribed words before you cancel, and run echo cancellation so the agent's own voice coming back through a speakerphone doesn't interrupt itself.

Telephony: WebRTC vs SIP and PSTN#

WebRTC (browser or app) SIP trunk / PSTN (phone number)
Audio Opus, wideband (RFC 6716) Usually G.711 at 8 kHz narrowband
Latency Lower; you control the path Higher; carrier and gateway hops
STT accuracy Better on 16 kHz+ audio Worse on 8 kHz, especially names and numbers
Echo and noise Browser AEC built in Speakerphones, cars, call-centre noise
Reach Users who have your app or site Anyone with a phone
Extra pieces Media server (LiveKit, Daily, your own SFU) Number provider, SIP trunk, media gateway, DTMF handling, call transfer

Twilio's Media Streams sends and expects mu-law audio at 8 kHz, mono, base64 encoded over a WebSocket (docs). So the phone path means resampling for STT and encoding TTS output to mu-law at 8 kHz on the way back. Ask your TTS vendor for that format directly so you don't transcode twice.

Phone calls also bring features a web widget doesn't need: warm transfer to a human with a summary, DTMF input for PINs (better than speaking them), voicemail detection on outbound calls, and call recording with consent rules that differ by country.

Tool calls during a call#

Tool calls are where voice agents differ most from chat. A 2-second database lookup is fine in a chat window and painful on a call.

  • Say something first. If a tool will take more than about 500 ms, have the agent speak a short acknowledgement while the call runs. Generate it from a fixed set, not from the LLM, so it costs no tokens and no latency.
  • Keep tools fast or make them async. Pre-fetch what you can at call start (caller ID lookup, account summary) and put it in the prompt. For slow tasks (booking, refunds), confirm the request and send the result by SMS.
  • Confirm before side effects. Read back the amount, date or address and wait for a yes. Speech errors turn "fifteen" into "fifty".
  • Handle interruption during a tool call. If the user barges in while a refund API is running, you can cancel the speech but not always the API call. Make side-effecting tools idempotent with a request ID so a retried turn doesn't refund twice.

The guardrail and tool-design patterns from my agentic system design interview guide apply unchanged; only the time limits are tighter.

Cost per minute, with the math#

Prices change often, so treat these as worked examples with list prices I checked on 21 Nov 2026, and rerun the script with your own rates.

Assumptions for one minute of an inbound phone call: 4 exchanges per minute, the agent speaks 40% of the time, spoken English runs about 900 characters a minute, each LLM turn sends 2,500 input tokens (system prompt, tool schemas, average history) and gets 60 tokens back.

Item Price used Math $/min
Twilio US inbound local + Media Streams $0.0085 + $0.0044 /min (pricing) 0.0085 + 0.0044 0.0129
STT, Deepgram Nova-3 streaming $0.0048 /min, promo rate (pricing) the whole minute of audio is streamed 0.0048
TTS, ElevenLabs Flash $0.04 per 1k chars (pricing) 0.4 × 900 = 360 chars → 0.36 × 0.04 0.0144
LLM, gpt-4.1-mini $0.40 in / $1.60 out per 1M (pricing) 4 × (2,500 × 0.40 + 60 × 1.60) / 1M 0.0044
Total ≈ 0.037

The LLM is the cheapest line. TTS and telephony dominate, which surprises people who come from text agents. A larger LLM changes that: at ten times the per-token price, the LLM becomes the biggest line at about $0.044 a minute.

Why speech-to-speech costs grow with call length#

OpenAI's Realtime API bills user audio at 1 token per 100 ms and agent audio at 1 token per 50 ms, and sends the entire conversation to the model for each response, so later turns cost more. With gpt-realtime list prices ($32 per 1M audio input, $0.40 cached, $64 output, $4 text input) and the same call shape, the script gives:

Call length No cache hits 80% of history cached
2 min $0.18/min $0.08/min
5 min $0.34/min $0.12/min
10 min $0.61/min $0.17/min

These include the same $0.0129 telephony. Two things follow. Speech-to-speech here costs roughly 2 to 17 times the cascade per minute, depending on call length and caching. And the cost per minute rises the longer the call goes on, because each response re-reads a longer history. The Realtime docs note that caching is best-effort, that changing instructions or tools mid-session lowers the hit rate, and that truncation breaks the cache near the start of the conversation. If you choose speech-to-speech, keep the session prefix fixed and cap call length or history. The -mini realtime models cost about a third as much on audio.

Here is the script that produced both tables. It has no API calls, only arithmetic, so you can run it as is (tested with Python 3.15, standard library only):

# voice_budget.py: latency budget and cost-per-minute estimates for a voice agent.
# Every number below is an assumption or a list price checked on 2026-11-21. Swap in your own.

LATENCY_MS = {                     # user stops talking -> first agent audio at the caller's ear
    "uplink network + jitter buffer": 40,
    "endpointing (silence + turn model)": 300,
    "STT final transcript after endpoint": 50,
    "LLM time to first token": 250,
    "LLM tokens until first clause": 40,
    "TTS time to first audio byte": 90,
    "downlink network + playout buffer": 40,
}

def latency_report(stages, extra=None):
    stages = dict(stages, **(extra or {}))
    total = 0
    for name, ms in stages.items():
        total += ms
        print(f"  {name:<40}{ms:>5} ms   (running {total} ms)")
    return total

# --- cascaded STT -> LLM -> TTS, one minute of call ---
TURNS_PER_MIN = 4          # assumption: ~15 s per user+agent exchange
AGENT_TALK_SHARE = 0.4     # agent speaks 40% of the minute
CHARS_PER_SPOKEN_MIN = 900 # ~150 words/min x ~6 chars incl. spaces
PROMPT_TOKENS = 2500       # system prompt + tool schemas + average history per turn
OUT_TOKENS = 60            # one or two short spoken sentences

PRICE = {
    "twilio_inbound_min": 0.0085, "twilio_media_stream_min": 0.0044,
    "stt_nova3_min": 0.0048,                   # Deepgram promo rate
    "tts_per_1k_chars": 0.04,                  # ElevenLabs Flash/Turbo
    "llm_in_per_m": 0.40, "llm_out_per_m": 1.60,   # gpt-4.1-mini
    "rt_audio_in_per_m": 32.0, "rt_audio_cached_per_m": 0.40, "rt_audio_out_per_m": 64.0,
    "rt_text_in_per_m": 4.0,                   # gpt-realtime text input
}

def cascaded_per_min():
    tel = PRICE["twilio_inbound_min"] + PRICE["twilio_media_stream_min"]
    stt = PRICE["stt_nova3_min"]                    # streaming STT bills all audio sent
    tts = AGENT_TALK_SHARE * CHARS_PER_SPOKEN_MIN / 1000 * PRICE["tts_per_1k_chars"]
    llm = TURNS_PER_MIN * (PROMPT_TOKENS * PRICE["llm_in_per_m"] + OUT_TOKENS * PRICE["llm_out_per_m"]) / 1e6
    return {"telephony": tel, "STT": stt, "TTS": tts, "LLM": llm}

def realtime_call_cost(minutes, cache_hit):
    """Speech-to-speech: the whole conversation is re-sent as input on every response."""
    user_tok_per_turn = (1 - AGENT_TALK_SHARE) * 600 / TURNS_PER_MIN   # 1 token / 100 ms of user audio
    agent_tok_per_turn = AGENT_TALK_SHARE * 1200 / TURNS_PER_MIN       # 1 token / 50 ms of agent audio
    text_prompt = 2000
    history, cost = 0.0, 0.0
    for _ in range(int(minutes * TURNS_PER_MIN)):
        new_in = user_tok_per_turn
        cached = history * cache_hit
        uncached = history * (1 - cache_hit) + new_in
        cost += (uncached * PRICE["rt_audio_in_per_m"] + cached * PRICE["rt_audio_cached_per_m"]
                 + text_prompt * PRICE["rt_text_in_per_m"] * (1 - cache_hit)
                 + text_prompt * PRICE["rt_audio_cached_per_m"] * cache_hit
                 + agent_tok_per_turn * PRICE["rt_audio_out_per_m"]) / 1e6
        history += new_in + agent_tok_per_turn
    tel = minutes * (PRICE["twilio_inbound_min"] + PRICE["twilio_media_stream_min"])
    return (cost + tel) / minutes

if __name__ == "__main__":
    print("WebRTC, streaming cascade:")
    web = latency_report(LATENCY_MS)
    print(f"  TOTAL {web} ms\n")
    print("Same pipeline over PSTN via a media gateway (assumed extra hops):")
    pstn = latency_report(LATENCY_MS, {"carrier + gateway + G.711 transcoding": 120})
    print(f"  TOTAL {pstn} ms\n")
    c = cascaded_per_min()
    for k, v in c.items():
        print(f"  cascaded {k:<10} ${v:.4f}/min")
    print(f"  cascaded TOTAL     ${sum(c.values()):.4f}/min\n")
    for mins in (2, 5, 10):
        for hit in (0.0, 0.8):
            print(f"  speech-to-speech, {mins:>2}-min call, cache hit {hit:.0%}: ${realtime_call_cost(mins, hit):.4f}/min")

Output (PSTN latency lines trimmed to the total):

WebRTC, streaming cascade:
  uplink network + jitter buffer             40 ms   (running 40 ms)
  endpointing (silence + turn model)        300 ms   (running 340 ms)
  STT final transcript after endpoint        50 ms   (running 390 ms)
  LLM time to first token                   250 ms   (running 640 ms)
  LLM tokens until first clause              40 ms   (running 680 ms)
  TTS time to first audio byte               90 ms   (running 770 ms)
  downlink network + playout buffer          40 ms   (running 810 ms)
  TOTAL 810 ms

Same pipeline over PSTN via a media gateway (assumed extra hops):
  ...
  carrier + gateway + G.711 transcoding     120 ms   (running 930 ms)
  TOTAL 930 ms

  cascaded telephony  $0.0129/min
  cascaded STT        $0.0048/min
  cascaded TTS        $0.0144/min
  cascaded LLM        $0.0044/min
  cascaded TOTAL     $0.0365/min

  speech-to-speech,  2-min call, cache hit 0%: $0.1812/min
  speech-to-speech,  2-min call, cache hit 80%: $0.0839/min
  speech-to-speech,  5-min call, cache hit 0%: $0.3425/min
  speech-to-speech,  5-min call, cache hit 80%: $0.1177/min
  speech-to-speech, 10-min call, cache hit 0%: $0.6113/min
  speech-to-speech, 10-min call, cache hit 80%: $0.1742/min

The model is simple on purpose. It ignores silence (some vendors bill connection time, others only speech), retries, the cascade's own history growth, and volume discounts. For a cascade I'd also log real cost per call; my LLM cost and latency tracker shows the pattern.

Failure modes#

Failure What the caller experiences Mitigation
Endpointing too eager Agent cuts in mid-number or mid-thought Semantic turn model; slower eagerness during slot filling
Endpointing too slow Long silences, "hello?" Turn model instead of a long silence timer; alert on p95
Echo triggers barge-in Agent stops itself mid-sentence on speakerphone Echo cancellation; minimum speech duration before cancel
History not truncated on barge-in Agent assumes the user heard things they didn't Playback marks; store only the heard text
STT mishears names, numbers, codes Wrong booking, wrong account Keyword boosting, read-back confirmation, DTMF for PINs
Slow tool call Dead air for seconds Cached acknowledgement, pre-fetch, async with SMS follow-up
Duplicate side effects Two refunds after a retried turn Idempotency keys on every write tool
Vendor outage or latency spike Call fails or stalls Timeouts per stage, fallback STT/TTS vendor, transfer to human
Prompt injection by voice Caller talks the agent into a forbidden action Same tool permissions and policy checks as a text agent
Long-call cost creep (speech-to-speech) Cost per minute rises through the call Fixed prefix for caching, history caps, call time limit

How to evaluate a voice agent#

Test the components and the whole call. A low WER with bad turn-taking still produces a bad call.

What Metric How
STT WER overall and on entities (names, numbers, addresses) Recorded test calls with reference transcripts; jiwer computes WER. Use 8 kHz audio if you serve phone calls
Latency p50, p95, p99 voice-to-voice, plus per stage Timestamp end of user speech and first agent audio at the client; the median hides the turns people complain about
Turn-taking False cut-in rate, mean time to respond after end of turn Scripted callers who pause mid-sentence
Barge-in Time from interruption to agent silence; was history truncated? Scripted interruptions at random points in agent speech
Task success Share of calls that complete the goal correctly Simulated callers (an LLM playing a persona over TTS) against a checklist, plus human review of a sample of real calls
Safety Wrong side effects, policy violations Adversarial scripts; check tool calls, not only the words spoken
Cost $ per call and per successful call From billing logs, not estimates

Measure latency at the caller's side. Server logs show when you sent audio, not when the caller heard it, and on phone calls the gap can be large. Simulated calls catch most regressions when you change the prompt, the turn settings or a vendor; the harness pattern in evaluating an AI agent in Python carries over once you add a TTS voice for the simulated user and an STT pass on the agent's replies.

Decision table#

Situation Choice Reason
Regulated domain (banking, health, insurance) Cascade Transcript before speech, text guardrails, audit trail
Non-English or mixed-language callers, local names Cascade Pick the STT that wins on your audio; swap it alone
Lowest possible latency, open conversation, expressive voice Speech-to-speech One model, no text hop
High call volume, cost-sensitive Cascade with a small LLM Roughly $0.04 a minute in the example above
Long calls (10+ minutes) Cascade, or speech-to-speech with strict history caps Re-sent history makes long speech-to-speech calls expensive
Phone numbers, inbound support SIP/PSTN with a media gateway Reach, transfer to humans, DTMF
In-app assistant WebRTC Wideband audio, lower latency, browser echo cancellation
Many slow tool calls Cascade, with acknowledgements and async results Easier to control what is said while tools run

For a first build I'd start with a cascade on an open-source orchestration framework such as Pipecat or LiveKit Agents, measure p95 voice-to-voice latency and task success on 50 scripted calls, and only then decide whether the latency gap justifies a speech-to-speech model. If the agent also needs to remember callers between calls, the read and write paths in my agent memory design apply, with the read path pre-fetched at call start so it adds nothing to turn latency.

Sources#