GyaanStack. Blog

Articles / agentsvoicepython

Build a voice AI agent in Python: STT, LLM, TTS, turn-taking and barge-in

A local voice agent in Python with faster-whisper, Ollama and macOS say. Covers VAD end-of-turn, tool calls, barge-in and real per-stage latency.

By Raghunath · · 19 min read

A voice agent is three models in a row (speech-to-text, an LLM, text-to-speech) plus the code that decides when the user has stopped talking and what to do when they talk over the agent. Most tutorials wire the three models together, record five seconds from the microphone, and stop. The result works in a demo and feels broken in a conversation, because the fixed recording window cuts people off or leaves them waiting, and the agent cannot be interrupted.

This post builds the whole loop on a Mac with no API keys: Silero VAD for turn detection, faster-whisper for speech-to-text, Qwen3 4B on Ollama for reasoning and one tool, and the built-in macOS say command for speech. Input comes from WAV files generated with say, fed in 32 ms frames as if they were arriving from a microphone, so every run is repeatable. Every stage is timed, and the numbers below come from real runs.

The architecture, the 800 ms latency budget, telephony and cost per minute are covered in the voice AI agent system design post. This post is the hands-on half: the code, and what the stopwatch says when you run it.

What we are building#

 WAV file ──► 32 ms frames ──► Silero VAD ──► endpointer
                                                 │ end of turn
                                                 ▼
                                          faster-whisper (base.en)
                                                 │ text
                                                 ▼
                                    Qwen3 4B via Ollama (streaming)
                                       │ tool call?  ──► calculate()
                                       │ tokens
                                       ▼
                               sentence chunker ──► say -o (one WAV per sentence)
                                                         │
                                                         ▼
                                     playback clock ◄── VAD on user audio (barge-in)

Two choices in this diagram matter for latency. The LLM streams tokens, and a chunker cuts the stream into sentences so TTS can start on sentence one while the model is still writing sentence two. And a second VAD keeps listening while the agent speaks, so the user can interrupt.

Setup#

Python 3.12 in a virtual environment. faster-whisper 1.2.1 pulls in CTranslate2 and onnxruntime, and it ships the Silero VAD model as an ONNX file, so you do not need PyTorch.

uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python faster-whisper ollama numpy
ollama pull qwen3:4b-instruct

Versions used: faster-whisper 1.2.1, ctranslate2 4.9.1, onnxruntime 1.32.0, ollama-python 0.6.3, on an Apple M5 with 24 GB of RAM.

Test audio comes from say. The second clip has a 700 ms silence in the middle of the sentence ([[slnc 700]] is a say embedded command), which is the case that breaks naive turn detection:

mkdir -p audio
say -v Samantha -o audio/q1.aiff "What is eighteen percent of two thousand four hundred and fifty rupees?"
say -v Samantha -o audio/q2.aiff "Book a table for [[slnc 700]] four people tomorrow at eight."
say -v Samantha -o audio/interrupt.aiff "Stop. Just give me the total."
for f in q1 q2 interrupt; do
  ffmpeg -loglevel error -y -i audio/$f.aiff -ar 16000 -ac 1 -c:a pcm_s16le audio/$f.wav
done

Whisper and Silero both expect 16 kHz mono, so convert once here rather than resampling inside the loop.

Step 1: turn-taking with VAD and a silence timer#

Voice activity detection gives a speech probability for each short frame of audio. Silero VAD works on 512-sample windows at 16 kHz, which is 32 ms. VAD only says "speech or not". Deciding that the user's turn is over is a separate job, done here by an endpointer: once at least 250 ms of real speech has been heard, it waits for a run of silent frames of a set length and then declares the turn finished.

SR = 16_000
FRAME = 512                      # Silero VAD window: 512 samples = 32 ms at 16 kHz
FRAME_MS = FRAME * 1000 / SR


class Endpointer:
    """Declares end of turn after `silence_ms` of non-speech following real speech."""

    def __init__(self, silence_ms=600, min_speech_ms=250, threshold=0.5):
        self.silence_ms, self.min_speech_ms, self.threshold = silence_ms, min_speech_ms, threshold
        self.speech_ms = self.quiet_ms = 0.0
        self.started = False

    def feed(self, prob):
        """Feed one frame's speech probability. Returns 'start', 'end' or None."""
        if prob >= self.threshold:
            self.speech_ms += FRAME_MS
            self.quiet_ms = 0.0
            if not self.started and self.speech_ms >= self.min_speech_ms:
                self.started = True
                return "start"
        elif self.started:
            self.quiet_ms += FRAME_MS
            if self.quiet_ms >= self.silence_ms:
                return "end"
        return None


vad = get_vad_model()            # Silero VAD as ONNX, ships inside faster-whisper


def frame_probs(audio):
    # Scored in one batch for speed; we then walk the frames in order like a live stream.
    return vad(audio).reshape(-1)


def listen(audio, endpointer):
    """Walk frames in order. Returns (cut sample index, speech-end ms, endpoint ms)."""
    last_voiced = None
    for i, p in enumerate(frame_probs(audio)):
        if p >= endpointer.threshold:
            last_voiced = i
        if endpointer.feed(p) == "end":
            return (i + 1) * FRAME, (last_voiced + 1) * FRAME_MS, (i + 1) * FRAME_MS
    return len(audio), None, None

min_speech_ms stops a click or a cough from opening a turn. silence_ms is the knob that trades responsiveness against cutting people off, and step 6 shows what each setting does on the clip with the pause.

VAD costs almost nothing. Scoring the 4-second question took 20 ms for 125 frames, 0.16 ms per 32 ms frame. Running it on every frame of a live stream is fine.

Step 2: speech-to-text with faster-whisper#

stt = WhisperModel("base.en", device="cpu", compute_type="int8")


def transcribe(audio):
    segments, _ = stt.transcribe(audio, language="en", beam_size=1, vad_filter=False)
    return " ".join(s.text.strip() for s in segments)

The English-only .en checkpoints suit an English agent. int8 quantization keeps the model small on CPU, and beam_size=1 (greedy decoding) is faster than the default beam of 5 for short commands. vad_filter=False because the endpointer has already trimmed the audio. transcribe returns a lazy generator, so nothing runs until you iterate over the segments; time the join, not the call.

Median of five runs on the 4-second question, CPU, int8:

Model Download Median STT time Transcript
tiny.en 75.5 MB 606 ms What is 18% of 2,450 rupees?
base.en 145.2 MB 1,071 ms What is 18% of 2,450 rupees?

Both got this clean synthetic clip right, so tiny.en saves about 450 ms here for free. Real microphone audio with accents and background noise is where base and small earn their cost, so test on your own recordings before choosing. Whisper processes audio in 30-second windows, which is why a 4-second clip still costs most of a second: you pay for the window, not the words. If you need it faster on Apple silicon, whisper.cpp runs on Metal, and a streaming STT service (see the hosted section) returns partial transcripts while the user is still talking.

Step 3: the LLM, behind a small interface, with one tool#

The agent talks to the model through one method, stream(messages, cancel), which yields text pieces. Swap LocalLLM for a class that calls a hosted API and nothing else changes. Ollama's Python client accepts a plain Python function as a tool and builds the JSON schema from its signature and docstring (tool calling docs), and tool calls arrive inside the stream (streaming docs).

OPS = {ast.Add: operator.add, ast.Sub: operator.sub, ast.Mult: operator.mul,
       ast.Div: operator.truediv, ast.USub: operator.neg}


def calculate(expression: str) -> str:
    """Evaluate a basic arithmetic expression such as '0.18 * 2450'."""
    def ev(n):
        if isinstance(n, ast.Constant) and isinstance(n.value, (int, float)):
            return n.value
        if isinstance(n, ast.BinOp) and type(n.op) in OPS:
            return OPS[type(n.op)](ev(n.left), ev(n.right))
        if isinstance(n, ast.UnaryOp) and type(n.op) in OPS:
            return OPS[type(n.op)](ev(n.operand))
        raise ValueError("unsupported expression")
    try:
        return f"{ev(ast.parse(expression, mode='eval').body):g}"
    except Exception as exc:
        return f"error: {exc}"


TOOLS = {"calculate": calculate}
SYSTEM = ("You are a voice assistant. Replies are spoken aloud: two or three short sentences, "
          "no lists, no markdown. Use the calculate tool for any arithmetic.")


class LocalLLM:
    """Swap this class for a hosted model. stream() must yield text pieces."""

    def __init__(self, model="qwen3:4b-instruct"):
        self.model = model

    def stream(self, messages, cancel):
        while True:
            calls = []
            for chunk in ollama.chat(model=self.model, messages=messages, stream=True,
                                     tools=[calculate], options={"temperature": 0}):
                if cancel.is_set():
                    return                      # barge-in: stop generating
                if chunk.message.tool_calls:
                    calls += chunk.message.tool_calls
                if chunk.message.content:
                    yield chunk.message.content
            if not calls:
                return
            messages.append({"role": "assistant", "content": "", "tool_calls": calls})
            for c in calls:
                result = TOOLS[c.function.name](**c.function.arguments)
                yield {"tool": c.function.name, "args": c.function.arguments, "result": result}
                messages.append({"role": "tool", "content": result, "tool_name": c.function.name})

Do not use Python's eval for a calculator tool, even in a demo. The model writes the argument, and anything that reaches the model (including a transcript of someone speaking) can steer what it writes. The ast walker above allows numbers and four operators and rejects the rest.

The system prompt matters more in voice than in chat. Markdown, bullet lists and long paragraphs all get read aloud by the TTS engine, so tell the model the reply will be spoken and keep it to a few short sentences. The cancel check inside the token loop is what lets barge-in stop generation mid-stream. For the request and response shapes of tool calls across providers, see function calling, end to end.

Step 4: speak sentence by sentence#

say cannot stream, but it can write a WAV file. The chunker groups streamed tokens into sentences and hands each finished sentence to TTS, so the first sentence is synthesized while the LLM is still generating the rest.

def tts(text):
    """macOS `say` to a 16 kHz WAV. Returns (path, duration_s)."""
    path = tempfile.mktemp(suffix=".wav")
    subprocess.run(["say", "-v", "Samantha", "-o", path, "--file-format=WAVE",
                    "--data-format=LEI16@16000", text], check=True)
    with wave.open(path) as w:
        return path, w.getnframes() / w.getframerate()


SENTENCE_END = re.compile(r"(.+?[.!?])(\s+|$)", re.S)


def sentences(pieces, on_tool):
    """Group streamed tokens into sentences so TTS can start before the LLM finishes."""
    buf = ""
    for p in pieces:
        if isinstance(p, dict):
            on_tool(p)
            continue
        buf += p
        while (m := SENTENCE_END.match(buf)) and m.end(1) < len(buf):
            yield m.group(1).strip()
            buf = buf[m.end():]
    if buf.strip():
        yield buf.strip()

The m.end(1) < len(buf) check waits for one more character after the full stop before cutting. Without it, "2." in "2.5 lakh" would be sent to TTS as a sentence. The rule still misfires on "Dr. Rao" and similar abbreviations; production chunkers keep a short abbreviation list or wait for a minimum length.

Step 5: one timed turn, with barge-in#

run_turn ties the stages together. A producer thread runs the LLM, the chunker and TTS. The main thread plays the sentences back on a clock. To simulate an interruption, it runs VAD on the interrupt clip as if the user started talking barge_at_s seconds into the agent's reply. When the barge-in is confirmed it sets cancel, which stops the LLM loop and the TTS producer, and it saves only the words the user heard to the history.

def run_turn(llm, history, wav_path, silence_ms=600, interrupt_wav=None, barge_at_s=None,
             play=False):
    audio = load_wav(wav_path)
    cut, speech_end_ms, endpoint_ms = listen(audio, Endpointer(silence_ms))
    t = {"endpoint_wait": (endpoint_ms - speech_end_ms) / 1000}

    t0 = time.perf_counter()
    text = transcribe(audio[:cut])
    t["stt"] = time.perf_counter() - t0
    print(f"  user  : {text}")
    history.append({"role": "user", "content": text})

    cancel = threading.Event()
    spoken = queue.Queue()                    # (sentence, wav, duration, ready_at)
    t_llm = time.perf_counter()
    first_token, stage = [], {}

    def pieces():
        for p in llm.stream(list(history), cancel):
            if not first_token and isinstance(p, str):
                first_token.append(time.perf_counter() - t_llm)
            yield p

    def producer():
        for s in sentences(pieces(), lambda c: print(f"  tool  : {c['tool']}({c['args']}) -> {c['result']}")):
            if cancel.is_set():
                break
            if not stage:
                stage["first_sentence"] = time.perf_counter() - t_llm
            t1 = time.perf_counter()
            path, dur = tts(s)
            stage.setdefault("tts_first_sentence", time.perf_counter() - t1)
            spoken.put((s, path, dur, time.perf_counter() - t_llm))
        spoken.put(None)

    th = threading.Thread(target=producer)
    th.start()

    # Playback timeline. With --play we really play; otherwise we advance a clock.
    heard, clock, first_audio, barge_detected = [], None, None, None
    if interrupt_wav:
        ia = load_wav(interrupt_wav, lead_s=0, tail_s=0)
        ep, n = Endpointer(min_speech_ms=250), 0
        for n, p in enumerate(frame_probs(ia)):
            if ep.feed(p) == "start":
                break
        barge_detected = barge_at_s + (n + 1) * FRAME_MS / 1000   # VAD needs 250 ms of speech

    while (item := spoken.get()) is not None:
        s, path, dur, ready = item
        if first_audio is None:
            first_audio, clock = ready, 0.0
        if play:
            subprocess.run(["afplay", path])
        if barge_detected is not None and clock + dur > barge_detected:
            frac = max(0.0, (barge_detected - clock) / dur)
            words = s.split()
            heard.append(" ".join(words[: int(len(words) * frac)]) + " ...")
            cancel.set()                      # stop LLM and TTS
            print(f"  BARGE : user spoke at {barge_at_s:.2f}s, detected {barge_detected:.2f}s, "
                  f"cut mid-sentence")
            break
        heard.append(s)
        clock += dur
    th.join()

    t["llm_first_token"] = first_token[0] if first_token else float("nan")
    t.update(stage)
    t["first_audio_ready"] = first_audio
    t["voice_to_voice"] = t["endpoint_wait"] + t["stt"] + first_audio
    history.append({"role": "assistant", "content": " ".join(heard)})
    print(f"  agent : {' '.join(heard)}")
    print("  timing: " + "  ".join(f"{k}={v*1000:.0f}ms" for k, v in t.items()))
    return t

load_wav (in the full script) reads the 16-bit WAV, adds 0.4 s of silence before and 1.5 s after, and pads to a whole number of frames. The truncated history uses a word-count estimate of how far playback got. A real client gets this from playback position events; telephony providers send a marker when each audio chunk finishes playing.

The main block warms the model up with a one-token request, then runs four turns: the arithmetic question, the paused sentence with a 500 ms silence timer, the same sentence with 900 ms, and the same again with a barge-in 1.5 s into the reply.

Step 6: what the run printed#

warm-up (model load): 0.12s

[turn 1] arithmetic question, tool call
  user  : What is 18% of 2,450 Rupees?
  tool  : calculate({'expression': '0.18 * 2450'}) -> 441
  agent : 18% of 2,450 Rupees is 441 Rupees.
  timing: endpoint_wait=608ms  stt=1121ms  llm_first_token=2491ms  first_sentence=3756ms  tts_first_sentence=2798ms  first_audio_ready=6554ms  voice_to_voice=8283ms

[turn 2] mid-sentence pause of 700 ms, silence timer 500 ms
  user  : Book a table for.
  agent : I can't book a table for you. Please contact the restaurant directly for reservations.
  timing: endpoint_wait=512ms  stt=946ms  llm_first_token=202ms  first_sentence=883ms  tts_first_sentence=2546ms  first_audio_ready=3429ms  voice_to_voice=4887ms

[turn 3] same audio, silence timer 900 ms
  user  : Book a table for. Four people tomorrow at 8.
  agent : I can help you book a table for four people tomorrow at 8. Let me know if you'd like to proceed with that.
  timing: endpoint_wait=928ms  stt=1354ms  llm_first_token=187ms  first_sentence=1244ms  tts_first_sentence=2904ms  first_audio_ready=4147ms  voice_to_voice=6429ms

[turn 4] barge-in 1.5 s into the agent's reply
  user  : Book a table for. Four people tomorrow at 8.
  BARGE : user spoke at 1.50s, detected 1.95s, cut mid-sentence
  agent : I can help you book a table ...
  timing: endpoint_wait=928ms  stt=1230ms  llm_first_token=85ms  first_sentence=1095ms  tts_first_sentence=2644ms  first_audio_ready=3739ms  voice_to_voice=5898ms
  history now ends with: 'I can help you book a table ...'

The warm-up line is low because the model was already in memory; the first load from disk took 1.8 s. The machine was also running other jobs during these runs (load average around 11), so treat the absolute numbers as an upper bound. The proportions held across three runs.

Turn 3, the clean case, broken down:

Stage Time Share of 6.4 s
Endpoint wait (silence timer) 928 ms 14%
STT, base.en on CPU 1,354 ms 21%
LLM to first full sentence 1,244 ms 19%
TTS of first sentence (say) 2,904 ms 45%
Voice-to-voice total 6,429 ms

The run shows four things.

The silence timer decides correctness as well as speed. With 500 ms, the 700 ms pause ended the turn after "Book a table for", and the agent answered half a request with a refusal. With 900 ms the full sentence got through, at a cost of 416 ms on every turn. The system design post covers semantic turn detection, which is the real fix; the silence timer is the baseline it beats.

say is the bottleneck. At 2.5 to 2.9 s for one sentence it takes almost half the total. Timed on its own, the same sentence took 2.6 to 3.7 s with the Samantha voice and 1.6 s with the default voice. say renders the whole sentence before it returns, so the first audio is ready only when the last is. A neural TTS that streams audio (Piper locally, or a hosted service) is the first thing to replace.

The tool call costs a full extra model round trip. In turn 1 the first spoken token came 2.5 s after the LLM started, against about 200 ms without a tool: the model first generates the tool call, the script runs calculate, and only then does a second request produce speech. Voice agents usually hide this with a short filler line ("Let me check that") spoken while the tool runs.

Barge-in detection lags the user by 450 ms. The user started talking at 1.50 s and the agent stopped at 1.95 s. That gap is the 250 ms minimum speech rule plus the short silence at the start of the clip. Shorter would cut on coughs; longer and the agent keeps talking over the user. In the history, the assistant turn is saved as "I can help you book a table ...", so on the next turn the model knows the user never heard the rest.

The LLM is the fast stage here. Qwen3 4B produced its first token in 85 to 200 ms once warm. Small local models are good enough for the reasoning step of a narrow voice agent.

Testing before you change anything#

Per-stage timings show regressions, but you also want checks that fail loudly. These four checks cover the endpointer, the tool, the chunker and STT:

from agent import Endpointer, calculate, sentences, transcribe, load_wav, listen

# 1. Endpointer: 0.5 s speech, 0.4 s pause, 0.5 s speech, then silence (32 ms frames)
probs = [0.9] * 16 + [0.1] * 13 + [0.9] * 16 + [0.1] * 40
def end_frame(silence_ms):
    ep = Endpointer(silence_ms=silence_ms)
    return next(i for i, p in enumerate(probs) if ep.feed(p) == "end")
assert end_frame(300) < 29, "300 ms timer should cut inside the pause"
assert end_frame(600) > 45, "600 ms timer should wait for the second phrase"

# 2. Tool is safe: no eval of arbitrary code
assert calculate("0.18 * 2450") == "441"
assert calculate("__import__('os').system('ls')").startswith("error")

# 3. Sentence chunker yields the first sentence before the stream ends
chunks = ["The total ", "is 441 rupees.", " Anything ", "else?"]
assert list(sentences(chunks, print)) == ["The total is 441 rupees.", "Anything else?"]

# 4. STT on a known clip: every expected keyword present
audio = load_wav("audio/q1.wav")
cut, *_ = listen(audio, Endpointer())
text = transcribe(audio[:cut]).lower()
for word in ["18", "2,450", "rupees"]:
    assert word in text, (word, text)
print("all checks passed:", text)
all checks passed: what is 18% of 2,450 rupees?

Grow the STT check into a small set of real recordings with reference transcripts and track word error rate. For end-to-end checks of the agent's answers and tool use, the approach in evaluating an AI agent in Python carries over: run scripted turns, assert on tool calls and final text, and keep the latency numbers per stage so a slow change shows up next to a wrong one. The LLM cost and latency tracker shows how to log those numbers over time.

Failure modes this build shows, and the fixes#

  • Cut-off turns. A fixed silence timer ends turns on mid-sentence pauses. Raise it per conversation state (longer when collecting numbers or addresses), or add a semantic end-of-turn model.
  • Whisper hallucinations on silence. Whisper can return short phantom phrases for near-silent or noisy audio. Gate STT behind VAD (done here) and drop transcripts shorter than a word or two.
  • Echo. On a laptop speaker the microphone hears the agent, and VAD treats it as a barge-in. Use headphones in testing and echo cancellation in production; this build avoids the problem by using files.
  • Read-aloud markdown. A reply with asterisks or a numbered list sounds wrong. Say so in the system prompt and strip markdown before TTS as a backstop.
  • Stale history after interruption. If you save the full reply when the user only heard half, the model assumes facts the user never got. Save what was played.
  • Cold starts. Loading the LLM took 1.8 s on first use, and a contended Ollama server took 14 s before it was isolated. Warm models at startup and keep them resident.

Moving to hosted components#

The LocalLLM class, transcribe and tts are the three seams where hosted services plug in. Names below are from each vendor's docs as of this writing; check their pricing pages yourself, since prices change often.

  • Speech-to-text with turn detection: Deepgram's Flux (flux-general-en) is a streaming STT model with end-of-turn detection built in, which replaces both the endpointer and transcribe.
  • Speech-to-speech: OpenAI's Realtime API (gpt-realtime-2.1 on its guide page) takes audio in and returns audio out, so it replaces all three stages.
  • Text-to-speech that streams: ElevenLabs Flash v2.5 and Cartesia Sonic 3 both stream audio back as it is generated. Locally, Piper is a fast neural TTS that runs on CPU.
  • Orchestration: Pipecat (open source, BSD-2) and LiveKit Agents handle streaming transport, VAD, interruption and the pipeline wiring that this post does by hand.

Building the loop yourself once makes those frameworks easier to configure, because their settings (VAD thresholds, minimum interruption duration, endpointing delay) are the same knobs as Endpointer above.

Cleanup#

The two Whisper checkpoints downloaded to ~/.cache/huggingface/hub: tiny.en is 75.5 MB and base.en is 145.2 MB. Delete them by removing the models--Systran--faster-whisper-* folders and the blobs their model.bin links point to. The Qwen3 4B model takes 2.5 GB in Ollama; ollama rm qwen3:4b-instruct removes it if you pulled it only for this.

Sources#