Where the milliseconds go in a voice agent

In conversation, people answer each other in a few hundred milliseconds. Stretch that past about a second and the exchange starts to feel like a bad video call: both sides hesitate, then talk at once. A voice agent can have the best voice in the world and still feel broken if it's slow.

This is the second part of a series on building a human-sounding voice agent. It covers where the time goes in a real pipeline, what I measured, and which fixes paid off.

The budget

A voice turn has five stages, and each one spends time before you hear anything back:

1. 01Endpointing: deciding you've stopped talking (Silero VAD plus the Smart Turn model).
2. 02Speech-to-text: turning your audio into words.
3. 03The language model: time until the first token of the reply.
4. 04Text-to-speech: time until the first audio of the reply is ready.
5. 05Transport: getting audio back to the browser over WebRTC.

The rule that matters is that everything streams. The LLM streams tokens to a sentence aggregator, the aggregator hands complete sentences to TTS, and TTS streams audio to the browser while it's still generating the rest. You never wait for a whole reply. You only wait for the beginning of one.

The 931 ms bug

When I first instrumented the TTS stage, it took about 930 ms from receiving a sentence to sending audio. That seemed reasonable for a 1.7B-parameter model on a laptop until I looked closer. The model itself produced its first audio after about 55 ms, and it ran at a real-time factor of 0.47, roughly twice real time. The rest of the delay was my own code: a helper that batched audio into fixed 2-second blocks before sending anything, to avoid gaps in playback.

The fix was an adaptive ramp. The first block is short, 300 ms, so audio starts almost at once. Every block after that is sized from two measurements: how much audio is already buffered on the listener's side, and how fast the model is currently generating. Two rules keep it safe:

• A block may never take longer to generate than the buffered audio ahead of it, with a 0.7 safety factor for jitter.
• If the buffered lead falls to 120 ms, whatever has been generated is flushed immediately, even if the block isn't full.

Before the deadline rule existed, one test case ran a 66 ms deficit, which you hear as a stutter. With it, the same case kept an 88 ms cushion.

TTS time to first audio

931 → 166 ms

TTS time to first audio

first chunk, cold vs warmed-up model

2,479 → 72 ms

first chunk, cold vs warmed-up model

time for the model's own first audio

~55 ms

time for the model's own first audio

Then the clicks

Faster audio came with a new problem: faint clicks and gaps. WebRTC sends audio in 40 ms frames (960 samples at 24 kHz). When a block didn't divide evenly into frames, the leftover was padded with silence, and that silence landed in the middle of a word. Across four test phrases it added up to about 297 ms of dead air. Rounding every block except the last down to a whole number of frames fixed it.

Not every idea worked. Before the adaptive ramp, I tried capping every block at 100 ms. Latency improved, but the gaps between chunks were audible, and I reverted it two days later. Small blocks are only safe when you know the buffer ahead of them can cover the next one.

Warm-up matters more than you'd think

MLX, Apple's machine-learning framework, compiles a GPU pipeline the first time it sees a new tensor shape. That makes the first sentence of every call slow. I measured the first audio chunk at 2,479 ms on a cold model and 72 ms on a warm one.

The fix is simple: generate about four seconds of throwaway audio (50 codec tokens) when the app starts and again at the start of each session, before the greeting. If the user starts talking during warm-up, live speech wins and the warm-up is abandoned. Anyone running local models on GPUs, with any framework, should check for the same first-call penalty.

Faster and more accurate speech recognition

For speech-to-text I compared MLX Whisper small.en with NVIDIA's Parakeet-TDT 0.6B v2, also running on MLX, using the Pipecat STT benchmark set. Parakeet won on both counts:

I also tried Confucius4-R2T2, a true streaming STT model that emits words while you're still talking. It was even more accurate on the same benchmark, but it didn't make replies any faster, because the LLM still has to wait for the end of your turn. Live captions are nice; they aren't latency. I parked it on a branch.

Endpointing without false alarms

Responding fast is pointless if the agent jumps in while you're still thinking. Silero VAD settings are a trade-off between reacting quickly and reacting to noise. My first settings were too eager: a cough or a chair creak counted as an interruption. Raising the speech confidence threshold from 0.55 to 0.75, the minimum speech duration from 0.2 to 0.35 s, and the minimum volume from 0.01 to 0.04 removed most false barge-ins.

VAD alone can't tell a pause from a finished sentence, so Smart Turn v3 listens to the end of each utterance and predicts whether you're done. On top of that, a couple of watchdogs catch the cases where the pipeline would otherwise hang, such as a turn that ends with no transcript.

Cut at sentences, not full stops

To start speaking early, the agent sends the reply to TTS one sentence at a time. Splitting on every full stop sounds fine until the model writes "Dr. Smith" or "e.g." Each TTS call renders a slightly different take on the voice, so a cut in the wrong place produces an audible seam mid-sentence. The sentence aggregator ignores abbreviations, initials, ellipses, and list numbers, and when a boundary is ambiguous it waits for one more token from the LLM before deciding. That costs a few milliseconds and makes speech sound continuous.

Pipecat vs LiveKit, and the real bottleneck

I wanted to know whether the framework mattered, so I built the same agent on LiveKit Agents and benchmarked both on real speech. These are median (p50) figures from a snapshot partway through the project:

The totals were within 30 ms of each other. LiveKit stopped talking faster when interrupted, and Pipecat detected the end of a turn faster. Neither framework is the problem. The single largest share of the budget went to the language model, a 26B mixture-of-experts Gemma model running locally. I turn its reasoning mode off for voice, because a model that thinks before it speaks adds seconds you can't hide.

Keep everything else off the critical path

The last lesson is about discipline. Memory lookups, memory writes, logging, and recordings all add latency if they run inline. Memories are loaded once at the start of a call and new ones are written only while the LLM is idle (more on that in the memory article). The rule I use: if the user can't hear it, it doesn't get to run between their last word and the agent's first.