Open-source voices now sound human. Here's what it takes to ship one.

If you've pressed Talk to Maya on this site, you've heard the result of six months of tinkering. Maya is a voice agent I built from scratch. She listens, works out when you've finished talking, thinks, and answers in a voice fine-tuned on recordings of a single speaker. Every model in that loop is open, and the speech side runs on a single Apple Silicon machine, not a paid cloud API.

Two years ago that sentence would have been a stretch. The best synthetic voices were behind commercial APIs, and the open models that existed sounded like open models: flat prosody, odd breaths, a slight metallic edge. That gap has mostly closed. This article is about how far the technology has come, and about the less glamorous engineering that decides whether a voice agent feels like a conversation or a phone tree.

How the open models caught up

Modern TTS models mostly work like language models. A neural audio codec compresses speech into a stream of discrete tokens, and a transformer learns to predict those tokens the same way a chat model predicts words. Because the model is predicting sound rather than reading out phonemes, it learns the things that make speech sound human for free: the little rise before a question, the breath before a long sentence, the way people speed up through words that don't matter.

The project went through three engines before it settled. It started on ElevenLabs, which sounded great and cost money every time Maya spoke. I moved to Kokoro, a tiny open model that's fast and clean but can't take on a new voice. Then Qwen3-TTS arrived, and the commit message from that day just says "Voice cloning. oh yea!"

Qwen3-TTS uses a 12.5 Hz codec, so each token represents 80 ms of audio, and it outputs 24 kHz speech. It comes in 0.6B and 1.7B parameter sizes. I run it through mlx-audio on Apple's MLX framework, quantised to 8-bit, on the Mac's GPU. On that hardware the model generates speech about twice as fast as it plays back, a real-time factor of roughly 0.47, which is the headroom you need for streaming.

parameters in the Qwen3-TTS models I run

0.6B–1.7B

parameters in the Qwen3-TTS models I run

real-time factor on Apple Silicon (lower is faster)

~0.47

real-time factor on Apple Silicon (lower is faster)

time to first audio after the streaming work

166 ms

time to first audio after the streaming work

Three ways to get a voice

The project supports three ways of creating a voice, and they suit different jobs.

• Voice cloning. Record 10 to 15 seconds of someone speaking, transcribe it automatically with Whisper, and the base model imitates that voice zero-shot. It's remarkably good for a quick start, but the clone drifts on long or emotional passages because the model only has a few seconds to go on.
• Voice design. Describe the voice in text ("a warm, mid-thirties New Zealand woman, unhurried, slightly amused") and the VoiceDesign model invents one. With a fixed random seed, the same description gives you bit-identical audio, so a designed voice can be kept as a brand asset.
• Fine-tuning. Train the model on 30 to 60 minutes of clean recordings of one speaker. This is how Maya's voice was made, and it's the only option that holds a consistent character across hours of conversation. It's also where most of the hard lessons were. I wrote those up in the fine-tuning article.

A note on ethics, because it matters. A cloned voice is a convincing voice. The project's fine-tuning documentation has a consent section for a reason: only train on voices you have explicit permission to use, and be clear with listeners that they're talking to an AI. On this site, the call panel introduces Maya as my AI guide before you hear a word.

The voice is the easy part now

Once the voice sounds right, you find out what people actually notice in a conversation. It isn't timbre. It's timing, interruptions, whether you remember them, and whether you talk over the dog barking in the background. These are the pieces that decide whether a voice agent works.

Maya runs on Pipecat, an open-source Python framework for real-time voice pipelines, behind a FastAPI server. Audio travels over WebRTC using Pipecat's SmallWebRTC transport, so the browser's own echo cancellation and noise suppression come for free. The pipeline looks like this:

mic → Silero VAD + Smart Turn v3 → Parakeet STT → LLM (Gemma 4) → sentence aggregator → Qwen3-TTS (MLX) → WebRTC → speaker

1. Latency, above everything

People start to feel a pause at around half a second. Every stage of the pipeline spends some of that budget, and they add up. The biggest single win in the project was in the TTS stage: the model produced its first audio in about 55 ms, but my streaming code held it back to fill a two-second buffer. Replacing that with an adaptive first block took time to first audio from 931 ms to 166 ms. Another fix warms the model up at the start of each session, because MLX compiles GPU kernels the first time it sees a new tensor shape. That took the first chunk from 2.5 seconds cold to 72 ms warm.

Speech recognition got the same treatment. I switched from Whisper to NVIDIA's Parakeet-TDT 0.6B, running under MLX. On the Pipecat STT benchmark set it was both faster (about 100 ms median versus 530 ms) and more accurate (about 3% word error rate versus 4%). The full story, including a benchmark against LiveKit, is in Where the milliseconds go.

2. Turn-taking and interruptions

Knowing when someone has finished is harder than detecting silence. People pause mid-thought. Silero VAD detects speech, and Smart Turn v3, a small model trained on how sentences end, decides whether a pause is the end of a turn. When you interrupt Maya, she stops, and only the words she actually said are kept in the conversation history. If she was cut off early she gets a short note saying so, which stops her from carrying on as if you'd heard the whole answer.

3. Knowing who's speaking

Put a voice agent in a kitchen and it will hear more than one person, plus the television. Speaker diarization compares a voice embedding of each turn against known speaker profiles, so Maya knows whether she's talking to Sam or Sam's partner. It also enables a setting I'm fond of: only known voices can interrupt. A newsreader on the TV no longer cuts her off mid-sentence.

4. Remembering people

A companion that forgets you every call is a novelty. Long-term memory uses Mem0 with a local vector store, scoped to each recognised speaker. The trick is keeping memory off the critical path. Memories are fetched once at the start of a call, and new ones are written only while the language model is idle, so remembering never makes a reply slower. That design is covered in Who's talking, and what do they remember?.

5. Writing for the ear

Language models write for screens. An output filter turns digits into words, strips markdown, emoji, and HTML, and replaces dashes that TTS would read awkwardly. Text is only sent to the voice at the end of a real sentence, not at every full stop, because "Dr." or "e.g." mid-sentence would create an audible seam. Maya also chooses her delivery: each reply starts with a hidden style line (emotion, tone, pace, energy) that's turned into an instruction for the fine-tuned voice and removed before you hear anything.

Tuning by ear

A TTS model has the same sampling settings as a chat model: temperature, top-k, top-p, and repetition penalty. They behave quite differently on audio tokens. Too low and the voice goes flat and robotic; too high and it slurs, mumbles, or drifts into a different person. I ended up writing a calibration sweep that scores dozens of combinations on speaker similarity and intelligibility. Top-k turned out to matter far more than top-p. The sampling article has the details.

Where this is useful

The same core now runs in four places: the website guide you can talk to here, a personal companion in a web app, an Android app, and an ESP32-S3 microcontroller board with a small AMOLED screen that acts as a dedicated speaker. That spread is the point. Once the speech stack is local and open, the voice becomes a component you can put anywhere.

• Front-of-house guides on websites and kiosks, which answer questions, ask good discovery questions, and hand off to a person with context.
• Intake and triage for service businesses, collecting structured details before a human picks up.
• Companions and accessibility tools where remembering the person matters more than speed of task completion.
• Private, on-premises voice for organisations that can't send audio to a third-party cloud.

Telephony was on the list too. I built and then removed a phone gateway, because the latency and audio quality of phone lines undid much of the work above. It's solvable, but it's a different project.

What I'd tell someone starting now

1. 01Pick a framework (Pipecat or LiveKit) and measure end-to-end latency from day one. You can't feel your way to 200 ms.
2. 02Start with voice cloning to validate the experience, then fine-tune once you know which voice you want.
3. 03Treat the language model as the slowest stage. In my benchmarks it was the bottleneck by a wide margin.
4. 04Design for interruptions, background noise, and more than one speaker before you design anything else.
5. 05Keep anything that isn't speech, such as memory, logging, and analytics, off the critical path.

Open-source speech has got good enough that the voice is rarely the weak link. What's left is mostly systems engineering, and that's the work I enjoy. If you're thinking about putting a voice in front of your customers, talk to Maya first, then book a call and tell me what you noticed.