Fine-tuning a voice: the dataset is the model

Zero-shot voice cloning is magic for about ten minutes. Give a modern TTS model 15 seconds of someone talking and it will imitate them convincingly. Then you ask it for a long, warm, slightly amused answer, and the voice slides towards someone else. A few seconds of reference audio can't carry a whole personality.

Fine-tuning fixes that. You train the model on 30 to 60 minutes of one speaker until the voice is part of the weights rather than a hint in the prompt. Maya's voice, the one you hear on this site, is a fine-tune of Qwen3-TTS, trained and served on Apple Silicon with MLX. It took five versions to get right. This is part three of the voice agent series, and it's mostly about those failures, because they're more useful than the success.

Consent comes first

A fine-tuned model reproduces a person's voice well enough to be mistaken for them, and the pipeline produces a speaker embedding, which many jurisdictions treat as biometric data. Only train on recordings you have the right to use, with the speaker's explicit permission for both the training and the intended use. Keep the source audio, the dataset, the embedding, and the weights out of any repository you publish.

The pipeline

The current pipeline has six stages, one command, and each stage skips itself if its output already exists, so a failed run resumes where it stopped:

1. 01Build. Transcribe the raw recordings with Whisper medium.en, then cut them into training clips.
2. 02Annotate. Label each clip with transcript, emotion, tone and speed, then re-label it relative to this speaker's own range.
3. 03Train. A full fine-tune, conditioning each clip on a natural-language instruction built from its labels.
4. 04Export. Fuse the weights, register the speaker, quantise to 8-bit, and install it where the voice service can load it.
5. 05Calibrate. Sweep sampling settings and recommend defaults for this voice (covered in the sampling article).
6. 06Validate. Render test sentences and score each one for word error rate, speaker similarity, and pause lengths.

Chunking is where quality is won

Long recordings have to be cut into short clips, and where you cut turns out to matter more than almost anything else. The rules the chunker follows now:

• Clips are 3 to 16 seconds long, aiming for about 10, resampled to 24 kHz.
• Cuts happen only at real sentence ends, at least 0.3 s into a silence.
• Two sentences more than 0.6 s apart are never joined into one clip.
• Each clip keeps 0.15 s before the first word and 0.25 s after the last, and no more.
• Every cut point is checked against the actual audio energy, because Whisper's word timestamps drift: starts miss plosives and ends miss breaths.
• Clips whose characters-per-second rate is an outlier are dropped, since they usually mean a bad transcript.

Each of those rules exists because a previous version broke without it.

Five versions, four failures

v1: the metric said yes, my ears said no

Qwen3-TTS predicts 16 codebooks of audio tokens per frame. The first codebook carries the broad content of speech; the other fifteen, produced by a smaller sub-model, carry much of the timbre: what makes a voice sound like a particular person. My first attempt used LoRA adapters and a loss on the first codebook only. The output was perfectly intelligible and sounded like someone else.

The speaker-similarity score was 0.93, which looks excellent. The clone was rejected the first time I played it. That's the most important lesson in this whole article: automatic speaker-similarity metrics are coarse. They're good at catching regressions between versions and bad at telling you whether a voice is right. Listening is the acceptance test.

v2 onwards is a full fine-tune using the loss Qwen recommends: the first-codebook loss plus 0.3 times the sub-model's loss. Learning rate 2e-6, effective batch size 32, up to 12 epochs with early stopping after three epochs without improvement. The rate is small on purpose: you want to move the voice, not unlearn speech.

v2: one bad file

The second version was better but still lost to zero-shot cloning. The cause was boring: one source recording had poor quality, and because it was long, it made up a large share of the clips. The model dutifully learned to sound a bit like that recording. Audit sources individually before you chunk them, because after chunking the damage is hard to see.

v3: the model learned to pause

For v3 I tried to make cuts "safe" by placing them at the quietest point of each pause. That baked between half a second and three seconds of silence into the edges of clips. The model learned exactly what it was shown: pause for ages at every comma and full stop.

Worse, trailing silence diluted the signal for when speech should end. Some renders kept generating near-silence until they hit the 4,096-token cap, which at 12.5 tokens a second is about five minutes. The model reproduces every artifact of the dataset, including the absence of sound. Audit the silence, not just the transcripts.

Teaching the voice to act

A fine-tuned voice that always sounds the same is a narrator, not a conversational partner. To give Maya range, every clip is annotated by Qwen3-Omni, a multimodal model that listens to audio, with an emotion, tone, and speed. A second pass then compares each clip with the speaker's own recordings, in thirds of her pitch, energy, speaking rate and brightness. "Fast" means fast for her, not fast compared with an average person.

Those labels become a plain-language instruction during training, something like "gently curious, warm, soft, slightly slow". At run time, the LLM chooses a style for each reply, and the same kind of instruction steers the fine-tuned voice. The result is a voice that can be reassuring, brisk, or playful without losing who she is.

0.6B or 1.7B?

The pipeline can fine-tune either size of Qwen3-TTS. The 0.6B model is the default: quick to train and quick enough to serve in real time with room to spare. The 1.7B model sounds richer, but its 8-bit export is about 2.8 times larger and it's slower to run. For a live agent where every millisecond is counted, the smaller model is often the right call. Maya currently runs on a 1.7B fine-tune, because her voice is the product.

A speech-to-speech detour

I also tested whether a single speech-to-speech model could replace the whole STT, LLM, and TTS chain. Qwen3-Omni can hear and speak, but locally it ran at about 1.4 times real time, too slow to hold a conversation. The cascaded pipeline wins for now, and it has the advantage that each piece can be swapped and tuned independently.