Who's talking, and what do they remember?

Voice agent demos happen in quiet rooms with one person who has never spoken to the agent before. Real use is different. There's a partner in the kitchen, a television in the corner, and someone who talked to the agent yesterday and expects it to remember. Two capabilities handle that: speaker diarization, which works out who is speaking, and long-term memory, which remembers what they've said.

This is the final part of the voice agent series. Both features are optional in the project, and both follow the same rule as everything else: they must never make a reply slower than it needs to be.

Diarization, the practical way

Classic diarization takes a finished recording and clusters it into speakers after the fact. A live agent can't wait for the recording to finish. It needs to know who's talking on every turn, as it happens.

So the project uses only the embedding part of pyannote: the WeSpeaker ResNet34 model, which turns a stretch of speech into a 256-number voiceprint. Each turn's voiceprint is compared, by cosine similarity, against speaker profiles stored in SQLite. The closest match above a threshold wins. The model is small enough to run on the CPU, in a dedicated single thread, so it never competes with the GPU running the voice.

• New voices are enrolled automatically as provisional speakers ("Speaker 2"). If they stop talking for a couple of turns, the provisional profile expires, so a passing visitor doesn't clutter the list.
• Known voices keep learning. Each match blends the new voiceprint into the profile, weighted towards recent turns, so a profile adapts to a cold, a different microphone, or a quieter room.
• Names come from conversation. The agent has tools to save and update voice profiles, so "I'm Sam" or "that was my daughter" can become a named profile without opening a settings page.
• Splits within a turn. If two people speak in one turn, it can be attributed sentence by sentence.

The LLM sees each line labelled with a name, like [Sam] what's on tomorrow?, along with a roster of who's on the call. That small change makes group conversations work: the agent can answer the right person and keep track of who asked what.

Who before what

Early on, the speaker identity arrived after the transcript, and the code had to go back and relabel lines already added to the conversation. It was fragile. Making diarization part of the turn itself, so the speaker is known before the words are, removed a pile of synchronisation code and fixed a bug that had silently dropped every speaker label after the first turn in one of the client apps. The turn now waits for the voiceprint, which costs a little time, and that trade was worth it for correctness.

The television problem

Put a voice agent in a living room and the news will talk to it. The first fix was an "ignore unknown voices" setting: turns from voices that don't match a named profile are shown greyed out but never reach the LLM or memory. Strangers get no answer.

That wasn't enough, because interruptions happen before transcripts. Voice activity detection fires about a third of a second into any speech, so a newsreader could still cut the agent off mid-sentence; she just wouldn't reply to him. The second fix, only known voices interrupt, holds the interruption back while the agent is speaking. It takes 0.8 s of the new speech, makes a voiceprint, and checks it against the named profiles. A known person interrupts a beat later than before. The television gets talked over.

Both features fail open. If diarization is off, there are no profiles, or the embedding fails, the agent behaves normally. Nobody gets locked out because of a noisy microphone.

Long-term memory without the lag

Memory uses Mem0, an open-source memory layer that extracts facts from conversations and stores them for later retrieval. It can run against Mem0's cloud service or fully locally. The local setup uses FastEmbed with the small bge-small-en-v1.5 embedding model, an embedded Chroma vector store, and the same local LLM as the agent to extract facts. Nothing leaves the machine.

The naive approach is to search memory on every turn: embed the user's words, query the vector store, add the results to the prompt. That puts an embedding call and a database query between the user's last word and the agent's first. In voice, that's time you can hear.

Instead:

1. 01Fetch once per call. When a speaker is identified, all of their memories are loaded into a cache in the background.
2. 02Select cheaply on each turn. Relevant memories are picked from that cache by simple word overlap with what was just said. No embedding, no network, effectively no time.
3. 03Widen for broad questions. "What do you know about me?" or questions about family and work raise the number of memories included.
4. 04Write only when idle. New memories are extracted in batches every few turns, one batch at a time, and only while the LLM isn't generating a reply. Remembering never competes with talking.

Diarization and memory reinforce each other. Memories are stored per speaker and grouped by name in the prompt, so the agent doesn't tell Sam about her partner's dentist appointment. Turns from ignored voices are never stored at all.

Remembering when, not just what

People ask "what did we talk about on Tuesday?" A tool called recall_memories_on_date lets the agent look up memories by date. An optional end-of-call summary is stored too, so the agent can recall the gist of a past conversation, not just isolated facts.

Simpler won

I also built a hybrid memory that combined vector search with a knowledge graph of people and relationships, plus a dedicated memory screen. On paper it was the more capable design. I reverted it. The cached, per-speaker approach above is what runs today: simpler to reason about, and it keeps memory entirely off the critical path.

Small things that make it feel human

• Choosing a delivery. Every reply starts with a hidden style line, such as [style: gently curious | warm | soft | slow], chosen by the LLM from a fixed vocabulary. It steers the fine-tuned voice's delivery and is removed before you hear anything.
• Honest interruptions. Only the words actually spoken are kept in history when the agent is cut off, so it never assumes you heard the rest.
• Picking the thread back up. After 30 seconds of silence the agent gently re-engages, and again at two minutes, rather than waiting forever.
• Knowing where and when. The current date, time, and city-level location are in context, so "what's on this weekend?" makes sense.