Blog

We have argued in another blog post that transcription alone is not enough to analyse a conversation, and that conversational context drives performance across the stack. This post assumes you accept that and moves to the part that is harder to find written down: how to actually wire a speaker intelligence layer into a streaming pipeline, what it costs you in latency, how to reconcile two asynchronous event streams, and how to prove the result is better than what you had.
Where the layer sits
A speaker-aware real-time pipeline has five stages:
Audio capture, single or multi-channel, from telephony, WebRTC, or a meeting bridge.
Streaming speaker intelligence, producing a live speaker timeline.
Streaming ASR.
Speech understanding: topics, entities, sentiment, computed per speaker.
LLM layer for summaries, action items, compliance checks, coaching.

Stage two sits between capture and ASR, but the important detail is that it does not sit in front of ASR in a blocking sense. Diarization operates on the waveform, ASR operates on the waveform, and neither needs the other's output. Chain them, and you add the diarization latency to the transcription latency for no benefit. Fan the same audio out to both, and the speaker timeline is available independently, which is what lets you drive a talk-time meter or an interruption alert before a single word has been transcribed.
Live-1 takes 16 kHz mono audio in 100 ms chunks over a WebSocket and emits two event types: diarization_speaker_start, carrying a speaker label and a start timestamp, and diarization_speaker_end, carrying the end timestamp. That is the entire output contract. It is small on purpose, and it is what you build the rest of the pipeline against.
The alignment problem
You now have two asynchronous streams: speaker events keyed to audio time, and ASR hypotheses that arrive later, get revised, and carry their own word or segment timestamps. Joining them is the part most teams underestimate.
The naive approach waits for diarization_speaker_end, takes the closed interval, and assigns every ASR word whose timestamp falls inside it. This is correct, and it is also useless for real-time, because a speaker turn can run for twenty seconds and you cannot leave the agent UI blank until it closes.
The approach that works is to attribute before the turn finishes. Maintain a stack of currently active speakers, pushed on diarization_speaker_start and popped on diarization_speaker_end. Attribute the forming transcript to whoever most recently took the floor, and lock the attribution when that speaker's turn ends:
A stack rather than a single current speaker is what makes overlap and fast turn-taking behave. During overlap, two labels are on the stack at once, and the most recent one takes the forming text, which matches how interruptions actually resolve: the person who took the floor last is usually the one continuing.
Both streams must share a clock. Timestamp everything against the audio timeline, not against wall time or arrival order. If your ASR provider reports offsets from the start of its own session and your diarization session opened 40 ms earlier, that offset is a persistent misattribution near every turn boundary.
The latency budget
Budget end-to-end and in percentiles. A rough allocation for an agent assist surface:
Stage | Typical contribution |
|---|---|
Capture and jitter buffer | 20 to 80 ms depending on transport |
Chunking to the diarization stream | 100 ms chunk boundary |
Streaming diarization | 300 ms (Live-1 from pyannote) |
Streaming ASR | 200 to 400 ms, provider dependent |
Understanding and rules | 10 to 50 ms |
LLM stage | 300 ms and up, usually the dominant term |
Two consequences follow. First, anything you can compute from the speaker timeline alone lands roughly an order of magnitude sooner than anything that requires the LLM, so put talk-time, interruption, and silence detection there rather than in a prompt. Second, the LLM stage dominates the budget, which means shaving milliseconds off diarization is rarely where the win is. Getting the labels right is.
Provisional and confirmed labels
Incremental diarization revises itself. Early in a call, the model has heard a few seconds of each voice, and a segment attributed to speaker A at second three may be reassigned once more evidence arrives. Design for this explicitly rather than treating a revision as a bug.
Run a two-tier model. Provisional labels drive low-stakes surfaces: a talk-time meter, a live transcript pane, a speaking indicator. These can flicker without consequence. Confirmed labels, emitted once the turn has closed, drive everything with a side effect: writing to the CRM, firing a compliance alert, appending to the record that feeds the post-call summary.
Anything downstream of the confirmed stream should be idempotent, keyed on segment identity rather than arrival order, so a revision replaces rather than duplicates. In batch, turnLevelConfidence and confidence give you per-turn and per-sample scores to set that threshold against evidence rather than intuition.
What the timeline gives you before transcription
The speaker event stream alone supports a set of features that never touch the ASR output:
Talk-time share. Cumulative speaking duration per label over a rolling window.
Turn length distribution. Time between matched start and end events, per speaker.
Gap and latency between turns. Interval between one
speaker_endand the nextspeaker_start. Negative intervals are overlaps.Interruption count and direction. A
speaker_startwhile another label is still on the stack, attributed to whoever started.Silence. No labels active for longer than a threshold.
None of these require language, which also means none of them require a language-specific model. They work identically on a French call and a Japanese one.
Identity, and where it actually runs
Diarization gives you SPEAKER_00 and SPEAKER_01. Mapping those to people is a separate step with a different deployment profile.
Speaker identification runs on Precision-2 in batch, comparing segments against enrolled voiceprints and returning identity labels alongside the diarized timeline in a single request, with confidence scores, a configurable matching threshold, and exclusive matching so two speakers cannot resolve to the same voiceprint. Voiceprints are built from roughly 10 seconds of audio, with 10 to 30 seconds giving reliable named recognition.
In a live pipeline, the practical pattern is therefore hybrid: derive identity from the channel where you have it, since the agent leg is known, and apply voiceprint identification at turn close or call close for cross-session continuity and analytics. Voiceprints are biometric data. pyannoteAI does not store them, and deployments that use identification need consent from the people enrolled.
Blueprint: real-time agent assist
Telephony feeds capture. Capture fans out to Live-1 and to your streaming ASR. The speaker event stream drives three UI surfaces directly, with no dependency on the transcript:
A talk-time balance meter over a rolling 60-second window, computed from cumulative segment duration per label.
An interruption alert triggered when agent-initiated overlaps exceed a threshold within a window, using overlap direction rather than raw overlap count. Direction is what separates an agent talking over a customer from a customer talking over an agent, and only one of those is a coaching signal.
A silence nudge when no label has been active beyond a threshold.
The joined speaker-attributed transcript then feeds the understanding and LLM stages: per-speaker sentiment trajectories rather than a call-level average, intent attributed to the party who expressed it, and compliance checks that verify a disclosure was spoken by the agent rather than merely present somewhere in the transcript.
For post-call, the batch path with transcription: true and a transcriptionConfig model produces the speaker-attributed transcript in a single call, which removes the alignment work described above entirely. Use the streaming path for what has to be live and the batch path for what does not.
Proving it works
Measure the layer and the outcome separately.
For the layer: DER is decomposed into missed speech, false alarm, and speaker confusion, because the three have different causes and different fixes. Confusion points at the clustering and embedding stage; false alarms usually point to VAD behavior under your noise conditions. Add tcpWER rather than plain WER, since WER is blind to whether the right label landed on the right words. Add turn-boundary precision, which predicts downstream alignment quality better than aggregate DER does.
For the outcome: run the same call set through a speaker-agnostic and a speaker-aware pipeline and compare classifier outputs against human labels. Sentiment prediction lift with speaker features added is the cleanest single measurement, because the speaker-agnostic baseline is genuinely handicapped there and the effect size is large. Compliance pass rate and coaching adoption follow, on longer cycles.
Run it on your own audio. Public benchmark DER tells you how models rank against each other. It does not tell you how one behaves on your codecs, your noise floor, and your speaker population.
Where this goes
The same event stream that drives an agent assist UI can drive a voice agent's own turn-taking. Gap duration and overlap give an endpointing policy something better than a silence timer, and identity resolution lets an agent distinguish the enrolled user from a background voice before it responds to one. Speaker-conditioned behaviour, rather than speaker-labelled output, is where this layer is heading.
Build it
Open a streaming session, push 16 kHz mono in 100 ms chunks, and handle the two event types. Then measure the two numbers that decide whether this belongs in your pipeline: DER on your own recordings, and p95 end-to-end latency under your real concurrency. The tutorials repository has a working Live-1 implementation paired with a commercial STT, and the trial runs for 30 days without a credit card.
