Why did we start over?

When we set out to build a streaming diarization model that could survive production, the first decision was the most uncomfortable one: not to extend what we already had. DIART, the framework our CTO, Juan, published in 2020 and one of the first rigorous attempts at low-latency online speaker diarization, was a real piece of work. It had real users. But we knew from running it ourselves, and from watching how customers tried to deploy it at scale, that the architecture would not get us where we needed to go. The honest answer was a rebuild.

This post is about how we built the Streaming Diarization and how it happened: what we kept from DIART, what we threw away, the constraints that drove the design, the engineering problems we did not expect, and the team structure that made it possible to ship in a reasonable timeframe.

The constraints came first

We did not start from a model and look for a use case. We started from the deployment constraints and worked backwards. Four prerequisites were on the wall before any architecture work began:

• Scalability across many parallel streams, not single-stream inference

• Low total latency, with an internal target of 300 ms all-inclusive

• Good quality in conversations with more than four speakers and on noisy, real-world audio

• Simple integration over WebSocket, with no heavy client-side machinery

Each of these excluded options. The latency budget alone ruled out anything that needed to look seconds into the past to commit to a decision. The multi-stream requirement ruled out one-model-per-stream layouts. The 4+ speaker requirement ruled out architectures tuned implicitly around dialogue. The integration constraint ruled out anything that required a non-trivial SDK on the client side. By the time the prerequisites were settled, the option of patching DIART was already gone.

Why DIART could not be the foundation

DIART is best understood through its runtime pipeline:

Audio passes through segmentation, embedding, and online clustering to assign or update speaker labels. The fundamental design choice is that this pipeline is a streaming adaptation of pyannote batch models. DIART processes windows comparable in size to batch inference, then keeps only the slice of each result relevant to the present. That dependency on past context forces the system to look far back in time, which limits achievable latency, and most of the per-window computation is redundant from one step to the next.

There is also a structural constraint. The implementation handles a single stream at a time, with models dedicated to each stream. There is no native multi-stream handling. We sometimes call this “fake” streaming internally: batch logic re-run quickly, with strong computational inefficiency. Fine for a researcher. Wrong shape for an enterprise running thousands of concurrent conversations.

What we kept, what we rebuilt

Conceptually, some things survived the rewrite. The objective itself, real-time diarization, obviously did. So did some of the larger reasoning blocks: the system still recognizably involves segmentation, embedding, and a tracking stage. Beyond that, almost everything was rethought. The problem was taken from scratch for streaming. The models were retrained specifically for this use case rather than adapted. The orchestration logic was redesigned natively for production rather than wrapped around batch primitives.

The one architectural change we can describe openly is chunk size. DIART analyzed windows several seconds long. The new system processes audio in chunks on the order of 100 ms. That single decision does most of the latency work: it removes the redundant computation that dominated the batch-adapted approach and shortens the dependency on past context. Other details, including the specifics of the inference strategy, remain our secret recipe.

The problems we did not expect to spend so much time on

The model architecture was the visible part of the work. What surprised us was how much engineering time went into problems that have nothing to do with diarization quality in the academic sense.

Distributed state consistency turned out to be the largest of these. A speaker who has been identified must remain the same speaker for the rest of the conversation, and that guarantee has to hold not only inside the model but across the infrastructure running it. When inference can move between instances and chunks arrive at 100 ms intervals, keeping that identity stable is not a model property. It is a system property.

A second category was orchestration: where and under what conditions GPU inference actually runs, how the building blocks of the system communicate with each other, and how to keep the latency-versus-quality tradeoff bounded under load. For the beta, there is not much exposed to the user here. The tradeoff is managed internally through architecture choices: chunk size, inference strategy, system layout, and network handling. Configurability is on the table for later, but the priority for now was getting the defaults right.

How did the team actually work?

A model like this does not come from research, handing a finished artifact to engineering, and engineering hardening it. We tried that on earlier projects, and it slows everything down. This one ran as a mixed squad: the technical side and the research side. Research did not deliver a completed block before engineering started; both advanced at the same time.

At the start, each side laid down its fundamentals in parallel: research on model architecture, engineering on system and infrastructure. Once those bases existed, the separation between the two largely dissolved. It was not two adjacent projects but one project, with continuous exchange. Research influenced what the system could ask of the models. Engineering influenced how the models were designed so they would be deployable.

That is what allowed the project to move faster than a sequential hand-off would have allowed, and it kept an entire category of scaling problems from surfacing late. Multi-stream inference, state consistency across instances, and chunk-level orchestration were addressed inside the model design rather than retrofitted after the fact, when fixing them would have meant rewriting the model.

What does the interface look like?

The intent throughout was to keep the integration path plain. A client mints a session URL, opens a WebSocket, streams raw PCM in fixed chunks, and reads back diarization events:

import json, os, requests, websocket

API_KEY = os.environ["PYANNOTE_API_KEY"]

# 1. Mint a single-use session URL
resp = requests.post(
    "<https://api.pyannote.ai/v1/live>",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={},
)
resp.raise_for_status()
ws = websocket.create_connection(resp.json()["url"])

# 2. Stream audio: 16 kHz mono float32 LE, 100 ms chunks = 6400 bytes
with open("audio.pcm", "rb") as f:
    while chunk := f.read(6400):
        ws.send(chunk, opcode=websocket.ABNF.OPCODE_BINARY)

# 3. Signal end of stream and drain remaining events
ws.send(json.dumps({"type": "end_of_stream"}))
while True:
    try:
        msg = ws.recv()
    except websocket.WebSocketConnectionClosedException:
        break
    if msg:
        print(json.loads(msg))

In production, the same microphone audio is typically fanned out to two consumers in parallel: pyannote for who is speaking and a streaming STT provider for what is said. Words are attributed by timestamp to whichever speaker turn covers their start, which produces a live speaker-labeled transcript without a separate reconciliation step.

Learning to measure a streaming model

One thing we did not anticipate at the start was how much of the work would be in defining good metrics. Batch diarization has settled evaluation conventions. Streaming does not, and the gap matters. Latency becomes a first-class metric. So does the difference between perceived quality in the moment and final quality once more context arrives. A prediction can be revised as the system sees more of the conversation, which makes any single-number DER an incomplete picture. We ended up building tooling around partial-result evaluation and stability over time, on top of the headline numbers we report. The model is currently the highest-accuracy, lowest-latency streaming diarization system we are aware of when run side by side against the other streaming offerings on the market, but the more interesting story is that the definition of "accuracy" had to be partly rebuilt to even claim that with confidence.

Closing thought

The hard part of this project was never the diarization theory. We have been conducting diarization research for over twelve years and working on streaming pipeline development for six. The hard part was making that theory survive contact with a distributed, real-time system: tracking identity across instances, fitting under a 300 ms total latency budget, holding up on multi-speaker noisy audio, and keeping the integration surface small enough that customers can drop the model into an existing Voice AI stack without rebuilding around it. The model is the visible output. Most of the work is underneath.