If you've shipped a product on a transcription API, you've probably seen it. The words are right. The speaker labels aren't. Speaker 1 becomes Speaker 3 halfway through the call. Two people merge into one. A customer's complaint gets attributed to the agent, and now your sentiment dashboard says your support team is furious.

Here's the short answer to why this keeps happening. Speaker identity lives in the audio signal rather than in the words, and diarization bundled into a transcription API is built as a feature of the transcript. Systems that label speakers after reducing audio to text are working from a signal that has already lost the information they need. This article walks through the four failure modes that follow from that design and covers the fix, including the version of the fix that keeps your existing transcription vendor.

The architecture problem underneath all of it

Transcription engines are optimized for one thing: getting the words right. Speaker labels are typically produced by segmenting or clustering around the word stream, then assigning each word to a speaker. That ordering has a consequence. STT timestamps are approximations, produced through buffering and forced alignment, and the speaker boundaries hang off those approximations. When the words are uncertain, the boundaries are uncertain. When two people talk at once, there's often only one word stream to assign. The diarization can only ever be as good as the transcript scaffolding it's built on.

A dedicated diarization layer inverts the order. It works on the raw audio first, finding speech, speaker turns, and overlap from voice characteristics, then hands that structure to transcription. The words attach to speakers instead of speakers attaching to words.

This is not a theoretical distinction, and pyannote's own documentation names the problem it solves. The Community-1 pipeline exposes a dedicated output, output.exclusive_speaker_diarization, described on the model card as simplifying "the reconciliation between fine-grained speaker diarization timestamps and (sometimes not so precise) transcription timestamps." That parenthetical is the whole article in seven words. Diarization timestamps are precise because they come from the audio. Transcription timestamps are not, because they come from decoding words. A pipeline that builds one on top of the other inherits the imprecision.

Failure mode 1: overlap breaks word-level assignment

People talk over each other constantly. Interruptions are how real conversations work: objections, clarifications, back-channels, two people finishing the same sentence. Even pyannote's public benchmark corpus, drawn from ten real-world domains, contains 9.3% overlapping speech across 259 recordings, and contested domains sit well above that average.

This is exactly where word-level speaker assignment falls apart, because when two voices produce one garbled word stream, the system has one sequence of words to hand out and two people who said things. Someone gets the other person's words, or the overlap region simply vanishes from the transcript.

The damage is worst precisely where the product value is highest. The interruption is the moment your call analytics should catch, the disputed statement is the moment your compliance check exists for, and these are the moments bundled diarization garbles. A dedicated layer detects overlap as overlap and attributes both speakers through the region, which is what the powerset segmentation in the current pyannote generation is built to do. Our deep dive on overlapping speech covers the mechanism.

Failure mode 2: speaker counting drifts on long audio

Bundled diarization tends to hold up on a five-minute clip with two clean voices, which is what most evaluations try. Then production sends it a 90-minute meeting. Long recordings are where speaker identity has to stay stable across breaks, room changes, and hours of acoustic drift, and it's where clustering around a word stream falls apart slowly rather than all at once. A speaker leaves and comes back as someone new. Two quiet participants merge. By the end of the recording, there are seven labels for five people, and every downstream per-speaker metric is quietly wrong.

This failure is sneaky because it doesn't look like an error in any single line of the transcript. Each utterance reads fine. The corruption only shows when you ask a question that spans the whole recording, like how long each person spoke, who committed to what, or whether the same customer called twice.

Counting is genuinely hard for every system, and being specific about that is more useful than claiming it away. Precision-2 predicts the exact speaker count on 70% of a 250-file internal set spanning two to ten speakers, up from roughly 50% for Precision-1. A dedicated layer also gives you two ways to sidestep the estimate entirely: passing a known or bounded speaker count when the roster is known, and voiceprints that hold the same person's label across breaks, sessions, and separate recordings. Our piece on speaker counting covers why the estimate is hard in the first place.

Failure mode 3: accents and language switching degrade labels

When diarization rides on top of a transcription engine, it inherits the engine's linguistic blind spots. Accented speech that raises word error rates also degrades the timestamps the speaker labels depend on. A speaker who switches languages mid-conversation can trip the engine badly enough that the label switches with the language, as though a new person entered the room.

Voice-based diarization does not have this failure, because voice characteristics do not change when the language does. pyannote's models operate on acoustic characteristics rather than linguistic content, which is why the same pipeline works across languages without a per-language model. For multilingual products, this single failure mode is frequently the reason teams go looking for a dedicated solution.

Failure mode 4: when it's wrong, there's nothing you can do

The final failure shows up after you find the error rather than in the output itself. Bundled diarization is a checkbox on someone else's product, so there's no speaker count to constrain, no sensitivity to adjust, no way to feed back a correction, and no roadmap you influence. The vendor's priority is the transcript, reasonably, because that's the product. Your ticket about speaker labels on nine-person calls competes with everything else in their queue.

Teams that build on a dedicated layer get the opposite:

• Speaker count parameters: numSpeakers for an exact count, minSpeakers and maxSpeakers for a range.

• Confidence scores on a 0 to 100 scale, per turn or per sample, so a reviewer sees the segments the model itself flags as uncertain.

• Voiceprint identification with a match threshold and exclusive matching, so two speakers cannot resolve to the same person.

• An exclusive diarization output for pipelines that need one speaker per moment for STT reconciliation.

• Deployment choices from cloud API to on-premise.

The fix, in two versions

Fixing speaker labels does not have to mean abandoning the transcript quality you chose your vendor for. The fix is separating the jobs: let a dedicated speaker layer work on the audio, and let a transcription engine do words. There are two ways to build that, and they differ in one respect worth being precise about.

Version one: keep your transcription vendor. Send the audio to pyannote's diarization API and to your existing STT in parallel, then reconcile the two outputs yourself. This is the version that literally keeps your vendor, and output.exclusive_speaker_diarization exists to make the reconciliation tractable, since it returns one speaker per moment and spares you from resolving overlap against imprecise word timestamps. More engineering on your side, and full control over which STT you use.

Version two: let pyannote orchestrate both. STT Orchestration runs Precision-2 diarization alongside a transcription model and applies its own reconciliation logic, returning word-level and turn-level speaker-attributed output in a single call. Worth knowing before you choose it: the transcription model is pyannote's, currently Nvidia's Parakeet-tdt-0.6b-v3 by default or OpenAI's whisper-large-v3-turbo via transcriptionConfig. Less engineering, and your current STT vendor comes out of the path.

Either way, the attribution comes from the audio. Overlap is handled as overlap. Speaker identity holds across long recordings. Language switches don't create phantom speakers.

On the public benchmark, Precision-2 posts the lowest diarization error rate in all ten domains tested, measured against nine other systems including the diarization built into AssemblyAI, Deepgram, Speechmatics, OpenAI's GPT-4o-transcribe-diarize, and AWS Transcribe. The methodology is published, so the comparison can be rerun on your own audio.

The pattern to remember is simple. Words are what was said. Speakers are who said it. Those are different problems in different signals, and production systems work better when each is solved by something built for it.

Frequently asked questions

Why are my transcript speaker labels wrong? Most likely because your transcription API assigns speakers from the word stream rather than the audio, so the speaker boundaries inherit the imprecision of word-level timestamps. Overlap, long recordings, accents, and language switches all break word-level assignment in predictable ways, covered above.

Is built-in diarization from transcription APIs accurate? On short, clean, two-speaker audio, often yes. On the public ten-domain benchmark covering hard real-world audio across 259 recordings, the STT providers' bundled diarization trails dedicated models in every domain. The gap grows with overlap, speaker count, and recording length.

How do I fix speaker confusion without switching transcription vendors? Run diarization separately from transcription and reconcile the outputs. Send audio to pyannote's diarization API alongside your existing STT, and use the exclusive diarization output to align speakers to words without fighting imprecise timestamps. If you would rather not build the reconciliation, STT Orchestration does it for you with pyannote's own transcription model.

Does dedicated diarization work across languages? Yes. Diarization from voice characteristics is language-agnostic, because the acoustic properties that distinguish one speaker from another do not change with the language being spoken. Diarization bundled into a transcription engine varies with how well that engine handles the language.

How do I know which speaker labels to trust? Turn on confidence scores, which are off by default and return a 0 to 100 value per diarization turn or per sample. Route anything below your threshold to review rather than treating every label as equally certain.

Ready to test it on your own audio? Start with the API quickstart.