Developers choosing an open diarization stack in 2026 usually end up comparing two names: pyannote, the most widely deployed open-source diarization framework, and NVIDIA NeMo, the research toolkit whose Sortformer models have posted strong results on academic benchmarks. Both are serious. They are also built for different jobs, and most of the practical differences follow from that.

The short answer, for readers who want it up front: NeMo's Sortformer is a capable end-to-end model inside a specific envelope, up to four speakers, primarily English, in a research-oriented toolkit. pyannote handles unconstrained speaker counts, works language-agnostically, ships as a production pipeline with a supported path from open source to hosted API, and posts the lowest error rates on the broadest published comparison currently available. The rest of this article is the evidence, with sources, so you can check every claim.

Two different architectures, two different philosophies

NeMo offers two diarization systems. The first is a cascaded pipeline: MarbleNet for voice activity detection, TitaNet for speaker embeddings, and the Multi-Scale Diarization Decoder (MSDD) on top of clustering. The second and newer is Sortformer, an end-to-end Transformer that outputs speaker labels directly from audio, resolving the permutation problem by sorting speakers in arrival-time order. It ships in an offline version and, since 2025, a streaming version with an Arrival-Order Speaker Cache for long-form audio. All of this is documented in NVIDIA's NeMo framework guide and the model cards on Hugging Face.

pyannote's open-source framework (pyannote.audio, currently the 4.0 generation with the Community-1 pipeline) composes segmentation, overlap-aware processing, speaker embeddings, and clustering into pretrained pipelines that run as a unit. The design difference matters downstream: Sortformer's end-to-end architecture fixes its maximum speaker count at training time, while pyannote's clustering-based approach estimates the number of speakers from the audio, with no architectural ceiling.

Accuracy: what the published numbers say

Start with the comparison pyannoteAI publishes and maintains. The public benchmark evaluates diarization error rate across ten DIHARD domains (broadcast, clinical, courtroom, telephone speech, meetings, restaurants, web video, and more) over 259 recordings and roughly 67 hours of audio, against commercial systems and against NVIDIA's open-source NeMo streaming Sortformer. On that benchmark, Precision-2 posts the lowest DER in all ten domains, and the open-source Community-1 outperforms most of the commercial systems tested. The benchmark page also states that no test data leaked into pyannoteAI training sets, a guarantee it notes the other compared providers do not publish.

Independent evaluation points the same direction. A third-party academic study published in late 2025 ("Benchmarking Diarization Models") evaluated five state-of-the-art systems across 196.6 hours of multilingual audio spanning English, Mandarin, German, Japanese, and Spanish, and found pyannoteAI achieved the best overall performance at 11.2% DER, with Sortformer variants among the compared systems. The same study identified missed speech and speaker confusion in high-speaker-count settings as the dominant error sources, which is exactly where architectural speaker limits bite.

Honesty requires the other side of the ledger, and it exists: NVIDIA's published results show Sortformer performing strongly within its envelope. On telephonic two-to-four-speaker English benchmarks, NVIDIA reports Sortformer matching or beating earlier end-to-end models, and the streaming version holds competitive accuracy at low chunk latencies. A team whose audio is reliably English with at most four speakers will find Sortformer a genuine contender on accuracy, and this article should not pretend otherwise. The comparison changes the moment either constraint breaks. For a like-for-like look at the streaming case specifically, our streaming diarization benchmark covers how real-time systems are measured and what the tradeoffs cost.

The four-speaker ceiling

The most consequential line in NVIDIA's own model card is the constraint: Sortformer models detect a maximum of four speakers, and performance degrades on recordings with five or more. This is architectural rather than a tuning default. The model's output layer is built for four speakers, and NVIDIA's fine-tuning documentation confirms the count is fixed.

Real-world audio breaks the assumption constantly: meetings run five or more participants, panel recordings and focus groups more, call center audio adds supervisors and transfers, and media content is unbounded. An independent 2026 experiment on a fourteen-speaker recording found the tested frameworks estimating between four and eight speakers, and noted that Sortformer's four-speaker limit made it impractical for the material despite its overlap-aware design.

Speaker counting is hard for every system, pyannote included, and our speaker-counting article treats those limits candidly. A hard architectural cap and a soft estimation error are different categories of problem. pyannote's clustering approach has no cap, and the speaker configuration controls let developers constrain the estimate when the true count is known: numSpeakers for an exact count, which the docs note "typically results in better overall diarization performance since the model can optimize for a specific speaker count," or minSpeakers and maxSpeakers where attendance varies.

Setup: manifest files and Hydra configs vs. a pipeline call

NeMo is a large multi-domain framework, and its diarization workflow reflects that heritage. Per NVIDIA's documentation, running Sortformer inference involves installing the NeMo toolkit and its dependencies (PyTorch, Cython, libsndfile, ffmpeg), preparing a JSON manifest describing the audio, invoking an example script from the repository with a model path and manifest path, and optionally applying per-dataset post-processing YAML configs to reproduce optimized results. Configuration runs through Hydra-style YAML files whose naming conventions encode loss type, speaker limit, and version. It is a workflow built by and for researchers who want every knob exposed, and it delivers that.

pyannote's open-source workflow is a pipeline call: install the package, load the pretrained Community-1 pipeline, pass an audio file, receive speaker-attributed segments.

python

from pyannote.audio import Pipeline

pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-community-1",
    token="HUGGINGFACE_ACCESS_TOKEN")

output = pipeline("audio.wav")

for turn, speaker in output.speaker_diarization:
    print(f"start={turn.start:.1f}s stop={turn.end:.1f}s speaker_{speaker}")

Add pipeline.to(torch.device("cuda")) to run on GPU. FFmpeg and a Hugging Face access token are the only prerequisites. The hosted API is a further step down in setup cost: a single POST to https://api.pyannote.ai/v1 with a bearer token, no local model, no GPU, and the same models behind it.

Neither workflow is wrong. They encode different assumptions about the user: NeMo assumes a researcher configuring experiments, pyannote assumes a developer shipping a feature.

Language coverage and long-form audio

NVIDIA's model cards state Sortformer was trained primarily on English public datasets, with the streaming model card noting strong performance on Mandarin as well. pyannote's models are language-agnostic by design, operating on acoustic voice characteristics rather than linguistic content, and the independent multilingual study above evaluated across five languages. Teams processing multilingual or code-switched audio should weigh this line heavily.

On duration: the original Sortformer paper notes computational limitations for long-form audio due to quadratic self-attention memory scaling, which the streaming version's speaker cache was specifically designed to mitigate, and NVIDIA's card for the streaming model still notes possible degradation on very long recordings. pyannote's pipeline processes long-form audio through windowed segmentation as standard operation, and the hosted API handles it as an asynchronous job so file length is a scheduling question rather than a memory one.

Production readiness: maintenance, support, and the path off your GPU

Production readiness is where the toolkit-versus-framework distinction stops being philosophical. Running NeMo diarization in production means owning the full stack: GPU provisioning, dependency and version management across a large framework, model updates on NVIDIA's research cadence, and community support through GitHub issues. NVIDIA's commercial support path runs through its enterprise platforms, which suits organizations already committed to that ecosystem.

pyannote's open source carries the largest deployment base in the field and an active community, and it is backed by a company whose business is diarization specifically: pyannoteAI maintains the models, publishes reproducible benchmarks, and offers commercial support and enterprise deployment.

The path off your own infrastructure is the final differentiator. Community-1 is available through the hosted API, which means the same model a team evaluates locally can go to production with zero GPU management, and Precision-2 sits above it for teams that want the accuracy ceiling: per the published benchmark, the lowest DER in every tested domain. Pricing is public. For teams that need the models to stay inside their own environment, the same models deploy on-premise without a change of stack. There is no equivalent zero-infrastructure path for a NeMo diarization deployment.

The honest verdict

Choose NeMo Sortformer when your audio is reliably four speakers or fewer, primarily English, your team is comfortable operating a research framework, and you are building inside the NVIDIA ecosystem already, particularly where its enterprise tooling is part of the stack. Within that envelope it is an accurate, well-documented model from a serious research team.

Choose pyannote when speaker counts are unconstrained, language is unpredictable, setup time matters, or the deployment is heading to production: which is to say, for most real-world applications. The open-source Community-1 outperforms most commercial systems on the published benchmark, installs as a pipeline rather than a framework, and has a supported path from laptop experiment to hosted API to on-prem deployment without changing models. Teams that want the accuracy ceiling on the same integration move to Precision-2.

Run the comparison on your own audio. Both stacks are free to evaluate, and the benchmark methodology documents how the published numbers were produced so the test can be reproduced rather than trusted.