Blog
Machine translation and synthetic voice have both improved faster than anyone in localization expected. A model can translate a line well, and a voice model can speak it in a convincing performance. Neither of those capabilities, on its own, produces a dubbed episode.
The reason is that a dubbing pipeline does not operate on a script. It operates on a finished mix: dialogue, music, effects, laughter, crowd noise, all arriving as one audio track with no metadata attached. Before anything can be translated or re-voiced, somebody has to work out who spoke, when they spoke, and which of them is the same person as the character in the previous episode. That work used to be done by hand, and in a lot of facilities it still is.
This is the layer we think the industry systematically underestimates. Localization teams budget for translation quality and voice quality, then ship a title where a line lands in the wrong character's voice, or a recurring guest changes voice between episodes, or a subtitle gets credited to the wrong person. Those are not translation defects. They are metadata defects, and they come from the speaker layer.
Localization breaks at the speaker layer. We built our models to fix it there.
What a dubbing pipeline actually needs from the audio
Localization runs as a chain. Source audio produces a transcript, the transcript gets translated, translated lines get cast to voices, synthesis or a voice actor produces the new audio, and the result gets mixed back against the original timing. Every stage after the transcript needs an answer to the same question: who said this line.
That answer has to carry three properties:
Timing precise enough to re-time speech. Dubbed lines have to land inside the original speech intervals.
Consistency across a whole title. The same on-screen person needs the same label in episode 1 and episode 12.
Independence from the source language. A catalog rarely arrives in one language.
How diarization supplies it
Speaker diarization answers who spoke when, working from the mixed audio alone. It runs in four stages. Voice activity detection separates speech from silence, music, and effects. Segmentation finds the boundaries where one speaker stops and another starts, including where they overlap. Speaker embeddings convert each stretch of speech into a vector describing the voice itself: pitch patterns, formant frequencies, speaking style. Clustering then groups those vectors, so every utterance from the same voice ends up under the same label.
The output is a timeline: start time, end time, speaker label, for every stretch of speech in the file. That maps directly onto the three properties above.
Timing comes from the segment boundaries, which are derived from the acoustics rather than from the word stream, so they stay accurate through music beds, laughter, and passages the transcription model cannot resolve. Consistency inside a title comes from clustering, which assigns one label per voice across the whole file, and consistency across a catalog comes from voiceprints, covered further down. Language independence comes from the embeddings, which encode the physical characteristics of a voice rather than the words it is producing. A vocal tract does not change when its owner switches language, which is why the same model handles a Korean drama and a French documentary with no configuration between them.
Worth separating three terms that get used interchangeably in localization briefs. Diarization groups a recording into speakers without knowing who they are. Identification matches those groups to known people using enrolled voiceprints. Transcription turns speech into words. A dubbing pipeline needs all three, and they fail in different ways.
The deliverable is the speaker metadata a localization stack needs, so that the right voice ends up on the right line, in any language, at broadcast scale, with frame-accurate timestamps across long-form content as the diarization property doing the work.
The output format that makes alignment tractable
A diarization job returns segments carrying start, end, and speaker. For dubbing, the useful option is exclusive mode.
Setting exclusive to true adds a diarization track with overlapping speech removed, so each segment contains exactly one speaker. That matters because ASR output does not handle overlap well, and aligning a transcript against a clean one-speaker-at-a-time track is far more reliable. The exclusive result arrives in an exclusiveDiarization field alongside the standard output, so a pipeline keeps both: the overlap-aware version for mixing decisions, the clean version for line assignment. The merge tutorial walks through the alignment itself.
Running transcription in the same call removes the reconciliation step entirely. The problem it replaces is familiar to anyone who has stitched two vendors together: diarization gives an accurate speech turn boundary at time t while the STT engine returns words at t plus some drift, and teams end up hand-writing overlap heuristics and fallback rules that break on the next title. We solved that once, inside the API.
With transcription set to true, STT Orchestration returns two arrays, wordLevelTranscription and turnLevelTranscription, each element carrying start, end, text, and speaker. turnLevelTranscription is the dubbing script. wordLevelTranscription is the timing budget, which is what a re-timing or lip-sync stage needs to know how much room a translated line has.
Speaker continuity across a catalog
Diarization labels speakers within one file. Dubbing needs the same person recognized across many files, which is what voiceprints do.
A voiceprint is a compact digital representation of a person's voice characteristics, closer to a fingerprint than to a recording. The constraints are worth knowing before building a catalog workflow:
Audio samples must be at most 30 seconds long
The recording must contain only the target speaker's voice, with no overlapping speakers
Create only one voiceprint per person
Job outputs, voiceprints included, are deleted 24 hours after completion, so they belong in your own store
An identification job accepts up to 50 voiceprints
Our models are language agnostic, so voiceprints can be created in any spoken language. A voiceprint built from an English interview identifies the same person in a Japanese one. For a media library, that means one voiceprint registry per person, rather than one per person per language.
The /identify endpoint takes those voiceprints with labels attached and returns segment-level matches with confidence scores. Two settings live inside its matching object. matching.threshold, a float from 0 to 100 that defaults to 0, sets the minimum confidence required for a match, and we recommend 50 to 70 for stricter matching. matching.exclusive, true by default, prevents multiple speakers from being matched to the same voiceprint, which is the failure mode you want blocked when two cast members have similar voices. Note that this is a different parameter from the top-level exclusive used for exclusive diarization.
The payoff is persistent identity across multiple sources, and a searchable voice index across an archive.
The source language never needs declaring
Localization catalogs arrive in whatever language they were shot in. Our diarization models are language-agnostic, and no retraining is needed per language.
There is no language parameter on the diarization endpoint. Language coverage enters only through the transcription model chosen inside transcriptionConfig. The default, parakeet-tdt-0.6b-v3, covers 25 languages, all of them European. faster-whisper-large-v3-turbo covers 100, including Japanese, Hindi, Cantonese, Korean, Thai, and Tagalog, which makes it the setting to change first when a title arrives from outside Europe.
Real production audio, not clean studio audio
Archive and field material is the hard case, and our public benchmark is built for it: 10 distinct domains, 259 recordings, roughly 67 hours of audio, and 9.3% overlapping speech. The domains are Broadcast Interview, Clinical, Courtroom, Conversational telephone speech, Map task, Meeting, Restaurant, Sociolinguistic (field), Sociolinguistic (lab), and Web video.
Overlap is the condition that breaks dubbing pipelines most often, because talk shows, panel formats, and live events are built out of interruption, simultaneous speech, and audience reaction. Precision-3 improves exactly there: across 15 benchmark datasets, missed detection on crosstalk drops 19.3% and speaker confusion on crosstalk drops 20.7% against Precision-2, with overall diarization error rate falling from 16.02 to 14.35.
Quality control: where human review should go
Human review is the expensive part of localization. Scores decide where it goes.
Precision-3 reports uncertainty at two useful levels. turnLevelConfidence returns a per-segment score from 0 to 100 describing how certain the model is that a turn belongs to the speaker it was assigned. Frame-level probabilities go finer, at a resolution of 20 ms: speechProbability for whether anyone is speaking, crosstalkProbability for whether more than one person is speaking at once, and speakerProbability for whether a particular speaker is active. Identification adds a third level, scoring how well a voiceprint matches a speaker rather than how certain the diarization was.
The workflow that follows is simple. Flag segments below your threshold for manual review instead of reviewing an entire transcript. For a dubbing house running QC on hundreds of episodes, that converts a full-pass review into a triage queue.
A request that covers the whole job
A single call to POST /v1/diarize produces the diarization, the clean alignment track, the speaker-attributed transcript, and the QC signal:
json
One constraint shapes how this fits a catalog workflow: transcription cannot be combined with speaker identification jobs, so identification runs as a separate pass against your stored voiceprint registry.
A timing note for anyone integrating now. Precision-3 becomes the default model on October 3, and Precision-2 is deprecated on October 17. Integrations that set model: "precision-2" or read the confidence output need updating before then, since Precision-3 replaces confidence with the frame-level probabilities above. turnLevelConfidence is unchanged.
For subtitle deliverables, our walkthrough on speaker-labeled subtitles covers converting speaker-attributed words into SRT or VTT. Its example code caps a cue at 84 characters, roughly two 42-character lines, and 6 seconds, and opens a new cue whenever the speaker changes or a gap exceeds 0.7 seconds.
Frequently asked questions
Does AI dubbing need speaker diarization? Yes. Translation and voice synthesis both operate per speaker. Without speaker labels and timestamps, a translated script cannot be cast to voices or re-timed against the original.
Can diarization keep the same dubbed voice for a character across episodes? Diarization labels speakers within one file. Voiceprints carry identity across files, so a character maps to the same voice throughout a series.
Does the source language affect diarization accuracy? Our diarization models are language-agnostic and need no retraining per language. Voiceprints carry the same property and can be created in any spoken language.
How do I align diarization with a transcript I already have? Align on timestamps and assign each transcript segment to the speaker with the greatest overlap. Enabling exclusive removes overlapping speech first, which makes that alignment far more reliable.
What about crowd noise, interruptions, and audience reactions? Overlap detection handles simultaneous speech, and our public benchmark includes restaurant, broadcast interview, and web video domains at 9.3% overlapping speech overall.
The takeaway
The defects audiences actually notice in a localized title tend to come from speaker attribution rather than from word choice: a line delivered in the wrong voice, a character who changes voice between episodes, a subtitle credited to the wrong person.
Getting the speaker layer right early means the translation and synthesis stages inherit clean, timestamped, identity-stable input. Start with the quickstart, or push a real episode through the API playground and look at what comes back before writing any alignment code.
