Blog

What speech-to-speech models cannot learn from audio alone

What speech-to-speech models cannot learn from audio alone

Diarization is the foundation of speech-to-speech training data

A voice model is only as good as the conversational data it learns from. This is true for speech-to-text and text-to-speech, and it is even more true for speech-to-speech (S2S) models, which are trained to take part in the conversation rather than transcribe or read it.

A full-duplex S2S model listens and speaks at the same time. To behave correctly, it has to learn when the other person has finished, when an "mm-hm" means "keep going", and when an interruption means it should stop talking. None of this information is labeled in the audio itself. A recording of two people contains their voices, the room, and the timing between them, but nothing in the file says who is speaking, where a turn ends, or whether an overlapping segment is a backchannel or a real barge-in.

The model can only learn this structure reliably if the training data carries it. Speaker diarization is the step that adds it: it answers "who spoke when?" and turns raw conversational audio into a timeline of speaker turns, boundaries, speaker changes, and overlap regions.

In this article, we explain why that timeline matters more for S2S models than for any other type of voice model, where diarization sits in an S2S training-data pipeline, what the model learns when diarization is missing or inaccurate, and how to check it on your own data.

Real conversational audio was never recorded as training data

Most of the audio that labs use to train voice models was recorded for another purpose. It comes from a mix of vendors, custom collections, and public datasets: podcasts, interviews, meetings, and call recordings. Files often last an hour or more, and the number of speakers is usually unknown in advance.

This audio reflects how people actually talk. Speakers interrupt each other, say "yeah" or "right" while the other person is still speaking, pause in the middle of a sentence, and resume after a long silence. The microphone also records the room: background noise, reverberation, music, far-field speech, and codec artifacts.

The volumes are large. Smallest AI, a voice AI lab that trains its own STT, TTS, and S2S models, sources close to one million hours of raw audio and processes over 100,000 hours per month through pyannoteAI. At that scale, any structure the data needs has to be produced automatically.

Why a cascade can ignore the interaction, and an S2S model cannot

A cascade pipeline (audio input, speech-to-text, LLM, text-to-speech, audio output) treats speech as a sequence of utterances. Each component is trained on segmented clips, and turn-taking is handled outside the models by voice activity detection and endpointing rules. The STT model only needs to know what was said in a clip, and the TTS model only needs one clean voice.

An S2S model removes these boundaries. Turn-taking, interruption handling, and backchanneling move inside the model, so they have to be learned from data. The training data therefore has to describe the interaction itself, not only the words and the voices.

Diarization, identification, separation and conversation intelligence

These four terms are often used interchangeably, but they answer different questions and produce different outputs. Speaker intelligence is the combination of the four.

Capability

Question it answers

Output used for training data

Speaker diarization

Who spoke when?

Speaker-labeled time segments, turn boundaries, speaker changes and overlap regions within one recording

Speaker identification

Which known person is this?

Anonymous labels (SPEAKER_00, SPEAKER_01) linked to enrolled identities across files or sessions

Speaker separation

Can each voice be isolated?

One audio stream per speaker, extracted from overlapping or mixed audio

Conversation intelligence

What is happening in the conversation?

Event labels derived from the timeline: turn-taking, interruptions, backchannels, pauses, floor transfers

Diarization comes first because the other three depend on it. Identification needs speaker segments to compare against enrolled voiceprints. Separation is applied to the regions that diarization marks as overlapping. Conversation events are computed from the timing of turns and overlaps that diarization produces.

What each type of voice model needs from the data

Every voice model needs speaker-aware data, but not in the same way.

Model type

What the training data must provide

STT / ASR

Clean speaker turns and accurate timestamps, so transcripts are correctly attributed and label noise stays low

TTS

Single-speaker clips with complete words, so the model preserves voice consistency, prosody and speaker identity

Voice cloning and personalization

Correct identity linkage across files, so one speaker's voice never leaks into another's profile

S2S and full-duplex

Complete sessions with turns, boundaries, overlaps and backchannels attributed to the right speaker

Overlap is contamination for some models and signal for others

The clearest difference between these model types is how they treat overlapping speech.

For STT, TTS, and voice cloning, overlap is mostly a problem to remove. A single-speaker clip that contains a second voice for as little as 100 ms can pollute a fine-tuning set, and the error carries through to the model's output.

For S2S, removing every overlap would remove exactly the behavior the model needs to learn. Real conversations contain interruptions, simultaneous starts, backchannels, and moments where one speaker successfully takes the floor from another. A full-duplex model needs many examples of each, and these examples are only useful when they are correctly attributed. The goal is not to delete overlap, but to detect it, label it, and represent it accurately.

This also decides which diarization output to keep. Standard diarization returns overlapping segments, so two speakers can be active at the same time. Exclusive diarization returns a single active speaker at any moment, which is convenient for aligning transcripts but removes the overlap information. For STT clips, exclusive output is often enough; for S2S training data, the overlapping output is the one to keep.

Where diarization sits in an S2S training-data pipeline

Diarization is the first structural layer of the pipeline: every later stage reads the speaker-time structure it produces.

  1. Voice activity detection removes silence, music, and non-speech regions, and gives the first estimate of where speech starts and stops.

  2. Speaker diarization labels each speech region with a speaker and produces turn boundaries, speaker changes and overlap regions. This is the timeline every later stage builds on.

  3. Overlaps and turns are turned into conversation events: interruptions, backchannels, pauses between turns, and floor transfers.

  4. Speaker separation isolates the voices inside overlap regions, so each speaker's contribution can be represented as its own stream.

  5. Speaker identification links anonymous labels to known identities with voiceprints, so the same speaker keeps the same identity across files and sessions. This also helps deduplicate speakers and balance the corpus.

  6. Routing uses confidence and quality checks to send each segment or session to the stage it can support: fine-tuning, mid-training, or pre-training. Audio that cannot support any stage is dropped.

The result is speaker-aware conversational training data: time-aligned sessions where each turn, overlap, and backchannel is attributed to the right speaker.

Routing matters because training tiers have different tolerances. Pre-training can absorb some noise; fine-tuning cannot. Without a routing step, data that misses the fine-tuning bar is either discarded or sent to a manual correction queue. With per-turn confidence scores from diarization, it can be reused at an earlier training stage instead.

What poor diarization teaches a model

Diarization errors do not stay in the data. A model trained on a wrong timeline learns the wrong conversational behavior, and the error shows up in production, often far from its cause.

Failure mode

What it looks like in the data

What the model can learn from it

Mid-word cuts

Segments that start or end inside a word, trimming the first or last syllable

Truncated speech in TTS output; turn ends placed too early, so an S2S model responds before the user has finished

Missed overlap

A second voice left inside a segment labeled as one speaker, or an interruption that is not marked

Voice bleed in TTS and cloning data; an S2S model that treats a barge-in as part of its own turn and keeps talking

Backchannel taken for a turn

An "mm-hm" labeled as a full speaker turn

An S2S model that stops or yields every time the user acknowledges what it is saying

Pause taken for an end of turn

A mid-sentence silence closed as a turn boundary

An S2S model that responds during a hesitation

Unstable speaker labels

The same person split into two labels, or two people merged into one, within a session

Incorrect turn statistics: who interrupts whom, how long turns last, how speakers alternate

Cross-file identity errors

One speaker given different identities across files, or two speakers given the same one

Cross-speaker contamination in personalization data; unbalanced or duplicated speakers in the corpus

The errors are cheap to introduce and expensive to find. When the segmentation step produces them, teams usually catch them with manual review, and that QA does not scale: the curation process stays slow and expensive as the corpus grows. Earlier approaches such as fixed-length chunking or VAD-only segmentation also tend to add a post-processing step to fix boundaries, but that step corrects symptoms and adds another source of variation before training.

What changes when diarization is part of the pipeline

When diarization is treated as infrastructure rather than a preprocessing detail, the effects show up in four places: cleaner fine-tuning sets, a higher share of usable audio, less manual QA, and better routing between training stages.

Smallest AI is a good example because it trains models across the stack: Lightning for TTS, Pulse for STT, and Hydra, a speech-to-speech model built for production. Its pipeline has three stages: sourcing, then diarization and segmentation with pyannoteAI, then task-specific processing for each model type.

Before this setup, the team tried pyannote's open-source model, VAD-based chunking, and other open-source diarization tools. The same two problems came back each time: overlaps of around 100 ms that were not caught, and segment boundaries that cut through words. As Rishabh Dahale, Lead Data Scientist at Smallest AI, puts it, with those boundaries "the audio is polluted, because the start or end word is trimmed." Bad diarization output degraded the next stage of the pipeline, and the degradation was visible in the final model output.

After moving to pyannoteAI's commercial model, the team reported three changes:

  • Manual QA for overlap and boundary errors dropped substantially.

  • The post-processing step that corrected boundaries was removed, which gave the final stage cleaner input: "pyannote helped us eliminate that middle post-processing step."

  • Output that missed the bar for mid-training or fine-tuning is now routed to pre-training instead of being discarded.

The integration itself was light. Smallest AI runs on GCP, and pyannoteAI accepts signed URLs, which removed a separate audio upload step. "That simplified our whole life," says Rishabh. The main operational change was moving from polling for job completion to webhooks.

Diarization does not solve every recording. Very noisy files, heavy crosstalk, and cocktail-party audio can still produce weak results. Smallest AI treats this audio as low quality and drops it rather than forcing a fix, which is the correct decision when the alternative is training on a wrong timeline. Confidence scores make this decision systematic: low-confidence segments can be filtered out or sent to the tier that tolerates them.

Full-duplex models need conversations, not isolated clips

The usual output of a curation pipeline for STT or TTS is a set of short single-speaker clips. That format discards what an S2S model needs most: the timing between speakers. A full-duplex model learns from sessions in which both sides of the conversation are preserved and time-aligned.

This changes what diarization is used for. In an STT or TTS pipeline, the timeline is mostly used to cut audio. In an S2S pipeline, the timeline is kept as training data in its own right: it tells the model which speaker each stream belongs to and where each turn begins and ends. It also provides the statistics used to check a corpus: turn lengths, pause durations, overlap rates, and how often each speaker takes the floor.

The same applies to synthetic data, which labs increasingly mix with real recordings to fill specific gaps. Generated dialogues come with perfect labels, but they tend to be too regular and can inherit artifacts from the generator. Running diarization on synthetic sessions helps detect a generator that drifts to another voice mid-sentence, and comparing their turn-taking statistics with real conversations shows whether the synthetic data behaves like real speech. On real data, speaker intelligence makes the audio usable; on synthetic data, it makes it trustworthy.

Check a sample of your corpus with Precision-3

The fastest way to judge the timeline your data needs is to run diarization on a few representative sessions and look at the overlaps it finds. The script below sends one recording to the pyannoteAI API with Precision-3, waits for the result, and lists every region where two speakers are active at once. It flags short turns that sit entirely inside another speaker's turn as backchannel candidates, using a simple duration threshold you can adjust.

import os
import time

import requests

API = "<https://api.pyannote.ai/v1>"
HEADERS = {"Authorization": f"Bearer {os.environ['PYANNOTE_API_KEY']}"}
BACKCHANNEL_MAX_S = 1.0  # adjust to your data

# 1. Submit one session from your corpus (public or signed URL)
job = requests.post(
    f"{API}/diarize",
    headers=HEADERS,
    json={"url": "<https://storage.example.com/session-0001.wav>", "model": "precision-3"},
).json()

# 2. Wait for the result (use the webhook field instead of polling at scale)
while True:
    result = requests.get(f"{API}/jobs/{job['jobId']}", headers=HEADERS).json()
    if result["status"] in ("succeeded", "failed", "canceled"):
        break
    time.sleep(10)

if result["status"] != "succeeded":
    raise RuntimeError(f"Job {job['jobId']} ended with status {result['status']}")

# 3. List overlap regions and backchannel candidates
segments = sorted(result["output"]["diarization"], key=lambda s: s["start"])

for i, a in enumerate(segments):
    for b in segments[i + 1:]:
        if b["start"] >= a["end"]:
            break
        if b["speaker"] == a["speaker"]:
            continue
        end = min(a["end"], b["end"])
        inside = b["end"] <= a["end"]
        short = b["end"] - b["start"] < BACKCHANNEL_MAX_S
        kind = "backchannel candidate" if inside and short else "overlap"
        print(f"{b['start']:8.2f}s {end:8.2f}s  {a['speaker']} / {b['speaker']}  {kind}")
import os
import time

import requests

API = "<https://api.pyannote.ai/v1>"
HEADERS = {"Authorization": f"Bearer {os.environ['PYANNOTE_API_KEY']}"}
BACKCHANNEL_MAX_S = 1.0  # adjust to your data

# 1. Submit one session from your corpus (public or signed URL)
job = requests.post(
    f"{API}/diarize",
    headers=HEADERS,
    json={"url": "<https://storage.example.com/session-0001.wav>", "model": "precision-3"},
).json()

# 2. Wait for the result (use the webhook field instead of polling at scale)
while True:
    result = requests.get(f"{API}/jobs/{job['jobId']}", headers=HEADERS).json()
    if result["status"] in ("succeeded", "failed", "canceled"):
        break
    time.sleep(10)

if result["status"] != "succeeded":
    raise RuntimeError(f"Job {job['jobId']} ended with status {result['status']}")

# 3. List overlap regions and backchannel candidates
segments = sorted(result["output"]["diarization"], key=lambda s: s["start"])

for i, a in enumerate(segments):
    for b in segments[i + 1:]:
        if b["start"] >= a["end"]:
            break
        if b["speaker"] == a["speaker"]:
            continue
        end = min(a["end"], b["end"])
        inside = b["end"] <= a["end"]
        short = b["end"] - b["start"] < BACKCHANNEL_MAX_S
        kind = "backchannel candidate" if inside and short else "overlap"
        print(f"{b['start']:8.2f}s {end:8.2f}s  {a['speaker']} / {b['speaker']}  {kind}")
import os
import time

import requests

API = "<https://api.pyannote.ai/v1>"
HEADERS = {"Authorization": f"Bearer {os.environ['PYANNOTE_API_KEY']}"}
BACKCHANNEL_MAX_S = 1.0  # adjust to your data

# 1. Submit one session from your corpus (public or signed URL)
job = requests.post(
    f"{API}/diarize",
    headers=HEADERS,
    json={"url": "<https://storage.example.com/session-0001.wav>", "model": "precision-3"},
).json()

# 2. Wait for the result (use the webhook field instead of polling at scale)
while True:
    result = requests.get(f"{API}/jobs/{job['jobId']}", headers=HEADERS).json()
    if result["status"] in ("succeeded", "failed", "canceled"):
        break
    time.sleep(10)

if result["status"] != "succeeded":
    raise RuntimeError(f"Job {job['jobId']} ended with status {result['status']}")

# 3. List overlap regions and backchannel candidates
segments = sorted(result["output"]["diarization"], key=lambda s: s["start"])

for i, a in enumerate(segments):
    for b in segments[i + 1:]:
        if b["start"] >= a["end"]:
            break
        if b["speaker"] == a["speaker"]:
            continue
        end = min(a["end"], b["end"])
        inside = b["end"] <= a["end"]
        short = b["end"] - b["start"] < BACKCHANNEL_MAX_S
        kind = "backchannel candidate" if inside and short else "overlap"
        print(f"{b['start']:8.2f}s {end:8.2f}s  {a['speaker']} / {b['speaker']}  {kind}")

A few notes for production use:

  • The request does not set exclusive, so the output keeps overlapping segments. That is what the output S2S training data needs.

  • Add "turnLevelConfidence": true to the request to get a confidence score per turn, which you can use for routing between fine-tuning, mid-training, and pre-training.

  • For private buckets, pass a pre-signed URL with a short validity, or upload files to temporary storage and use a media:// reference.

  • Job output is deleted 24 hours after completion, so store the diarization next to the audio it describes.

Run it on a handful of sessions that represent your corpus, then compare the overlaps and turn boundaries with what your current segmentation produces on the same files.

Treat speaker intelligence as training-data infrastructure

S2S models cannot learn natural conversation from audio alone. They need to know who is speaking, when each turn begins and ends, where overlap occurs, and how speakers respond to each other. Diarization is the layer that provides this speaker-aware, time-aligned structure, and the accuracy of that layer sets a limit on the conversational behavior the model can learn.

If you are building an S2S, full-duplex, or voice-agent model, evaluate speaker intelligence as a data-preparation layer rather than a transcription add-on. Start with the script above on a representative sample of your corpus, and compare what Precision-3 returns with the output of your current segmentation.

To go further:

Speaker Intelligence Platform for developers

Detect, segment, label and separate speakers in any language.

Speaker Intelligence Platform for developers

Detect, segment, label and separate speakers in any language.

Make the most of conversational speech
with AI

Detect, segment, label and separate speakers in any language.