Written by Audri Saulters, Research Team, pyannoteAI

Why do we care about gender and age?

Tasks that rely on self-supervised learning, including automatic speech recognition, speech translation, and text-to-speech, have a well-documented history of uneven performance across demographic groups such as gender, race, and disability. These disparities typically reflect imbalances in training data, and they carry real consequences: when a model performs disproportionately worse on certain groups, particularly marginalized ones, it can perpetuate and amplify existing harms.

Within speaker diarization specifically, a known challenge is the acoustic similarity between women's and children's voices, whose overlapping pitch and spectral characteristics can cause them to be confused with one another. This matters most in conversations where women and children are speaking together, where such confusion can degrade diarization quality and affect any downstream tasks that depend on speaker attribution, such as transcription or translation. Speaker diarization is a language-agnostic task, but it should also be equally robust to demographic variation, including age and gender. Yet, children's voices are acoustically challenging for diarization systems, and they appear most frequently in high-stakes settings such as pediatric clinical interviews where attribution errors carry great consequences, from child welfare proceedings to educational recordings.

At pyannoteAI, we investigated how well our newly released Precision-3 model and open-source Community-1 model perform with respect to gender. We specifically investigated whether male, female, or children's voices are disproportionately affected by diarization errors.

Methodology

To measure robustness to gender, we applied a voice type classifier (VTC) to DIHARD-III, one of our standard benchmarks, to assign gender labels to each speaker. We combined these labels with the ground-truth annotations to determine the gender composition of the benchmark, and also used these labels to determine how the models’ diarization errors are distributed across gender groups.

Given the limitations of the available classifiers, speakers are treated as one of three categories: adult male, adult female, or child (without an assumed gender). The VTC produces a diarization-like output, where it segments the audio speech and labels each segment with a predicted gender. Each speaker was assigned to their most probable category based on a majority vote across their labeled segments, which helps smooth out spurious per-segment predictions. For segments where the VTC returned low confidence across all categories, we fell back to the highest-scoring prediction to avoid leaving reference segments unannotated.

We then categorized three types of diarization errors: missed detections, false alarms, and confusion errors by the gender of the affected speaker. Confusion errors were further broken down into same-gender confusion (e.g. one female speaker confused for another female speaker) and cross-gender confusion (e.g. a female speaker attributed to a male speaker). Error rates were then normalized by the total speech duration for each gender group, making them comparable across groups of different sizes. Notably, adult men and women were almost equally represented in the dataset, while children’s voices represented a smaller percentage of the data.

Results

Community-1

The chart below shows the percentage of each group’s speech affected by each error type. Hatching indicates errors occurring within overlapping speech regions.

From Community-1, women are slightly more affected by missed detections than men, while men are slightly more affected by false alarms, though both differences are modest. The more obviously affected group is children: the child group is disproportionately affected across all three error types. In confusion errors especially, cross-category confusion of children reaches 6.9%, meaning that nearly 7% of children’s speech is attributed to a different, adult speaker. To clarify, this means the model is assigning children’s speech to the wrong speaker, and that speaker tends to be categorized as an adult.

Precision-3

Precision-3 shows an improvement over Community-1across every error type for every voice type category, and missed detections and false alarms are more evenly distributed between male and female speakers. Overall, these are positive findings as this suggests that pyannote models do not systematically disadvantage either adult gender group, and Precision’s performance on child speech is much better.

Confusion Breakdown

Since the most notable improvement from Community-1 to Precision-3 is in confusion errors on children’s speech (with cross-category errors dropping from 6.9% to 1.0%), we use a confusion breakdown heatmap to understand where exactly these cross-category errors were occurring.

In Community-1 that 6.9% (rounded up to 7% in the plot) of children’s cross-category confusion is exclusively predicted to be adult female speakers. Notably, no female speech is confused for children’s speech, suggesting that the issue is not sourced from biases within the female training data. Instead, this shows that the main area for improvement is performance on child speech, which is where Precision-3 delivers that improvement, as only 1% of children’s speech is confused for adult female speech and the child-for-other-child confusion rate drops to 3%, becoming comparable to the intra-category confusion of the adult voice types. This reduction in child-as-adult female confusion is the largest driver in Precision-3’s overall improvement on children’s speech.

Children's voices, however, remain the most affected group, as the intra-category error rate alone is still higher than the total confusion rate for adult voices. This reflects the inherent acoustic difficulty of processing children's speech: beyond its overlap with adult female frequencies, it is more frequently affected by recording artifacts, disfluencies, and vocal variability, and these factors compound the segmentation challenge regardless of model quality.

Why Precision-3?

The results of this analysis demonstrate that Precision-3 is both more accurate and more equitable. The performance gap between all demographic groups is smaller, and the groups most vulnerable to confusion errors (particularly children) benefit the most from the improvement.

We recommend using Precision-3, especially in cases where children are present in the audio, as children's speech is not evenly distributed across use cases: they concentrate in domains where accurate speaker attribution is most consequential. A misattributed segment in a clinical trial transcript, safeguarding interview, or classroom recording is not a minor quality issue, as it can affect medical records, legal proceedings, or educational assessments. Halving the cross-gender confusion rate on children's voices, as Precision-3 does relative to Community-1, is therefore both a technical improvement and an improvement in trustworthiness in settings where its failure would otherwise cause the most harm.

Limitations

This analysis has some limitations worth acknowledging:

• The VTC is an automated system and not error-free. Without ground-truth gender annotations for DIHARD-III, its actual accuracy on this dataset is unknown. A small percentage of speakers whose gender could not be determined by the VTC were excluded from this analysis.

• DIHARD-III contained relatively little children’s speech. Although one subdomain focuses on clinical interactions involving children and they appear in other subdomains as well, they account for less than 5% of total speech duration. Error estimates for this group carry more uncertainty as a result.

• This analysis cannot speak to potential harms affecting transgender or gender non-conforming individuals. This is a limitation of the specific VTC used and the broader landscape of available annotated data. Much in the way that not all cisgender women or all cisgender men sound the same, not all transgender people sound the same, and creating a classifier to identify when a speaker is transgender or gender non-conforming is ethically problematic. The risks of misclassification and creating a new vector for identifying a marginalized group outweigh analytical benefit, and this is an area where the absence of findings reflects a gap in the analysis but not an absence of concern.

What does this mean for your Voice AI pipeline?

If you're building pipelines that handle speech from mixed-gender conversations, the choice of diarization model has measurable consequences for accuracy across your speakers. Community-1 is a strong general-purpose option, but if your audio regularly includes children's voices or scenarios where women and children speak together, Precision-3's reduced confusion rates translate into more accurate and reliable output.

While benchmark scores such as diarization error rate (DER) are helpful indicators of a model’s performance, it's worth asking not just how accurate it is, but accurate for whom. These results are a reminder to consider models that are tested on the actual demographic range of your users.

Try the analysis yourself!

This repository (attach link) contains a tutorial to try out the analysis tool used in this blog, where you can assess a speaker diarization model’s robustness to any speaker-stable category (age, gender, accent, and so forth). If you have prediction and ground-truth annotations, you can add either automatically generated annotations or a manual mapping of speaker labels to categories to investigate a model’s inclusivity for yourself. The repository includes sample data too.