Blog

Written by Audri Saulters, Research Team, pyannoteAI
Following our last post on VAD aggressiveness and crosstalk sensitivity, you may be wondering how well pyannote models perform in the presence of background noise. Using pyannote.metric’s TemporalErrorAnalysis tool, we can both answer this question and demonstrate why the Precision-3 model is a worthy upgrade from both pyannote’s community-1 and other competitor models.
Methodology
To evaluate how robust any speaker diarization model is to noise, we used the pyannote brouhaha model to get signal-to-noise ratio (SNR) values for all of the DIHARD benchmark audio. This measures how much of the main signal stands out from background noise. In the case of conversational audio data, the higher the SNR, the cleaner the speech, and the lower the SNR, the more speech is buried in background noise. The model continuously outputs these SNR values roughly every 15 milliseconds as time-series data, which, combined with model predictions and ground-truth annotations, can be used to get a detailed error analysis breakdown using the pyannote.metric TemporalErrorAnalysis tool. This tool works by aligning the fine-grained resolution of the time series data with the segment-based error analysis and accumulating the duration of each error type that occurs within various SNR bands.
Results

The above is the tool’s output, showing the error analysis breakdown by incremented SNR ranges. This shows us that a lot of the audio has some level of perceptible background noise, some small amounts where the audio is pristine with only speech (seen in the higher SNR ranges to the right), and a significant amount where background noise is just as loud or even louder than speech (lower SNR ranges to the left). Toward these lower SNR ranges, missed detections occur somewhat more often than the other error types, and overall, more errors occur in the presence of background noise.
Below is a percentage-based view of the error analysis, where the duration of errors is presented in proportion to the total audio in that SNR band. It compares the performance of Precision-3 and Community-1.

As noted before, performance generally degrades as background noise increases, and this is true for both models. However, using Community-1 as a baseline, we see a vast improvement in performance by Precision-3, where there are consistent gains across these lower SNR bands and as much as a 9% improvement in the -10 to 0 decibel range (where background noise is louder than the speech itself). Given that these gains are not in proportion to the duration of audio in each band but rather correlated with the level of SNR, we can see that Precision-3 is more robust to background noise than Community-1.
Also consider
While the tendency for degraded performance for speaker diarization models in noisier audio is somewhat intuitive, it can also be explained by the correlation between a greater number of simultaneous speakers and lower SNR, as seen below. This shows that performance in the presence of noise is related to the challenge of diarizing overlapping speech, and this reminds us that background speech is also noise and that there is usefulness in tuning VAD aggressiveness and crosstalk sensitivity.

Competitor Comparison

Above compares Precision-3's performance to previous pyannote and competitor speaker diarization models across different SNR levels. Competing for the best performance are Precision-2 and Precision-3, where the latter wins by an average of 0.8% across all SNR bands. Overall, we can see that Precision-3 maintains both the highest accuracy and best robustness to the presence of background noise in audio.
Key takeaway
Whether you are diarizing on polished single-speaker audio or a chaotic multi-speaker conversation with lots of background noise, Precision-3 is your best option, as it outperforms both competitors and previous pyannote models on every level of background noise as well as on every error type. Speaker diarization model performance begins to degrade as soon as there is any level of background noise, which is also introduced with the more speakers you have in your audio, so ensuring robustness to noise will improve the accuracy of your results and all downstream tasks.
Try it yourself
If you’re curious about how well any speaker diarization model performs in terms of noise or any time-series-based data you may have (e.g., acoustic metrics such as reverb, energy, presence of noise), try the TemporalErrorAnalysis tool yourself.
