Customer story
About Smallest AI
Smallest AI is a voice AI lab building its own models across the stack: Lightning for text-to-speech, Pulse for speech-to-text, Electron as a sub-3B language model, and Hydra as one of the first speech-to-speech models built for production. The company's premise is that small, specialized models can match or beat much larger ones when they're trained on the right data.
That thesis puts unusual weight on training data quality. Every model Smallest AI ships depends on audio that has been correctly split by speaker, with clean boundaries and no bleed between speakers.
“What Smallest is going for, what we are aiming for, is to be anything and everything related to voice AI. We started by building voice agents, and pretty quickly we started developing our own TTS and ASR models rather than relying on off-the-shelf ones." Rishabh Dahale, Lead Data Scientist at Smallest AI
For a team training its own models end-to-end, the data pipeline isn't a side process. It's what everything else is built on.
The data problem behind the models
Smallest AI sources close to a million hours of raw audio from a mix of vendor and custom collections. Before any of it can be used to train a model, it has to be cut into clean, speaker-separated chunks.
That sounds simple until you look at what the raw audio actually contains: long recordings, sometimes an hour or more, with multiple speakers, background noise, and constant interruptions. Getting from that to training-ready audio means solving two problems at once: knowing exactly who is speaking when, and cutting the audio without damaging it.
Smallest AI splits its training data into three tiers with different tolerances: pre-training, mid-training, and fine-tuning. Pre-training can absorb some noise. Fine-tuning cannot.
Rishabh Dahale
Lead Data Scientist at Smallest AI
Before pyannoteAI, the team tried three approaches: pyannote's open-source model, WAV/VAD-based chunking, and other open-source diarization tools. All three ran into the same two failures: speaker overlap that wasn't fully caught, and segment boundaries that landed mid-word.
"If we directly pick up those segment boundaries, the audio is polluted, because the start or end word is trimmed. And then how do we fix that becomes a major challenge, because we don't know where the word actually started or ended." Rishabh says.
The switch to pyannoteAI
Smallest AI benchmarked pyannoteAI's commercial model against the open-source options it had already tried and moved its pipeline over. Three things drove the decision.
First, accuracy on the two failure modes that mattered most: overlap detection and segment boundaries. The commercial model's cuts were clean enough to use directly, without the manual post-processing the team had been doing to compensate for the open-source output.
Second, output that the team could trust without extra correction. The segment boundaries provided by pyannote were quite sharp and quite good, and they could directly pick them up. Other diarization tools required manual correction that didn't reliably fix the underlying problem.
Third, the cost made sense at their processing volume.
“Processing speed was actually one of the last things we cared about. As long as the accuracy is there, and once the results come in, we don't have to do a lot of post-processing on top of them. That mattered more than speed. And with our enterprise deal, the costing made sense to us.”
Rishabh Dahale
Lead Data Scientist at Smallest AI
Integration was straightforward. Smallest AI's pipeline already ran on GCP, and pyannoteAI's support for signed URLs meant audio didn't need to be uploaded through a separate step. "That simplified our whole life," Rishabh says. The one real change on their side was operational rather than technical: moving from polling the API for job completion to using webhooks.
What changed
The clearest change was in manual QA. Before pyannoteAI, cleaning up overlap and boundary errors was a manual, recurring task. After switching, that work dropped substantially. Smallest AI team saw the manual QA reduced quite massively after the integration of pyannote Precision-2 model.
The team was also able to process more raw audio, because output that didn't meet the bar for mid-training or fine-tuning could still be routed into pre-training instead of being discarded or stuck in a correction queue. Today, Smallest AI processes upwards of 100,000 hours of audio through pyannoteAI every month, across English, Spanish, French, Hindi, and several regional Indian languages, in parallel.
"It was actually quite real," Rishabh says of the garbage-in, garbage-out problem the team had before. "Running the next stage of the pipeline on bad diarization output produced a very bad result, and we actually saw that translate into the model output itself. pyannote helped us eliminate that middle post-processing step and gave us good input going into the last stage of the pipeline."
Even if the team is careful not to overstate the attribution. Smallest AI's Pulse STT model currently ranks 3rd on Artificial Analysis's independent benchmark, but the team points out this reflects work across the whole pipeline, not one component in isolation.
Some audio still resists automated diarization. Very noisy recordings, heavy crosstalk, or cocktail-party-style audio with many overlapping voices can produce weaker results, including cases where no speaker is detected across an entire hour-long file. Smallest AI treats these as low-quality data and drops them rather than trying to force a fix.
What clean diarization enables
pyannoteAI sits in the second of Smallest AI's three-stage data pipeline: sourcing, then diarization and segmentation, then task-specific processing for each model. By the time audio reaches that third stage, it's already split into clean, speaker-labeled, boundary-accurate chunks, whether it's headed for TTS (which also needs denoising and emotion tagging) or ASR (which tolerates more variation).
That reliability is what lets Smallest AI keep scaling its training data without scaling its manual QA team in step. As the company expands into new languages and dialects, including low-resource Indian languages and Spanish and French variants, the diarization stage needs to hold up without added review overhead.
"The team has always been super responsive, super helpful whenever anything came up," Rishabh says of working with pyannoteAI. Issues have been infrequent and resolved quickly when they've come up.
What's next
Smallest AI's near-term priority is language coverage: expanding beyond the North American, European, and Indian markets it currently serves, with attention to regional Latin American, European Spanish, and Indian dialects. Real-time diarization for production voice agents is also a subject of experimentation.
Training a voice AI model and need to handle complex audio conditions? Get started with pyannoteAI.
Discover more stories
