How we measure

Samples were drawn from the LibriSpeech dataset with each run evaluating the same utterances across STT models. Reference comparisons were drawn from the attached transcripts for each sample.

All providers received the same audio payloads in identical order. Latency is measured client-side from request dispatch to first transcript token (TTFT). Accuracy is computed against the reference transcripts after text normalization: number and hyphenated-compound canonicalization, case-folding, punctuation stripping, and whitespace collapse.

Samples were drawn from LibriTTS-R, with each run evaluating the same prompts across TTS models. During the verification stage each output speech sample was run through ASR to compute WER/CER against normalized source text.

Latency is measured from request dispatch to the first audio chunk returned by the provider. Quality verification uses a single ASR model held constant across providers so the resulting WER/CER reflects the synthesized audio, not differences in transcription.

Samples were drawn from Harper Valley Bank caller turns and reused across all workflow variants. Each workflow produced the next spoken agent reply, either via native speech-to-speech or staged transcribe → answer → speak pipelines.

A verifier ASR (GPT-4o Transcribe) transcribed the spoken output for transcript correctness, and a GPT Audio judge scored the naturalness of each synthesized reply. Task-level goal completion is judged by GPT-5.1 in a separate evaluation suite and is not reported on this page. Multistage pipelines incur the cumulative latency of each component; native speech-to-speech models are evaluated end-to-end as a single call.

Measured with huggingface/open_asr_leaderboard's own evaluation harness (pin cb614857), not this site's mmbench-based STT/TTS benchmarks. Symphony and Rhythm (run under its pre-rename open-weight checkpoint id, Qwen3-ASR-1.7B) were served under prod-representative configuration and evaluated with the harness's own provider client against its published test sets.

Whisper Large v3 values are the public leaderboard's own published results as of the run date, not independently reproduced. This page's shared-run guarantees (same inputs, same order, no per-provider preprocessing) apply to Symphony and Rhythm, which ran together under the pinned harness; Whisper did not participate in that run and is shown for reference against the public leaderboard's own reported results. English results cover AMI, Earnings22, Gigaspeech, LibriSpeech clean/other, SPGISpeech, and Voxpopuli; multilingual results cover FLEURS, Common Voice, and MLS across German, French, Spanish, Italian, and Portuguese, limited to the language/dataset pairs the leaderboard reports for all three models.