How we measure

Every STT model transcribed the same LibriTTS-R utterances in the same order, 20 per model and transport. Each model was measured on every transport it supports: HTTP batch, HTTP streaming, and WebSocket. Every transport keeps its own clock, so a WebSocket figure is never pooled or plotted with an HTTP one. Latency is measured client-side with the network round trip included and voice activity detection excluded.

HTTP batch is timed from the request to the complete transcript. HTTP streaming is timed from the request to the first and last transcript payloads.

WebSocket is timed from the end of speech to the final transcript the provider confirms, not the last partial. Every clip is loudness-normalized to -20 dBFS, and one offline voice activity detector, the same for every provider, marks where its speech ends. The clip is cut 100 ms after that marker and streamed in real time over a session that is already open, with provider endpointing turned off, so the clock starts at the same point for every provider. Connection setup is excluded and the network round trip is included. The time to the first partial transcript is not reported on WebSocket, because it depends on how each provider segments speech and cannot be put on a common clock.

Accuracy is scored against each sample's reference transcript after Whisper's English text normalizer, lower-casing, and punctuation removal. HTTP WER is the mean across samples; WebSocket WER is pooled across the corpus, summing errors over every clip and dividing by the total reference words, so short clips do not outweigh long ones. Cost is the provider's published list rate, normalized with fixed conversion assumptions that are the same for every provider. It is not an invoice, and measured request costs do not enter the comparison. STT prices are per hour of input audio; token-priced output assumes 15,000 text tokens per hour.

Every TTS model synthesized the same LibriTTS-R prompts in the same order, 20 per model and transport. Each model was measured on every transport it supports: HTTP batch, HTTP streaming, and WebSocket. Every transport keeps its own clock, so a WebSocket figure is never pooled or plotted with an HTTP one. Latency is measured client-side with the network round trip included and voice activity detection excluded.

HTTP batch is timed from the request to the finished audio. HTTP streaming and WebSocket are timed from the request to the first and last audio payloads; on WebSocket the connection is already open.

Each output was transcribed by one reference ASR model, OpenAI GPT Transcribe, held constant across providers, and scored against the normalized source text, so WER and CER reflect the synthesized audio rather than differences in transcription. Cost is the provider's published list rate, normalized with fixed conversion assumptions that are the same for every provider. It is not an invoice, and measured request costs do not enter the comparison. TTS prices are per million input characters, assuming 60,000 characters per hour of generated speech.

Every configuration answered the same 10 spoken caller turns, each a single turn in a fresh session that was already open, with audio streamed in real time over WebSocket. This run measures latency only; no quality judge scored the replies and no task success is claimed.

Native speech-to-speech models and STT → LLM → TTS cascades are compared as one class. Every cascade uses Sprag Symphony as its language model over HTTP and runs as a live pipeline in which the language model and speech synthesis overlap, so the cascades differ only in their speech-to-text and text-to-speech models.

Provider endpointing is turned off. One shared voice activity detector, waiting for 500 ms of silence, decides when each caller has finished, and that decision reaches every configuration at the same instant. Response is timed from that decision to the first nonempty packet of reply audio, as received, with the network round trip included. The detector's own wait is identical across configurations, so it is left out of the figure.

Prices use published rates for a reference conversation hour of 30 minutes of caller speech and 30 minutes of reply, with instruction and history text, tools, and cache discounts excluded for every row.

Measured with huggingface/open_asr_leaderboard's own evaluation harness (pin cb614857), not this site's mmbench-based STT/TTS benchmarks. Symphony and Rhythm (run under its pre-rename open-weight checkpoint id, Qwen3-ASR-1.7B) were served under prod-representative configuration and evaluated with the harness's own provider client against its published test sets.

Whisper Large v3 values are the public leaderboard's own published results as of the run date, not independently reproduced. This page's shared-run guarantees (same inputs, same order, no per-provider preprocessing) apply to Symphony and Rhythm, which ran together under the pinned harness; Whisper did not participate in that run and is shown for reference against the public leaderboard's own reported results. English results cover AMI, Earnings22, Gigaspeech, LibriSpeech clean/other, SPGISpeech, and Voxpopuli; multilingual results cover FLEURS, Common Voice, and MLS across German, French, Spanish, Italian, and Portuguese, limited to the language/dataset pairs the leaderboard reports for all three models.