# Speech-to-Speech vs the Traditional Voice Pipeline: The cost of Modularity Voice agents usually chain ASR, an LLM, and TTS. Every handoff adds latency, and transcription throws away tone and emphasis before the synthesizer sees them. Omni models collapse that into one system. We built Symphony on Qwen3-Omni: against our own Rhythm + Chorus pipeline, it finishes a turn about 500 ms sooner. Pipelines still win for component swaps, compliance transcripts, and voice cloning. ## The traditional stack Most AI speech-to-speech pipelines look the same. **user speech → ASR → transcript → LLM / tool loop → TTS → audio response** The modularity of this pipeline is highly favorable to developers (though each step can add latency and increase the margin of error for lost context, addressed in the next section). In April we benchmarked this loop against native speech-to-speech and found the native path has become more competitive (see: [Speech-to-Speech vs Pipelined Voice Agents](/blog/speech-to-speech-vs-pipelined-voice-agents)). The user never hears ASR or the LLM. They hear synthesized audio, so that stage carries whatever the rest of the stack did, and it is the stage that changes the most when the loop collapses into a single model. ## What the boundaries cost **Latency stacks.** Time to first audio is the sum of three model calls plus the network between them, and none of it overlaps. ASR waits until it is confident the user stopped talking. The LLM waits for that transcript, then has to produce enough text for synthesis to start. TTS waits for that text, then produces its first packet. Every stage is blocked on the one before it, so the delays add up instead of absorbing each other, and the slowest box sets the floor for everything downstream. The TTFA numbers further down are what that addition costs in practice. **Text is a lossy middleman.** This is the part that is discussed much less. The user's audio carried hesitation, stress, pitch, pacing, and whatever was audible in how they said it. ASR returns a string. By the time TTS runs, every acoustic cue is gone, and the synthesizer is reconstructing prosody from punctuation and whatever style tags the application remembered to attach. The pipeline asks a model to guess at something another model in the same request already knew and discarded. These characteristics of speech are arguably as vital to understanding the messaging of what someone said as the actual words they spoke. ## What an omni model changes An omni model can take audio in and return audio out, so the synthesizer is not reading a transcript. It is generating from the same internal representation that processed the input. Qwen3-Omni is the open model that makes this practical. It uses a Thinker-Talker Mixture-of-Experts architecture, where the Thinker handles multimodal understanding and text generation and the Talker produces streaming speech tokens directly from the Thinker's representation. It uses multi-codebook speech codec prediction and a lightweight causal ConvNet for streaming generation, with a theoretical cold-start first-packet latency of 234 ms. It supports text interaction in 119 languages, speech understanding in 19, and speech generation in 10 (see: [Qwen3-Omni Technical Report](https://arxiv.org/abs/2509.17765)). Sprag built on top of Qwen3-Omni to produce Symphony while leveraging vLLM-Omni as hosted inference. Weights are open and the architecture is documented in the [Qwen3-Omni repo](https://github.com/QwenLM/Qwen3-Omni). The practical difference for the TTS stage: synthesis stops being a final hop with a text-shaped input and becomes part of generation. ## Serving has been a blocker Omni models have existed for a while. What kept them out of production stacks was the operational cost of deploying and running one. vLLM-Omni has changed that. It decomposes multimodal architectures into interconnected stages with independent execution backends, per-stage batching, flexible GPU allocation, and connectors between stages. In the project's own evaluation, that reduces job completion time by up to 91.4 percent against baseline methods. The direct mapping of Qwen3-Omni is a good example. Thinker handles understanding and semantic generation, Talker converts that to speech codec codes, and Code2Wav turns codes into waveform. Each stage batches and streams independently rather than forcing the whole request down one execution path, which is what keeps the speech stage from reintroducing the serialization tax the pipeline was paying. ## The numbers Our agentic benchmark runs 30 caller turns from the Harper Valley Bank set through each pipeline to the next spoken reply. Cascaded rows carry the full sequential STT, LLM, and TTS latency. The interactive chart lives on the [benchmarks page](/benchmarks?tab=agentic). | Provider / Model | TTFA p50 | TTFA p95 | E2E p50 | | --------------------------------- | -------- | -------- | -------- | | Sprag Symphony | 1,181 ms | 1,780 ms | 1,632 ms | | Sprag Rhythm + Chorus | 1,212 ms | 2,007 ms | 2,150 ms | | Cartesia Ink + Sonic 3 | 1,310 ms | 2,995 ms | 3,445 ms | | OpenAI GPT Audio | 1,458 ms | 1,975 ms | 3,168 ms | | Cartesia Line Agent | 1,896 ms | 2,893 ms | 7,443 ms | | ElevenLabs Scribe v2 + Flash v2.5 | 2,864 ms | 3,389 ms | 5,115 ms | | AssemblyAI Voice Agent | 3,107 ms | 4,127 ms | 9,076 ms | | Deepgram Voice Agent | 3,299 ms | 4,447 ms | 5,338 ms | | Deepgram Nova 3 + Aura 2 | 4,246 ms | 6,846 ms | 4,588 ms | The cleanest comparison in that table is between our own two rows. Symphony and Rhythm + Chorus run on the same infrastructure, against the same samples, with the same tuning effort behind them. The only difference is that one collapses the loop and the other chains STT and TTS around an LLM. They start speaking at nearly the same moment, 1,181 ms against 1,212 ms, but the full turn completes in 1,632 ms versus 2,150 ms. > [!TIP] > > Time to first audio hides more than it reveals. Cartesia's Line Agent starts talking at 1,896 ms and does not finish the turn until 7,443 ms. Deepgram's Nova 3 + Aura 2 is the slowest to start in the set and still finishes ahead of it. The goal is to optimize for the whole turn and not just the first packet. ## Where pipelines still win The traditional pipeline still has advantages, especially in the TTS stage. **Component swaps.** If a better reasoning model ships next month, a modular pipeline takes it without touching the voice path. An omni model upgrades as one unit. **Compliance that needs transcripts.** Some workloads require a raw, auditable transcript of both sides. A traditional pipeline produces these as a byproduct. With an omni model you are adding a path to get that component out. **Deeper voice customization.** Cloning from reference audio and designing a voice from a description are still better served by dedicated TTS models. Qwen3-TTS can still be run for this use, but it is a different tool from the agent loop. ## Takeaway Traditional pipelines are not going away in the short term. But the pipeline is no longer the obvious default. Symphony starts a turn at the same moment our pipeline does and finishes it half a second sooner, without a transcript in the middle. Keeping a separate TTS hop now means paying for three sequential boundaries and a text bottleneck to get modularity that you may not need. If you are building realtime voice, benchmark the native path against your own workload before assuming the pipeline is the safe default. [The Sprag team can help you set that up](/contact-sales).