Speech-to-Speech vs the Traditional Voice Pipeline: The cost of Modularity

Voice agents usually chain ASR, an LLM, and TTS. Every handoff adds latency, and transcription throws away tone and emphasis before the synthesizer sees them. Omni models collapse that into one system. We built Symphony on Qwen3-Omni: against our own Rhythm + Chorus pipeline, it finishes a turn about 500 ms sooner. Pipelines still win for component swaps, compliance transcripts, and voice cloning.

#The traditional stack

Most AI speech-to-speech pipelines look the same.

user speech → ASR → transcript → LLM / tool loop → TTS → audio response

The modularity of this pipeline is highly favorable to developers (though each step can add latency and increase the margin of error for lost context, addressed in the next section).

In April we benchmarked this loop against native speech-to-speech and found the native path has become more competitive (see: Speech-to-Speech vs Pipelined Voice Agents). The user never hears ASR or the LLM. They hear synthesized audio, so that stage carries whatever the rest of the stack did, and it is the stage that changes the most when the loop collapses into a single model.

#What the boundaries cost

Latency stacks. Time to first audio is the sum of three model calls plus the network between them, and none of it overlaps. ASR waits until it is confident the user stopped talking. The LLM waits for that transcript, then has to produce enough text for synthesis to start. TTS waits for that text, then produces its first packet. Every stage is blocked on the one before it, so the delays add up instead of absorbing each other, and the slowest box sets the floor for everything downstream. The TTFA numbers further down are what that addition costs in practice.

Text is a lossy middleman. This is the part that is discussed much less. The user's audio carried hesitation, stress, pitch, pacing, and whatever was audible in how they said it. ASR returns a string. By the time TTS runs, every acoustic cue is gone, and the synthesizer is reconstructing prosody from punctuation and whatever style tags the application remembered to attach. The pipeline asks a model to guess at something another model in the same request already knew and discarded. These characteristics of speech are arguably as vital to understanding the messaging of what someone said as the actual words they spoke.

#What an omni model changes

An omni model can take audio in and return audio out, so the synthesizer is not reading a transcript. It is generating from the same internal representation that processed the input.

Qwen3-Omni is the open model that makes this practical. It uses a Thinker-Talker Mixture-of-Experts architecture, where the Thinker handles multimodal understanding and text generation and the Talker produces streaming speech tokens directly from the Thinker's representation. It uses multi-codebook speech codec prediction and a lightweight causal ConvNet for streaming generation, with a theoretical cold-start first-packet latency of 234 ms. It supports text interaction in 119 languages, speech understanding in 19, and speech generation in 10 (see: Qwen3-Omni Technical Report). Sprag built on top of Qwen3-Omni to produce Symphony while leveraging vLLM-Omni as hosted inference.

Weights are open and the architecture is documented in the Qwen3-Omni repo.

The practical difference for the TTS stage: synthesis stops being a final hop with a text-shaped input and becomes part of generation.

#Serving has been a blocker

Omni models have existed for a while. What kept them out of production stacks was the operational cost of deploying and running one.

vLLM-Omni has changed that. It decomposes multimodal architectures into interconnected stages with independent execution backends, per-stage batching, flexible GPU allocation, and connectors between stages. In the project's own evaluation, that reduces job completion time by up to 91.4 percent against baseline methods.

The direct mapping of Qwen3-Omni is a good example. Thinker handles understanding and semantic generation, Talker converts that to speech codec codes, and Code2Wav turns codes into waveform. Each stage batches and streams independently rather than forcing the whole request down one execution path, which is what keeps the speech stage from reintroducing the serialization tax the pipeline was paying.

#The numbers

Our agentic benchmark runs 30 caller turns from the Harper Valley Bank set through each pipeline to the next spoken reply. Cascaded rows carry the full sequential STT, LLM, and TTS latency. The interactive chart lives on the benchmarks page.

The cleanest comparison in that table is between our own two rows. Symphony and Rhythm + Chorus run on the same infrastructure, against the same samples, with the same tuning effort behind them. The only difference is that one collapses the loop and the other chains STT and TTS around an LLM. They start speaking at nearly the same moment, 1,181 ms against 1,212 ms, but the full turn completes in 1,632 ms versus 2,150 ms.

#Where pipelines still win

The traditional pipeline still has advantages, especially in the TTS stage.

Component swaps. If a better reasoning model ships next month, a modular pipeline takes it without touching the voice path. An omni model upgrades as one unit.

Compliance that needs transcripts. Some workloads require a raw, auditable transcript of both sides. A traditional pipeline produces these as a byproduct. With an omni model you are adding a path to get that component out.

Deeper voice customization. Cloning from reference audio and designing a voice from a description are still better served by dedicated TTS models. Qwen3-TTS can still be run for this use, but it is a different tool from the agent loop.

#Takeaway

Traditional pipelines are not going away in the short term. But the pipeline is no longer the obvious default. Symphony starts a turn at the same moment our pipeline does and finishes it half a second sooner, without a transcript in the middle. Keeping a separate TTS hop now means paying for three sequential boundaries and a text bottleneck to get modularity that you may not need.

If you are building realtime voice, benchmark the native path against your own workload before assuming the pipeline is the safe default. The Sprag team can help you set that up.