Speech-to-Speech vs Pipelined Voice Agents

#Introduction

We've been talking to companies building voice agent workflows (that is to say, a user speaks and the agent responds with speech) about how they've architected their agent systems. The standard system follows a pipelined loop that looks something like this.

user speech → ASR → transcript → LLM/tool loop → TTS → audio response

Theoretically natively multimodal models with the internal machinery to convert speech directly back to speech should have a latency advantage over pipelined approaches but historically most companies still prefer the pipelined solution. Although more complicated, they have a few distinct advantages:

• They are modular allowing companies to take advantage of the latest frontier models for the thinking component of the LLM / tool loop.

• Generating raw transcripts simplifies compliance requirements

• Serving infrastructure for multimodal models was much less mature

The tradeoff is, of course, speed; each call boundary adds latency and implementation complexity to the final response. A lot has changed in the past year with major developments both in the quality of open source multimodal models like Qwen3-Omni and the infrastructure necessary to efficiently serve them with vLLM-Omni so we wanted to know how a native multimodal model would stack up against traditional approaches from frontier labs.

#Methodology

STT: Samples were drawn from the LibriSpeech dataset with each run evaluating the same utterances across STT models. Reference comparisons were drawn from the attached transcripts for each sample.

TTS: Samples were drawn from LibriTTS-R, with each run evaluating the same prompts across TTS models. During the verification stage each output speech sample was run through ASR to compute WER/CER against normalized source text.

Agentic: Samples were drawn from Harper Valley Bank caller turns and reused across all workflow variants. Each workflow produced the next spoken agent reply, either via native speech-to-speech or staged transcribe → answer → speak pipelines. A verifier ASR transcribed the spoken output for transcript correctness, and a provider-backed judge assessed task success for each turn.

#Results

The headline result is that native speech-to-speech finally feels like more than a demo.

In our benchmarks, Qwen3-Omni provided the best overall balance between latency and quality as measured by task success. It combined the strongest task-level score in the set with the lowest average end-to-end latency, and it did so without the dropped-turn failures that showed up elsewhere.

A few systems were able to start talking very quickly, but "starts talking" and "successfully completes the turn" were often very different things. The end-to-end Cartesia agent is a good example: it was responsive and sounded polished, but it mostly failed the underlying banking tasks. The stronger multistage pipelines were more balanced, but they still paid the usual tax for going through transcription, text reasoning, and synthesis as separate steps.

So the interesting outcome here isn't that pipelines are suddenly obsolete. It's that, for the first time, the open native path actually looks preferable. If you care about realtime interaction, the extra boundaries in a transcribe → answer → speak stack are no longer obviously buying enough to justify themselves.

#Why vLLM-Omni matters

Qwen3-Omni is the kind of model that makes this shift possible: it is a natively end-to-end omni model that can take text, images, audio, and video and stream back both text and natural speech, and the Qwen project explicitly recommends the latest vLLM-Omni for deploying the Qwen3-Omni series.

But the model is only half the story. The reason vLLM-Omni changes the game is that it turns multimodal serving into infrastructure instead of bespoke glue code. The project extends vLLM beyond text-only generation into text, image, video, and audio workloads, supports streaming outputs and an OpenAI-compatible API server, and introduces a stage-oriented, disaggregated execution model for more complex multimodal systems.

That matters for direct agent to agent workflows because every forced conversion back into text is a design concession. If agent A has to listen, transcribe, reason, synthesize, and then hand audio off to agent B so B can transcribe it again, the system keeps paying serialization tax at every hop. A speech-native model lets speech be the medium rather than a temporary encoding, and a serving layer like vLLM-Omni gives that setup a shape you can actually operate.

#Takeaway

Pipelines still make sense when you want to swap best-of-breed components independently, or when your text model materially outperforms your native speech model. But if the question is whether open multimodal stacks are ready for serious realtime workflows, this benchmark pushed me from "not quite" to "yes, probably." Qwen3-Omni is what makes direct speech-in / speech-out agents look viable. vLLM-Omni is what makes them look deployable. And once those two things are true, direct agent to agent workflows stop sounding like a research demo and start looking like the default architecture.