Speech to speech

Speech to speech lets you build a voice conversation with a single model over a single WebSocket. You stream the user's audio in, the model decides what to say, and the reply streams back as audio. There is no transcription step to run, no text model in the middle, and no synthesis call to chain: one session carries the whole exchange.

This page covers connecting, configuring the session, and handling audio in both directions. Models lists the models that serve speech-to-speech sessions; the examples below use symphony.

If you already have the words you want spoken, use text to speech. If you want a transcript without a spoken reply, open the same endpoint with intent=transcription, covered under speech to text.

How a session works

A session is a WebSocket connection that stays open for the length of a conversation. Once it is open, the exchange follows a fixed rhythm:

• Configure once. After the server sends session.created, send session.update with your instructions and turn-detection settings. The server confirms with session.updated.
• Stream audio up. Send captured audio as base64 PCM16 inside input_audio_buffer.append events. Server-side voice activity detection finds where each user turn starts and stops.
• Receive audio down. The reply arrives as response.output_audio.delta events, followed by response.output_audio.done and response.done.
• Interrupt at any time. A new user turn cancels a reply that is still streaming, so the conversation stays responsive.

Connection

Sessions run over WebSocket. Name the model in the query string:

wss://api.sprag.ai/v1/realtime?model=symphony

Authentication

Authenticate the upgrade with your API key in the bearer header:

Authorization: Bearer $SPRAG_API_KEY

A client that cannot set headers on a WebSocket upgrade, such as the browser WebSocket API, passes the key as a subprotocol pair instead: sprag-api-key followed by the key.

new WebSocket("wss://api.sprag.ai/v1/realtime?model=symphony", [
  "sprag-api-key",
  spragApiKey,
]);

A missing or rejected key fails the upgrade with 401, and an unknown model with 404, before the socket opens.

Quickstart

The fastest way to a working conversation is the OpenAI Agents SDK, which speaks the realtime protocol and drives turn detection for you. Your app supplies audio capture and playback.

npm install @openai/agents

import {
  OpenAIRealtimeWebSocket,
  RealtimeAgent,
  RealtimeSession,
} from "@openai/agents/realtime";

class SpragRealtimeWebSocket extends OpenAIRealtimeWebSocket {
  _interrupt() {
    this.emit("audio_interrupted");
  }
}

const transport = new SpragRealtimeWebSocket({
  useInsecureApiKey: true,
  createWebSocket: async ({ url, apiKey }) =>
    new WebSocket(url, ["sprag-api-key", apiKey]),
});

const agent = new RealtimeAgent({
  name: "Voice assistant",
  instructions: "Be concise and helpful.",
});

const session = new RealtimeSession(agent, {
  transport,
  model: "symphony",
  config: {
    audio: {
      input: {
        format: { type: "audio/pcm", rate: 16000 },
        transcription: null,
        turnDetection: {
          type: "server_vad",
          threshold: 0.8,
          silenceDurationMs: 1500,
        },
      },
    },
  },
});

session.on("audio", ({ data }) => playPcm16(data, 24000));

await session.connect({
  apiKey: process.env.SPRAG_API_KEY,
  url: "wss://api.sprag.ai/v1/realtime?model=symphony",
});

capturePcm16(16000, (chunk) => session.sendAudio(chunk));

Session configuration

After the server emits session.created, send session.update and wait for session.updated before streaming audio. Three settings shape the conversation:

• instructions: the system prompt. Use it for persona, behavior, and task guidance, as you would in a chat request.
• turn_detection.threshold: the speech-probability floor. Raise it to reject breaths and background noise.
• turn_detection.silence_duration_ms: the quiet time that ends a turn. Raise it to keep mid-sentence pauses inside one turn.

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "instructions": "System prompt or behavior instructions.",
    "audio": {
      "input": {
        "format": { "type": "audio/pcm", "rate": 16000 },
        "transcription": null,
        "turn_detection": {
          "type": "server_vad",
          "threshold": 0.8,
          "silence_duration_ms": 1500
        }
      }
    }
  }
}

Audio format

Input is linear PCM16. Declare your capture rate in audio.input.format, as an object of the form {"type": "audio/pcm", "rate": 16000}. Transcode other codecs, including G.711 telephony audio, to PCM16 before the session.

Send audio base64-encoded in input_audio_buffer.append events. Server VAD closes each turn for you.

Output is PCM16 at 24 kHz unless you declare another format in audio.output.format:

{
  "type": "session.update",
  "session": {
    "audio": {
      "output": { "format": { "type": "audio/mpeg", "rate": 24000 } }
    }
  }
}

A field you leave out keeps its current value, and session.updated states the format in force. If the format or rate is not in the table, the server sends an error event with the code unsupported_audio_format and applies none of the update.

The deltas of one response, joined in order, form one stream. An audio/wav stream carries its header in the first delta and is written before its length is known, so read its duration from the audio rather than from the header. Each response starts a new stream.

A new format applies from the next response. If a response is in progress or a committed turn is waiting for one, the server sends an error event with the code output_format_change_in_flight, applies the rest of the update, and keeps the current format.

Receiving audio

With the Agents SDK, the session.on("audio", ...) handler in the quickstart receives each delta as it arrives.

Barge-in

A new user turn cancels a reply that is still streaming. Stop local playback at the same moment, since audio already queued on your side would otherwise keep playing. The transport subclass in the quickstart does this for you.

Limits

If your organization is at its cap of concurrent sessions, the WebSocket upgrade fails with HTTP 429 and the error code rate_limit_exceeded. More information about rate limits can be found in the API docs.

Browsers and most WebSocket libraries answer pings automatically. If yours answers only while a receive is pending, keep receiving between turns.

Close codes