Speech to speech

One multimodal model hears the user and speaks the answer over a single WebSocket. You send audio, the model decides what to say, and the reply comes back as audio. The realtime model is Symphony.

The session takes audio, not strings. To speak a sentence you already have, use text to speech. To transcribe without a spoken reply, open the session with intent=transcription instead, covered under speech to text.

Connection

Realtime sessions run over WebSocket only. There is no WebRTC transport, no SDP exchange, and no separate signaling endpoint.

wss://api.sprag.ai/v1/realtime?model=symphony

Authentication

Server-side clients authenticate the upgrade with a bearer token.

Authorization: Bearer $SPRAG_API_KEY

Browsers cannot set that header on a WebSocket upgrade, so they pass the credential as the second WebSocket subprotocol instead: sprag-jwt with a short-lived Sprag JWT, or sprag-api-key with a raw key. Use sprag-jwt in frontend code; a raw key does not belong there.

Handle a rejected credential or an unknown model as an HTTP 403 on the upgrade. Both fail before the socket opens, so no close frame arrives.

Quickstart

Install @openai/agents, then point the realtime WebSocket transport at Sprag. Your app supplies PCM capture and playback; the SDK handles the protocol and the server-VAD state machine.

Use the transport subclass below. The stock transport's barge-in handling expects server-side response state that Sprag sessions do not keep, so without it, interrupting the model mid-reply fails instead of stopping playback.

import {
  OpenAIRealtimeWebSocket,
  RealtimeAgent,
  RealtimeSession,
} from "@openai/agents/realtime";

class SpragRealtimeWebSocket extends OpenAIRealtimeWebSocket {
  // Sprag cancels the reply server-side; only local playback needs stopping.
  _interrupt() {
    this.emit("audio_interrupted");
  }
}

const credential = await getShortLivedSpragJwt();
const transport = new SpragRealtimeWebSocket({
  useInsecureApiKey: true,
  createWebSocket: async ({ url, apiKey }) =>
    new WebSocket(url, ["sprag-jwt", apiKey]),
});

const agent = new RealtimeAgent({
  name: "Voice assistant",
  instructions: "Be concise and helpful.",
});

const session = new RealtimeSession(agent, {
  transport,
  model: "symphony",
  config: {
    audio: {
      input: {
        format: { type: "audio/pcm", rate: 16000 },
        transcription: null,
        turnDetection: {
          type: "server_vad",
          threshold: 0.8,
          silenceDurationMs: 1500,
        },
      },
    },
  },
});

session.on("audio", ({ data }) => playPcm16(data, 24000));

await session.connect({
  apiKey: credential,
  url: "wss://api.sprag.ai/v1/realtime?model=symphony",
});

capturePcm16(16000, (chunk) => session.sendAudio(chunk));

Session configuration

After the server emits session.created, send session.update and wait for session.updated. Three settings change what you hear: instructions, and threshold and silence_duration_ms under turn_detection.

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "instructions": "System prompt or behavior instructions.",
    "audio": {
      "input": {
        "format": { "type": "audio/pcm", "rate": 16000 },
        "transcription": null,
        "turn_detection": {
          "type": "server_vad",
          "threshold": 0.8,
          "silence_duration_ms": 1500
        }
      }
    }
  }
}

instructions is the system prompt forwarded to the model: persona, behavior, task guidance. A higher threshold rejects breaths and background noise, and a longer silence_duration_ms keeps ordinary mid-sentence pauses inside one turn.

Do not rely on other keys; they are accepted without effect. output_modalities is echoed back and ignored, so a session declaring ["text"] still answers with audio, and create_response and interrupt_response change nothing.

Audio format

Generated speech is 24 kHz PCM16. A rate declared under audio.output.format is discarded and session.updated echoes 24000 back, so resample on your side if playback needs something else.

Declare the input format as an object carrying your capture rate, as in the quickstart. The bare pcm16 string carries no rate and leaves the session at its 24 kHz starting value, which turns 16 kHz microphone audio into slow, pitched-down speech. G.711 is refused in both of its spellings, the bare g711_ulaw and g711_alaw strings and the object types audio/pcmu and audio/pcma: the session.update is discarded whole and an unsupported_audio_format error comes back. An unrecognized format type is ignored instead, which leaves the session on its previous format with no error, so read session.updated rather than assuming a declaration took. Transcode telephony audio before the session.

Audio goes up base64-encoded in input_audio_buffer.append events, and server VAD closes each turn for you.

Receiving audio

Audio arrives as response.output_audio.delta events with base64-encoded PCM16 payloads. response.output_audio.done closes the audio part, and response.done closes the response. With the Agents SDK this is the session.on("audio", ...) handler above.

Barge-in

Barge-in is unconditional: a new user turn cancels a response still streaming, and no session setting turns that off. Stop local playback at the same moment, since buffers already queued on your side keep playing after the stream stops. Under the Agents SDK, the transport subclass in the quickstart does that.

Session limits

The default cap is 25 concurrent sessions per user.