Speech to text in realtime

To transcribe a live stream without cutting it into files, open a realtime WebSocket in transcription mode. Text arrives as OpenAI-compatible input-transcription delta and completed events.

Rhythm decodes a turn once the turn closes, so deltas for a turn follow its input_audio_buffer.speech_stopped rather than arriving mid-utterance. One session covers a whole conversation, and the server finds the turn boundaries, which is what this transport buys over one REST request per turn.

Connection

Realtime sessions run over WebSocket only. There is no WebRTC transport, no SDP exchange, and no separate signaling endpoint.

wss://api.sprag.ai/v1/realtime?intent=transcription

intent=transcription selects transcription over spoken conversation: the model transcribes and does not answer. For a session that speaks back, see speech to speech.

Authentication

Server-side clients authenticate the upgrade with a bearer token.

Authorization: Bearer $SPRAG_API_KEY

Browsers cannot set that header on a WebSocket upgrade, so they pass the credential as a subprotocol pair instead: sprag-jwt with a short-lived Sprag JWT, or sprag-api-key with an API key. Use the JWT from a browser, and never ship a long-lived API key in frontend code.

new WebSocket("wss://api.sprag.ai/v1/realtime?intent=transcription", [
  "sprag-jwt",
  shortLivedSpragJwt,
]);

A rejected credential or a rate limit fails the HTTP upgrade with 403 before the socket opens, so handle those failures as a connection error rather than a close frame.

Declaring the model

A transcription session starts with no model. Send a session.update naming one under session.audio.input.transcription.model within ten seconds of connecting, or the server closes the socket with code 1008. An unrecognized model id closes it the same way.

Name rhythm or symphony, the models that serve transcription sessions. Leave session.type out: a transcription session answers "type": "realtime" with an unsupported_session_type error.

{
  "type": "session.update",
  "session": {
    "audio": {
      "input": {
        "format": { "type": "audio/pcm", "rate": 16000 },
        "transcription": { "model": "rhythm" },
        "turn_detection": {
          "type": "server_vad",
          "threshold": 0.5,
          "prefix_padding_ms": 300,
          "silence_duration_ms": 500
        }
      }
    }
  }
}

The server emits session.created on connect and answers each accepted update with session.updated, whose payload echoes the resolved session.

Audio format

Declare your input format and capture rate in audio.input.format. The session decodes linear PCM16 only, and a new session starts at audio/pcm and 24000 Hz, so use the object form {"type": "audio/pcm", "rate": 16000} to state your own rate. The bare string pcm16 is accepted but leaves the rate where it was.

A declared G.711 codec is refused with unsupported_audio_format, in both the spellings the server recognizes: the bare strings g711_ulaw and g711_alaw, and the object types audio/pcmu and audio/pcma. The whole session.update carrying it is discarded, so transcode to PCM16 before streaming.

Mixing the two spellings, as in {"type": "g711_ulaw"}, produces a format object the server does not recognize: it is ignored rather than rejected, and the session keeps the format it already had. Read the session.updated echo to confirm which format took.

Sending audio

Stream base64-encoded PCM16 through input_audio_buffer.append. Audio travels inside JSON events; a binary WebSocket frame closes the session with code 1003.

{ "type": "input_audio_buffer.append", "audio": "<base64 pcm16>" }

Turn detection

Server VAD closes the user turn when input audio goes quiet for the configured duration. Every sub-field is optional.

For manual turns, set turn_detection to null and close each turn with input_audio_buffer.commit. Do not follow it with response.create: a transcription session has no reply to generate. A commit holding less than 100 ms of audio is refused with input_audio_buffer_commit_empty, and the audio it held is discarded rather than kept for a later commit.

Events

Deltas and the completed event both key off item_id, so accumulate text per item rather than per session.

{
  "type": "conversation.item.input_audio_transcription.completed",
  "item_id": "item_cd99c2b70002",
  "content_index": 0,
  "transcript": "Thanks, everyone, for joining.",
  "usage": { "type": "duration", "seconds": 1.696 }
}

A transcription session emits no response.* events, and no event carries word offsets. For word timings, use REST; see timestamps.

Limits and close codes