Pipecat: speech to text

Pipecat 1.8.1 ships two OpenAI transcription services. OpenAISTTService sends one HTTP request per user turn. OpenAIRealtimeSTTService holds a WebSocket open across turns and emits the transcript in fragments once each turn closes. Both run on Rhythm or Symphony.

Setup

import os

from pipecat.services.openai.stt import OpenAISTTService

stt = OpenAISTTService(
    api_key=os.environ["SPRAG_API_KEY"],
    base_url="https://api.sprag.ai/v1",
    settings=OpenAISTTService.Settings(
        model="rhythm",
    ),
)

Add it to a pipeline in the transcribe position, between the transport input and the LLM.

pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
])

Model

Rhythm transcribes thirty languages plus twenty-two Chinese dialects. See supported languages for the full list.

Turn timing

OpenAISTTService is segmented: it waits for VAD to detect silence, then sends the completed audio segment over HTTP as one request. You get final transcripts only, no interim results, and a delay proportional to the length of the user's turn.

Use it in agent pipelines, where the LLM needs a complete utterance before it can respond. Two cases where that delay matters:

• Captions. Text appears a turn at a time either way, but OpenAIRealtimeSTTService starts painting it sooner, because the audio has already reached Sprag by the time the speaker stops.
• Long turns. A speaker who talks for thirty seconds without pausing will wait for the whole segment to upload and process before anything comes back.

Streaming transcripts

OpenAIRealtimeSTTService opens a transcription session on Sprag's realtime WebSocket and pushes an InterimTranscriptionFrame per delta, then a TranscriptionFrame when the turn completes. Rhythm decodes a turn once the turn closes, so those deltas arrive in a burst after the user stops rather than while they are speaking.

import os

from pipecat.services.openai.stt import OpenAIRealtimeSTTService

stt = OpenAIRealtimeSTTService(
    api_key=os.environ["SPRAG_API_KEY"],
    base_url="wss://api.sprag.ai/v1/realtime",
    settings=OpenAIRealtimeSTTService.Settings(
        model="rhythm",
    ),
)

Pass the base URL without a query string: the service appends ?intent=transcription itself.

turn_detection defaults to False, which turns Sprag's server-side VAD off and commits the audio buffer when a local VAD processor in the pipeline reports the end of a turn. Pass turn_detection=None to let Sprag find the boundaries instead, and drop the local VAD processor if you do.

Concurrent realtime sessions are capped per user; see session limits.

Speaker labels and word timings

The Pipecat services surface transcript text, so neither speaker labels nor word timings reach a pipeline. Both ride on the same transcription request: POST /v1/audio/transcriptions returns word and segment times under response_format=verbose_json, and speaker_labels=true attributes each word to a speaker. A pipeline that needs them holds the utterance and posts it itself, which transcribes the same audio a second time. See timestamps and diarization.