Pipecat: text to speech

OpenAITTSService handles the convert-to-speech step of a Pipecat pipeline. Pointed at Sprag it runs on Chorus, once you register the Sprag voice ids with the service.

Setup

import os

from pipecat.services.openai.tts import VALID_VOICES, OpenAITTSService

for voice_id in ("serena", "aiden"):
    VALID_VOICES[voice_id] = voice_id

tts = OpenAITTSService(
    api_key=os.environ["SPRAG_API_KEY"],
    base_url="https://api.sprag.ai/v1",
    settings=OpenAITTSService.Settings(
        model="chorus",
        voice="serena",
    ),
)

VALID_VOICES is the list of OpenAI voice names the service checks voice against before it makes any HTTP call. An unregistered id yields an ErrorFrame and no audio, so register every Sprag id the pipeline can reach, including ones you switch to later with a TTSUpdateSettingsFrame. Leaving voice unset sends the service default, alloy, which Sprag rejects with a 400.

Add it to a pipeline in the convert-to-speech position, between the LLM and the transport output.

pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
])

Model and voice

chorus covers the preset and cloned speech models. It works as a model id but does not appear in GET /v1/models, which lists chorus-voices, chorus-design, and chorus-clone separately. See voice presets for the preset ids.

Style and pacing

instructions and speed map onto the fields of the same name on the Sprag speech endpoint, so they behave as they do in a direct API call.

settings=OpenAITTSService.Settings(
    model="chorus",
    voice="aiden",
    instructions="Warm and friendly, moderate pacing",
    speed=1.1,
)

Change settings mid-conversation with a TTSUpdateSettingsFrame, a Pipecat mechanism rather than a Sprag one. See the Pipecat TTS docs.

Sample rate

Chorus returns 24 kHz PCM, and the service tags the audio frames with the pipeline's output rate rather than measuring it. If the pipeline runs at any other rate, pass sample_rate=24000 to the service so the transport resamples from the rate Sprag actually sent.

Designed and cloned voices

Neither is reachable through this service.

A designed voice needs model set to chorus-design and no voice id, and OpenAITTSService requires a voice. A cloned voice is inline: every request carries the reference audio and its transcript as ref_audio and ref_text, and Settings has no field for either. Sprag stores no voices, so you cannot clone once and pass an id here afterwards.

Call the speech endpoint directly for both. See voice design and voice cloning.

Language

Chorus speaks ten languages, listed in supported languages. That set is narrower than what Rhythm transcribes, so a pipeline that accepts speech in a language Sprag cannot generate will transcribe the user correctly and have no voice to answer in. Check both ends before shipping a multilingual bot.

OpenAITTSService has a language setting but does not send it. To select a language explicitly, call the speech endpoint directly.