Pipecat: text to speech

Speak the reply in a Pipecat pipeline with a Sprag voice.

SpragTTSService speaks the reply in a Pipecat pipeline. It holds one WebSocket to wss://api.sprag.ai/v1/audio/speech and sends each sentence as the LLM produces it, so the first sentence plays while the rest of the reply is still being written. Audio comes back as 24 kHz PCM16.

Setup

from pipecat_sprag.tts import SpragTTSService

tts = SpragTTSService(
    settings=SpragTTSService.Settings(model="chorus", voice="marin"),
)

Add it to the pipeline in the speak position, between the LLM and the transport output:

pipeline = Pipeline([
    transport.input(),
    stt,
    user_aggregator,
    llm,
    tts,
    transport.output(),
    assistant_aggregator,
])

Argument	Required	Type	Notes
`settings.model`	No	string	A speech model id from Models. Defaults to `chorus`.
`settings.voice`	No	string	A preset id, or the id of a voice your organization created. Defaults to `wade`.
`api_key`	No	string	Your Sprag API key. Defaults to `SPRAG_API_KEY`.
`url`	No	string	Defaults to `wss://api.sprag.ai/v1/audio/speech`.

How it works

Each sentence is one response on the session. Responses are spoken in the order they were sent, and each response's audio is delivered to the turn it belongs to. SpragTTSService reports time to first byte and the characters spoken as Pipecat metrics.

If the user interrupts, SpragTTSService closes the connection and opens a new one, because a response cannot be stopped once it has started.

model and voice are fixed for the life of a session. If a TTSUpdateSettingsFrame changes either, SpragTTSService opens a new session with the new settings.

Voices

voice takes any Sprag voice id. A voice returned by POST /v1/audio/voices, designed or cloned, is passed the same way as a preset id. See Voice presets, Voice design, and Voice cloning. To clone a speaker in a Pipecat pipeline, create a voice from the clip and pass its id.

Errors

If voice is not a voice of the model, the session is refused with the error code unknown_voice. SpragTTSService pushes an ErrorFrame carrying that code, and the reply is not spoken. More information about the session's errors can be found in the WebSocket docs.

Pipecat's OpenAI service

OpenAITTSService posts each sentence to POST /v1/audio/speech and plays the response. It runs against Sprag with base_url and api_key passed to the constructor.

import os

from pipecat.services.openai.tts import VALID_VOICES, OpenAITTSService

for voice_id in ("marin", "dominic"):
    VALID_VOICES[voice_id] = voice_id

tts = OpenAITTSService(
    api_key=os.environ["SPRAG_API_KEY"],
    base_url="https://api.sprag.ai/v1",
    sample_rate=24000,
    settings=OpenAITTSService.Settings(
        model="chorus",
        voice="marin",
        speed=1.1,
    ),
)

OpenAITTSService validates voice against VALID_VOICES, its list of OpenAI voice names, before sending a request. Register every Sprag voice id the pipeline can reach, including ids you switch to later with a TTSUpdateSettingsFrame, and set voice explicitly. speed is sent as the field of the same name on POST /v1/audio/speech. Audio is 24 kHz PCM, so pass sample_rate=24000 for the transport to resample when the pipeline runs at another rate.