LiveKit: speech to text

Transcribe user audio in a LiveKit Agents session with Sprag models.

Overview

Sprag provides an OpenAI compatible speech-to-text API allowing you to use LiveKit's OpenAI plugin out of the box. For STT on LiveKit that means openai.STT is usable by simply setting Sprag's base URL and invoking one of our models.

Basic usage

Use openai.STT in an AgentSession, or as a standalone transcription service:

import os

from livekit.agents import AgentSession
from livekit.plugins import openai

session = AgentSession(
    stt=openai.STT(
        model="rhythm",
        use_realtime=True,
        base_url="https://api.sprag.ai/v1",
        api_key=os.environ["SPRAG_API_KEY"],
    ),
    # ... llm, tts, etc.
)

Configuration

Constructor arguments

Argument	Type	Default	Description
`model`	`str`	--	Required parameter when used with Sprag
`use_realtime`	`bool`	`False`	Streams over a realtime WebSocket.
`base_url`	`str`	--	`https://api.sprag.ai/v1`.
`api_key`	`str`	`OPENAI_API_KEY` env var	Your Sprag API key.
`turn_detection`	`dict`	server VAD	Endpointing on the realtime transport.

See speech-to-text models for a list of available models and supported languages for the languages each transcribes.

Transports

In most cases Sprag hosted models are available as REST and realtime connections over a websocket. With use_realtime set to True you will receive a persistent websocket for the duration of the request. With False, each turn is committed as a separate POST.

VAD

For realtime connections we offer an optional server configured VAD for automatic detection of turn boundaries, enabled through turn_detection. It applies only with use_realtime=True; LiveKit ignores it on the REST transport.

import os

from livekit.agents import AgentSession
from livekit.plugins import openai, silero

session = AgentSession(
    stt=openai.STT(
        model="rhythm",
        use_realtime=True,
        base_url="https://api.sprag.ai/v1",
        api_key=os.environ["SPRAG_API_KEY"],
        turn_detection={"silence_duration_ms": 200},
    ),
)

You are still free to use VAD configured through LiveKit as well.

turn_detection takes the same fields as the realtime API, and we fill in any you leave out.

Events

A realtime session reports the edges of each turn alongside its transcript.

Event	Meaning
`START_OF_SPEECH`	The turn started.
`INTERIM_TRANSCRIPT`	A fragment of the turn's transcript.
`FINAL_TRANSCRIPT`	The settled transcript for that turn.
`END_OF_SPEECH`	The turn ended.
`RECOGNITION_USAGE`	Measured duration for that turn.

Fragments concatenate into the final transcript, and a turn's fragments arrive together once it closes. Over REST you receive one FINAL_TRANSCRIPT per utterance.

Speaker labels and word timings

Over REST we return word and segment times under response_format=verbose_json, and speaker_labels=true attributes each word to a speaker.

You can find more information about timestamps and diarization in the docs.