Sprag extensions

Attach native audio and visual video evidence to Decisions requests and read media token usage.

Sprag extensions add inline audio, visual video evidence, and media token counts to POST /v1/decisions.

Send video

Video evidence requires a model whose catalog entry lists input_video among its supported parameters.

import base64
import os
from pathlib import Path

import requests

response = requests.post(
    "https://api.sprag.ai/v1/decisions",
    headers={"Authorization": f"Bearer {os.environ['SPRAG_API_KEY']}"},
    json={
        "model": os.environ["SPRAG_VIDEO_MODEL"],
        "input": "Delivery camera recording.",
        "input_video": {
            "data": base64.b64encode(Path("delivery.mp4").read_bytes()).decode("ascii"),
            "format": "mp4",
        },
        "questions": [
            {
                "type": "predicate",
                "name": "delivered",
                "instructions": "Does the video show a package being left at the door?",
            }
        ],
    },
)
response.raise_for_status()
print(response.json())

Illustrative response for SPRAG_VIDEO_MODEL=example-video-model:

{
  "model": "example-video-model",
  "answers": [
    { "type": "predicate", "name": "delivered", "probability": 0.9 }
  ],
  "usage": {
    "input_tokens": 256,
    "input_tokens_details": {
      "cached_tokens": 0,
      "cache_write_tokens": 0,
      "multimodal_tokens": { "video": 128 }
    },
    "output_tokens": 0,
    "output_tokens_details": { "reasoning_tokens": 0 },
    "total_tokens": 256
  }
}

Set SPRAG_VIDEO_MODEL to the id of a model that accepts video evidence. input_video attaches one video file alongside the required input and questions fields.

Field	Required	Type	Notes
`input_video.data`	Yes	string	Nonempty base64-encoded video file bytes.
`input_video.format`	Yes	string	A nonempty, free-form container or format hint, such as `mp4` or `webm`.

input_video supplies visual frames. To include the soundtrack, attach its audio separately as input_audio in the same request.

Send audio

{
  "model": "spev",
  "input": "Call recording attached.",
  "input_audio": { "data": "<base64 audio file bytes>", "format": "wav" },
  "questions": [
    { "type": "predicate", "instructions": "Does the caller ask to cancel?" }
  ]
}

input_audio attaches one audio file alongside input. Replace the example's data with the file's base64-encoded bytes.

Field	Required	Type	Notes
`input_audio.data`	Yes	string	Nonempty base64-encoded audio file bytes.
`input_audio.format`	Yes	string	A nonempty, free-form container or format hint, such as `wav` or `mp3`.

The model catalog lists input_audio, input_image, and input_video for models that accept each kind of evidence. Image evidence uses input_image parts inside user messages. For image fields and an example, see Send an image.

Read answers

Media requests return the same structured answers as text requests. answers follows question order and carries each question's name, or null.

Question	Answer fields	Meaning
Predicate	`probability`	Probability that the condition holds, from 0 to 1.
Choice	`choice`, `probabilities`, `confidence`	Selected string or boolean value and the distribution over your choices.
Score	`score`, `probabilities`, `confidence`	Expected zero-based level index and the distribution over your rubric.

If the model declines a question, its answer has type: "refusal". For the question and answer fields, see Ask a question.

Read token usage

usage.input_tokens_details.multimodal_tokens reports native input token counts for audio, image, and video, when supplied in the response. Absent modalities have no key. Text-only evidence has no media counts.

usage.input_tokens includes text and media tokens. usage.total_tokens is input_tokens plus output_tokens. For the full response fields, see the Decisions API.