Calls

Realtime WebSocket

Exchange live call audio, transcripts, and call-control events over one WebSocket.

The Realtime WebSocket API provides bidirectional audio over a single connection. It uses a focused realtime event subset for connecting ThunderPhone agents to your telephony stack.

One WebSocket connection represents one call. Realtime calls appear in call history and are billed like any other call at your product's per-minute rate. A session whose client never joins is not billed.

Create a managed realtime session#

POST /v1/realtime/sessions creates a LiveKit room and returns a scoped participant token. Authenticate with a secret API key and provide either a saved agent_id or an inline config, but not both.

curl -X POST https://api.thunderphone.com/v1/realtime/sessions \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"agent_id": 12}'

Inline configuration accepts only prompt, voice, product, tools, primary_language, additional_languages, background_track, telephony, placeholders, outbound, call_events, and max_hold_seconds (optional integer, 10–900 seconds; null uses the default). prompt is required. Unknown keys are rejected with a validation error rather than silently ignored.

{
  "config": {
    "prompt": "Listen carefully and help complete the call.",
    "voice": "olivia",
    "product": "spark"
  }
}

The operation returns 201 Created with the room credentials:

{
  "call_id": 123456789,
  "room_name": "realtime-123456789",
  "livekit_url": "wss://your-livekit-host.example.com",
  "token": "<scoped-participant-token>"
}

Endpoint#

Connect and authenticate#

Authenticate the WebSocket upgrade with your secret API key:

Authorization: Bearer sk_live_YOUR_API_KEY

Browser clients that cannot set an upgrade header can pass the key in the query string instead:

wss://api.thunderphone.com/v1/realtime?api_key=sk_live_YOUR_API_KEY

After the connection opens, the server sends session.created. Configure the call with session.update; the server confirms the accepted settings with session.updated.

Configure the session#

Send the GA nested session shape before streaming audio. The first session.update starts the call. When instructions is omitted or empty, ThunderPhone uses You are a helpful voice assistant.

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "instructions": "Listen to the caller and help them complete the call.",
    "audio": {
      "input": {
        "format": { "type": "audio/pcm", "rate": 16000 }
      },
      "output": {
        "format": { "type": "audio/pcm", "rate": 16000 },
        "voice": "olivia"
      }
    }
  }
}

Session fields#

session.config accepts only voice, product, tools, primary_language, additional_languages, background_track, telephony, placeholders, outbound, call_events, additional_audio_context, and max_hold_seconds (optional integer, 10–900 seconds; null uses the default). The call prompt comes from session.instructions; there is no separate prompt key over the WebSocket. Unknown keys are rejected with a validation error rather than silently ignored. Set telephony to false for the same wideband or browser-style input tuning selected by session.audio.input.noise_reduction.type set to near_field.

All audio is mono. PCM audio is signed 16-bit little-endian at 16000 or 24000 Hz; G.711 (audio/pcmu μ-law, audio/pcma A-law) is 8000 Hz, matching telephony trunks — no client-side transcoding needed. Input and output formats are configured independently, but neither can change after the call starts. After the call starts, agent configuration is immutable: instructions, voice, tools, product, and other session.config fields cannot change. Bridge-local transport options such as binary_output remain changeable.

For compatibility, flat instructions, voice, input_audio_format, and output_audio_format fields are also accepted on session.update. New integrations should use the nested shape above.

Inline sessions can define custom functions under session.config.tools. The function schema is available to the agent; your WebSocket client remains responsible for executing the function and returning its output.

{
  "type": "session.update",
  "session": {
    "instructions": "Help callers check an order.",
    "config": {
      "tools": [
        {
          "type": "function",
          "function": {
            "name": "lookup_order",
            "description": "Look up an order by id.",
            "parameters": {
              "type": "object",
              "properties": { "order_id": { "type": "string" } },
              "required": ["order_id"]
            }
          }
        }
      ]
    }
  }
}

Custom tools on saved-agent sessions continue to use the routing configured on the saved agent; they are not delivered to the WebSocket client.

Inline sessions are client-steered#

With inline configuration, ThunderPhone adds no automatic conversation behaviors. The agent never speaks first, there are no silence check-ins or timeouts, and by default nothing is spoken while a response is being prepared. If you want progress speech during slow responses — for example while the agent waits on one of your tool results — set placeholders in the session config to a list of short phrases in your own words (e.g. {"placeholders": ["One moment.", "Still checking."]}). They are spoken in the session voice, cycling only when a response takes long enough to need them. If the caller goes quiet, the session remains open until your client prompts the agent or hangs up. Silence-behavior config keys are not accepted and are rejected with a validation error.

To have the agent open the call, configure the session, append a system or user message, and explicitly request one response:

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "instructions": "Help the caller."
  }
}

{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "system",
    "content": [
      { "type": "input_text", "text": "Greet the caller and offer help." }
    ]
  }
}

{ "type": "response.create" }

session.created is sent as soon as the WebSocket connects. session.updated is sent only after the configuration has been accepted and the session fully provisioned, so it is the signal that the call is live; a rejected configuration produces an error event instead, and the session stays usable for a corrected retry. You do not need to wait, though: message items and response.create sent early are buffered and delivered in order once the agent is ready, so the opening sequence above is safe to send immediately after session.update.

Sessions created with agent_id instead retain the behaviors configured on the saved agent, including its greeting order and silence check-ins.

Use a saved agent#

To use an existing agent, connect with its id:

wss://api.thunderphone.com/v1/realtime?agent_id=12

The saved agent starts immediately, so a session.update with instructions is not required. Set its wire audio before minting with the optional input_audio_format, output_audio_format, input_rate, and output_rate query parameters. Formats accept audio/pcmu, audio/pcma, g711_ulaw, g711_alaw, or pcm16; PCM rates accept 16000 or 24000 (G.711 is always 8000 Hz). For example:

wss://api.thunderphone.com/v1/realtime?agent_id=12&input_audio_format=g711_ulaw&output_audio_format=pcm16&output_rate=24000

Agent instructions and voice come from the saved agent.

Client events#

Send client events as JSON text frames unless otherwise noted.

Append audio#

{
  "type": "input_audio_buffer.append",
  "audio": "<base64-pcm16>"
}

Send audio at its natural pace rather than uploading an entire recording at once. You can send each chunk as a raw binary frame instead; no session flag is required for binary input.

Send text guidance#

Message items are appended to the conversation and never trigger a response by themselves. Use user for a completed caller utterance and system, with developer accepted as an alias, for silent guidance. A response begins either from server turn detection on audio or from an explicit response.create. Message items are acknowledged with conversation.item.added followed by conversation.item.done; both echo the client-provided item.id, or a generated id when it is absent. Text is not converted into caller audio.

{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "user",
    "content": [
      { "type": "input_text", "text": "Ask whether they need anything else." }
    ]
  }
}

For silent guidance, set "role": "system" in the same message shape. Assistant-role message injection is unsupported.

Request a response#

Send response.create to trigger exactly one response using the conversation so far. If a response is already in progress, the server emits an error event with code conversation_already_has_active_response and does not start another.

After handling a custom function call, the client must send a function_call_output item with the received call_id. The output can be a JSON string or object. The default deadline is 30 seconds. If the deadline expires, the conversation continues with an error result so the agent can recover. Every function call names one of your declared tools and expects a function_call_output; platform actions arrive as call.* events instead (see below) and take no output.

{
  "type": "conversation.item.create",
  "item": {
    "type": "function_call_output",
    "call_id": "call_abc123",
    "output": "{\"status\":\"ok\"}"
  }
}

Server events#

Server events are JSON text frames. Each JSON event includes a unique event_id.

Caller transcripts#

A caller's transcript improves as the turn proceeds — a later pass can rewrite words an earlier one produced. ThunderPhone therefore does not stream partial caller text by default (opt in with live_transcripts). Each caller turn is announced once, when its transcript has settled:

input_audio_buffer.committed
conversation.item.added
conversation.item.input_audio_transcription.delta
conversation.item.input_audio_transcription.completed
conversation.item.done

Two consequences worth designing around:

• The single delta carries the entire transcript, identical to the completed transcript. Concatenating every delta for an item yields exactly the final text.
• The turn is announced once the agent has acted on it — begun replying, pressed keypad digits, or called a tool — because that is the earliest point the text is settled. Use previous_item_id to reconstruct conversation order rather than arrival order.

Transcript finality#

The text in completed matches the durable per-call record (the same text the dashboard and GET /v1/calls/{call_id}/history show) for every caller turn that finished before the call ended. When the call ends normally — the agent hangs up, or the far end does — the session reconciles against the final record before call.ended, so a turn is never left unannounced.

Two cases fall outside that. A turn still in progress when the call ends has no settled text yet: its words arrive as live deltas (when enabled) and its final text exists only in the durable record. And if the call is torn down abnormally — the media session drops rather than the agent or the far end hanging up — there is no final record to reconcile against, so the last turns are released with the most recent text the session had, which may be superseded in the durable record.

In both cases the durable record is authoritative and reachable with the call_id on session.updated. A client that must not miss a correction should reconcile against GET /v1/calls/{call_id}/history after the call; one that treats the WebSocket as the record of truth will be right on every normally-ended call.

The platform call_id for fetching that record appears on session.updated (session.call_id) and on call.ended.

Live transcripts#

Set session.live_transcripts: true to also receive caller words while the caller is speaking. A separate streaming transcription runs on the caller audio and its output is put on the wire immediately; the settled transcript described above still arrives and is still the text of record. Per caller turn the sequence becomes:

input_audio_buffer.committed
conversation.item.added                                  (status: in_progress)
conversation.item.input_audio_transcription.delta        (repeated, word fragments)
conversation.item.input_audio_transcription.completed    (the settled transcript)
conversation.item.done

Rules for a client:

• Deltas are genuine fragments. Concatenate the deltas of an item to render the live caption. They begin roughly a second behind the caller's voice, before input_audio_buffer.speech_started in some cases, and before the agent has replied.
• completed replaces the concatenated deltas, and it is terminal. It carries the settled transcript, which is generally better than the live text and can differ from it. Show that text and discard the fragments. An item receives exactly one completed, and never a delta after it; conversation.item.done closes the item.
• The item is opened before its place in the conversation is certain. Words are streamed as soon as they are heard, so at a turn boundary (a keypad press, the caller resuming right as the agent replies) the last words of one turn can briefly show on the previous item until that item's completed replaces its text. Anchor each item to its first delta's timestamp; the settled text always lands on the right item.
• Items are always closed. If a call ends before a live item's transcript settles, its completed carries the live text.

Live transcripts are a WebSocket-only addition billed at +1.5¢ per minute on top of the session's tier rate (see Pricing). They are off by default. Turn them on with session.live_transcripts: true in any session.update — the first one for inline sessions, or a later one for saved-agent sessions, which start at connect:

{ "type": "session.update", "session": { "live_transcripts": true } }

Once enabled, the surcharge applies to the whole session; a later session.update with live_transcripts: false stops the stream but does not remove the charge. If the server cannot provide live transcripts at any point, the session continues with settled transcripts only and a non-fatal error event says so.

One completed per item#

Each caller item receives exactly one conversation.item.input_audio_transcription.completed, and never a delta after it. Persist by item_id and treat completed as the final text. In the rare case a better transcript is produced after an item was announced, it is kept in the call's history and dashboard but is not sent again over the WebSocket.

Caller audio#

conversation.item.input_audio_transcription.completed carries an audio object with a pre-signed link to what the caller said for that turn:

{
  "url": "https://...",
  "expires_at": "2026-08-12T19:04:00Z",
  "format": "audio/wav"
}

The turn's duration is already on the same event as usage.seconds; it is not repeated here.

The recording is WAV, 16-bit PCM, mono, 16 kHz, and covers the whole turn. It is the audio as received on the call, not a noise-suppressed version. Issue a plain GET — the URL is already signed, so do not send an Authorization header. Links expire 24 hours after they are issued and are not reissued, so download and store the bytes rather than the URL.

The audio object arrives once, on the item's single completed, so store it on sight.

The audio field is omitted — not null — whenever no recording of the turn exists, including when call recording is disabled for your organization. Treat it as optional on every item. Whether a turn has a recording is decided once, so an item that arrives without audio will not gain one later.

It rides the transcription event rather than the item's content part because the GA schema types a content part's audio as a base64 string, so an object there would fail strict validation in the official SDK. It is an extra top-level key, which GA models accept because they permit unknown fields.

Note that ThunderPhone's call events are a different kind of addition: they are new event types, not extra keys, so a validator that rejects unknown event types will not recognise them. Route on type and ignore what you do not handle.

Conformance notes#

Item ids are opaque and carry no structure — do not parse them or derive ordering from them. Use previous_item_id for conversation order and item_id for identity.

Caller items are announced once their transcript settles, which is after the agent has begun replying, so arrival order is not conversation order (with live_transcripts on they open as soon as words are heard instead). previous_item_id always reflects conversation order: an agent item names the caller turn it answers, even though that turn is announced moments later.

One consequence to build for: previous_item_id can name an item you have not received yet. It arrives within the next few events. Accumulate items and resolve the order afterwards rather than assuming a predecessor is already present when an item arrives.

Usage counters on response.done are always zero. Calls are metered per minute rather than per token, so ThunderPhone reports honest zeros instead of fabricated token counts. The usage on transcription events is real and reports the caller audio duration.

ThunderPhone supports PCM at 16000 Hz as an extension. The OpenAI GA schema pins audio/pcm to 24000 Hz, so strict validators of session.created and session.updated should also allow rate 16000.

The normal audio response sequence is:

response.created
rate_limits.updated
response.output_item.added
conversation.item.added
response.content_part.added
response.output_audio.delta (repeated)
response.output_audio_transcript.delta (repeated)
response.output_audio.done
response.output_audio_transcript.done
response.content_part.done
response.output_item.done
conversation.item.done
response.done

Audio and transcript deltas can be interleaved within the response.

Function calls#

Function calls arrive as function_call output items and always name one of the tools your client declared. Read the function name and call_id from response.output_item.added, accumulate argument deltas, and act after response.function_call_arguments.done. Platform actions are never function calls — they arrive as call events.

Call capabilities#

Every session's agent can end the call and can deliberately stay silent for a turn while the far side is still talking — fundamental phone-call behaviors, always on. Sessions with outbound: true additionally wait out hold queues, and — with call_events enabled — press keypad digits to navigate menus. Transfers also require call_events: the agent decides to transfer, and your client executes it on your telephony stack from the call.transfer instruction. Capabilities that instruct your client are only granted when the event channel exists to carry the instruction; there is nothing else to configure or declare.

A set of function names is reserved for the platform and rejected in your tools with a validation error: end_call, transfer_call, send_keypad_input, no_response, wait_on_hold, search_knowledge_base, play_sound, send_email, speak_uninterruptible.

Call events#

Platform call actions and state changes are delivered as dedicated events in the call.* namespace — a ThunderPhone extension to the realtime event set. They are an explicit opt-in: set call_events: true in the session config to receive them. They are notifications: nothing is sent in reply, and there is no call_id or output. Handle every type below in your event loop — clients ported from other realtime APIs often switch on known event types and would otherwise drop them.

{ "type": "call.keypad", "event_id": "event_1a2b", "digits": "2" }

{ "type": "call.ended", "event_id": "event_3c4d", "reason": "agent_hangup" }

Outbound calls#

Set outbound: true in the session config when the session is an outbound call — the agent dialed out, so the far side may be an automated menu, a hold queue, voicemail, or a screener rather than a person. This enables the engine's outbound call handling: the agent recognizes and navigates automated systems reliably, responds immediately when a human picks up so they are not met with dead air, and applies cost-saving measures that keep the price down while the call is working through menus, waiting on hold, or handling voicemail. It is off by default; sessions without it treat the far side as a human caller. Menu navigation and hold handling need no further configuration — steer them through your instructions:

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "instructions": "Call the pharmacy, navigate the phone menu, and ask whether the prescription is ready.",
    "audio": {
      "output": { "voice": "olivia" }
    },
    "config": {
      "outbound": true,
      "call_events": true,
      "tools": [
        {
          "type": "function",
          "function": {
            "name": "report_status",
            "description": "Report the prescription status back to the app.",
            "parameters": {
              "type": "object",
              "properties": { "status": { "type": "string" } },
              "required": ["status"]
            }
          }
        }
      ]
    }
  }
}

A standalone function call follows this sequence:

response.created
response.output_item.added
response.function_call_arguments.delta
response.function_call_arguments.done
response.output_item.done
response.done

Using the official OpenAI SDK#

Existing OpenAI Realtime integrations work by changing only the WebSocket base URL and the API key. Pass the base ending at /v1 — the SDK appends /realtime itself, mirroring wss://api.openai.com/v1:

from openai import AsyncOpenAI
 
client = AsyncOpenAI(
    api_key="sk_live_YOUR_API_KEY",
    websocket_base_url="wss://api.thunderphone.com/v1",
)
 
async with client.realtime.connect(model="thunderphone-realtime") as connection:
    await connection.session.update(session={
        "type": "realtime",
        "instructions": "Listen to the caller and help them complete the call.",
    })
    async for event in connection:
        ...

The model argument is echoed back but does not change behavior; the agent configuration comes from the session instructions or a saved agent.

Minimal Python client#

Install websockets, set THUNDERPHONE_API_KEY, and provide an uncompressed mono pcm16 WAV file sampled at 16000 Hz:

pip install websockets
export THUNDERPHONE_API_KEY=sk_live_YOUR_API_KEY
python realtime_client.py caller.wav

import asyncio
import base64
import json
import os
import sys
import wave
 
import websockets
 
URL = "wss://api.thunderphone.com/v1/realtime"
RATE = 16_000
CHUNK_MS = 100
 
 
def read_wav(path):
    with wave.open(path, "rb") as wav:
        if (wav.getnchannels(), wav.getsampwidth(), wav.getframerate()) != (1, 2, RATE):
            raise ValueError("WAV must be mono pcm16 at 16000 Hz")
        return wav.readframes(wav.getnframes())
 
 
async def send_audio(ws, pcm):
    chunk_size = RATE * 2 * CHUNK_MS // 1000
    loop = asyncio.get_running_loop()
    deadline = loop.time()
    for offset in range(0, len(pcm), chunk_size):
        chunk = pcm[offset : offset + chunk_size]
        await ws.send(json.dumps({
            "type": "input_audio_buffer.append",
            "audio": base64.b64encode(chunk).decode("ascii"),
        }))
        deadline += CHUNK_MS / 1000
        await asyncio.sleep(max(0, deadline - loop.time()))
 
 
async def receive(ws):
    async for message in ws:
        if isinstance(message, bytes):
            continue  # output audio, when binary output is enabled
        event = json.loads(message)
        event_type = event["type"]
        if event_type == "conversation.item.input_audio_transcription.completed":
            print("caller:", event.get("transcript", ""))
        elif event_type == "response.output_audio_transcript.delta":
            print(event.get("delta", ""), end="", flush=True)
        elif event_type == "response.output_audio_transcript.done":
            print()
        elif event_type == "response.output_item.done":
            item = event.get("item", {})
            if item.get("type") == "function_call":
                arguments = json.loads(item.get("arguments") or "{}")
                print("function:", item["name"], arguments)
                # Run your declared tool here and return its output.
        elif event_type == "error":
            print("error:", event["error"], file=sys.stderr)
 
 
async def main(path):
    headers = {"Authorization": f"Bearer {os.environ['THUNDERPHONE_API_KEY']}"}
    async with websockets.connect(URL, additional_headers=headers) as ws:
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "type": "realtime",
                "instructions": "Listen carefully and help complete the call.",
                "audio": {
                    "input": {"format": {"type": "audio/pcm", "rate": RATE}},
                    "output": {
                        "format": {"type": "audio/pcm", "rate": RATE},
                        "voice": "olivia",
                    },
                },
 
            },
        }))
        while json.loads(await ws.recv())["type"] != "session.updated":
            pass
        await asyncio.gather(send_audio(ws, read_wav(path)), receive(ws))
 
 
asyncio.run(main(sys.argv[1]))

Protocol notes#

• Turn detection uses server-side voice activity detection and is always on. Clients do not need to commit or clear the input audio buffer.
• Transcript events are emitted as per-utterance segments, not token-by-token deltas. Use the completed and done events as final text.
• Unknown or unsupported client event types produce an error event without closing the WebSocket. The client can correct the request and continue.