TEN Framework

TEN Framework is an open-source framework for realtime, multimodal conversational agents built as graphs of extensions. ThunderPhone plugs in as a speech-to-speech model extension, thunderphone_mllm_python, alongside the framework's other realtime model extensions. The graph's transport carries audio (Agora RTC in the standard examples); ThunderPhone handles speech recognition, the language model, the voice, turn-taking, 47 languages and tools.

The extension has been submitted to the TEN Framework repository and is under review. Until it is merged, it can be used from the contribution branch by adding the extension directory to a TEN app. This page describes what the extension does today.

What the extension does

The extension implements TEN's multimodal LLM interface, so it emits the same events the framework's main control expects: session ready, caller transcript, agent transcript, interruption and function call. It is modeled on the framework's OpenAI realtime extension because ThunderPhone's realtime WebSocket speaks the same protocol, and it adds the ThunderPhone parts: the endpoint and its query parameters, sk_live_ keys, saved agents, one complete session start for inline sessions, and the platform's call events.

Saved agent. Set agent_id on the node. The agent's prompt, voice, engine, languages, tools and greeting live on ThunderPhone; the graph only moves audio. Tools registered in the graph are not sent.
Inline session. Leave agent_id empty and set prompt, with optional product, voice and language. Tools that other extensions register through the graph are sent to ThunderPhone when the call starts, and their results flow back the way the framework already handles function calls.

Every WebSocket session is one ThunderPhone call, visible in call history and billed per minute. When the ThunderPhone agent hangs up, the extension does not reconnect, because a reconnect would begin a new call.

Typical call flows

The realtime voice-assistant example

TEN's realtime voice-assistant example is a graph with an RTC node, a main-control node, a message collector and a speech-to-speech node. Swapping the speech-to-speech node for thunderphone_mllm_python keeps the rest of the graph as is. A saved agent then answers as a virtual receptionist would on ThunderPhone's own numbers; an inline session runs the prompt from the graph's property file.

Graph tools

A tool extension such as a lookup service registers itself at graph start. In an inline session the tool definition is sent to ThunderPhone with the session start, the agent calls it mid-conversation through standard function calling, and the main-control node returns the result.

Call events

call.ended, call.transfer, keypad and ignored-speech events are parsed and logged, and the ThunderPhone call id is attached to transcript metadata so an app can fetch the recording and transcript afterwards through the REST API.

Setup outline

Create a secret API key in the ThunderPhone dashboard and set THUNDERPHONE_API_KEY (and THUNDERPHONE_AGENT_ID for a saved agent).
Add the thunderphone_mllm_python extension to the app's manifest and reference it from the graph's speech-to-speech node.
For an inline session, set prompt and register any graph tools before audio starts; the extension waits briefly after the session is created for late registrations.
Run the graph and test barge-in, a tool round trip and an agent hang-up.

Limits worth knowing

Turn detection is server-side and always on.
Instructions, tools and voice are fixed once the call starts; tools registered later are refused with a log line.
Injected assistant messages are not accepted. Put the greeting in the prompt or on the saved agent.
Audio is 16-bit mono PCM at 24 kHz (16 kHz is also supported).