Test Conversations

The conversations an agent is tested against: written by simulated callers for a new agent, or imported from its real calls, with their hardest moments turned into validation examples, and kept up to date as the prompt changes.

A test conversation is a whole conversation with one agent, kept so that changes to the agent can be tested against it. Each one has a source:

The hardest moments of each conversation become validation examples, each with an answer key (the replies that would be acceptable at that moment, and what the agent is expected to do). Replay runs replay those moments against a changed configuration and judge the new replies against the keys.

A conversation keeps every version of its turns as revisions, so the version a run tested stays readable after the conversation changes. When a prompt change adds, removes or changes a behavior, a refresh brings each conversation up to the new prompt as a new revision.

The agent iteration toolkit#

These endpoints are part of the agent iteration toolkit, which is enabled per organization while it is in beta. Without it they answer 403 with code: "feature_not_enabled" and a next_step; ask support to enable it.

Dev and test splits#

Each conversation is assigned once to the agent's dev split or its test split, half and half by default. Imported calls are assigned by their caller, so one caller's calls never sit in both splits; generated conversations are assigned by their id, and adversarial ones always go to dev.

The dev split is for tuning: try a change, replay the dev examples, read the failures, repeat. The test split is for measuring a finished change once. To keep it that way, API keys and OAuth tokens (and so MCP clients and coding agents) never see a test-split conversation's content: they get its id, split, source, status and counts, with "hidden": true. People signed in to the dashboard see everything.

A test result spends its examples. Test conversations fill cohorts: each new one joins the agent's open cohort, which seals when it holds 20 conversations, and the next one opens another. A test replay run replays one sealed cohort and spends it with its first result, so the next test run, for any prompt, needs another. Importing calls or generating conversations fills new cohorts; with none left, a test run is refused with 409 no_test_cohort.

A production hold-out protects the moments taken from calls, their answer keys and the test conversations. In an organization with the toolkit, a call whose caller is in one of its agents' test splits is held out, on the calls API and MCP alike. An API key, an OAuth token or a legacy token gets it as its call_id, its start time and "held_out": true only, without its transcript, recording, history, grade or report, on the call list and search, detail, transcript, audio, grade, issue reports and exports; a write on it (grading, promoting it to a test scenario, regenerating a turn) is refused with 403 held_out_call. Views across many calls (report summaries, issue clusters, investigations, pipeline reports) leave held-out calls out of their counts and samples and withhold text a model wrote from them. A test scenario promoted from a held-out call is its id and "held_out": true, and so are the simulated calls run from it. Splits are per agent, so a caller held out by one agent can sit in another agent's dev split: a dev conversation or example copied from their call reads like a test-split one, with "held_out": true. People signed in to the dashboard see everything, and an admin can give one API key include_held_out_calls (off by default, set only in the dashboard) for an integration that needs every call; MCP never uses it. Leave held-out calls out when tuning. One caller sits in one split of an agent: examples promoted from a call follow the same caller rule as conversations.

When a call's recording is turned off, or the call is deleted, the conversations imported from it go too, with their revisions, their examples, the examples promoted straight from the call, the examples' pinned caller audio and their replay results, and Core's replay cache for the call. A purge that fails is retried until it finishes, so nothing derived from the call outlives it. Until it has, a result of such an example reads as its verdict and ids only, with content_withheld: "recording_disabled".

An agent runs one conversation job (generate, import or refresh) or one replay run at a time, so a refresh never rewrites the examples a replay is running on. A job start while another job is going is refused with 409 job_active (existing_job_id), and while a replay run is going with 409 replay_run_active (existing_run_id).

dry_run, quote_id and max_cost_cents#

Generating, importing and refreshing conversations spends model credits: personas, simulated caller and agent turns, and reads of each conversation to find its hard moments or its out-of-date turns. All three quote first. Send the request with "dry_run": true and the response is a cost quote, step by step, with a quote_id, and nothing runs. The quote counts every request the run can make (its call counts are the run's upper bound) at a typical size, without prompt-cache discounts.

A real run needs that quote_id and max_cost_cents, a cap in US cents from 1 to 10,000. The quote_id is good for 15 minutes, for the same agent and the same arguments, and for one job: it is used up once, durably, in the transaction that creates the job, and a second run with it is refused with quote_used. A start that is refused or fails before the job exists (409 job_active or replay_run_active, a database error) leaves it usable: retry with it. Once the job exists the quote is spent, so a job that then fails or stops is repeated with a new dry run and its new quote_id. It is refused when it is missing, expired, forged or issued for other arguments, when the models or prices changed, or when the run would now cost more than it quoted (the prompt, the calls or the conversations changed since). A refresh's quote also covers the prompt reviews it runs to read the prompts' behaviors, and binds the target prompt and the conversations it selected: when either changes before the run, it is refused with quote_mismatch.

The cap is hard. Before each request the job sets aside the most that request can cost (its whole input at the uncached price plus its output limit; an agent turn at the token budget the agent's model is held to) and stops instead of sending one that does not fit (status stopped), keeping what it already wrote. Because of that set-aside, a job stops with up to one request's bound per worker unspent: the quote's largest_reservation_cents is the largest, and to let a job finish, set max_cost_cents to about total_cents plus twice largest_reservation_cents. The job is charged what its requests really cost, not what it set aside.

Endpoints#

Conversation object#

{
  "id": 812,
  "agent_id": 123,
  "source": "simulated",
  "split": "dev",
  "status": "active",
  "retired_reason": "",
  "source_call_id": null,
  "persona": {
    "name": "Dana",
    "wants": "A plumber today for a leaking kitchen sink.",
    "knows": ["Lives at 14 Elm Street", "Free after 3 pm"],
    "traits": ["rambling"],
    "manner": "Friendly, gives more detail than asked.",
    "language": "en",
    "hangs_up_when": "A visit is booked or the agent cannot help.",
    "pushes_on": "",
    "tactics": [],
    "style": "normal"
  },
  "conversation_started_at": "2026-10-19T16:40:03Z",
  "current_revision": {
    "id": 990,
    "number": 1,
    "prompt_revision_id": 4182,
    "change_kind": "created",
    "row_count": 14,
    "audio_real_through": null,
    "created_at": "2026-10-19T16:40:31Z"
  },
  "revision_count": 1,
  "metadata": { "end_reason": "agent_ended" },
  "created_at": "2026-10-19T16:40:31Z",
  "updated_at": "2026-10-19T16:40:31Z",
  "rows": [
    { "role": "assistant", "content": "Northside Plumbing, how can I help?" },
    { "role": "user", "content": "Hi, my kitchen sink is leaking under the cabinet." }
  ]
}

A test-split conversation read with an API key has only id, agent_id, split, source, status, row_count, revision_count, cohort_number, hidden and hidden_reason.

List conversations#

curl 'https://api.thunderphone.com/v1/agents/123/conversations?split=dev&status=active' \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY"

Returns a page (count, limit, offset, next_offset, results) of conversations without rows, newest first.

Retrieve a conversation#

curl https://api.thunderphone.com/v1/conversations/812 \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY"

Returns the conversation object with rows.

Conversation revisions#

GET /v1/conversations/{conversation_id}/revisions lists every revision, newest first, with limit and offset. Nothing is deleted: a refresh adds a revision, and the ones before it stay readable here.

{
  "id": 1204,
  "number": 3,
  "parent_id": 1150,
  "prompt_revision_id": 4190,
  "change_kind": "refreshed",
  "change_note": "Refreshed for prompt revision 4190 (from 4182). Removed: Ask for the caller's date of birth before booking. 5 of 16 rows differ from the previous revision. Examples: 1 kept.",
  "behaviors": {
    "added": [],
    "removed": [{ "id": "b7a1c2d3e", "behavior": "Ask for the caller's date of birth before booking." }],
    "changed": [],
    "reworded": ["b4d1e09a2"]
  },
  "restored_from": null,
  "refresh": {
    "from_prompt_revision_id": 4182,
    "to_prompt_revision_id": 4190,
    "out_of_date": [
      { "turn": 3, "behavior": "b7a1c2d3e", "how": "does_removed", "caller_responds": true, "note": "Asks for the date of birth." }
    ],
    "rewrite": { "from_agent_turn": 3, "agent_turns_written": 3, "caller_lines_kept": 2, "caller_lines_written": 1, "caller_turns_dropped": { "4": "Answered the removed date-of-birth question." }, "end_reason": null, "tool_results": [] },
    "examples": { "repointed": [5101], "retired": [], "brought_back": [], "moment_changed": [5101], "answer_keys_rewritten": [5101] },
    "note": ""
  },
  "changed_rows": [6, 7, 8, 9, 10],
  "audio_real_through": 5,
  "created_by": "refresh_conversations",
  "created_at": "2026-10-19T18:02:44Z",
  "row_count": 16,
  "diff": [{ "op": "replace", "parent_rows": [6, 11], "rows": [6, 11], "parent_content": [], "content": [] }]
}

The response also carries the conversation. For a test-split conversation read with an API key, results is empty and hidden is true.

Generate conversations#

For an agent with few or no calls. A model writes invented callers from the agent's prompt and tools, each with a reason to call and a way of talking. Each caller then talks to the agent turn by turn: the agent's replies come from the same text pipeline a call uses, with the agent's real prompt, tools and model, and the agent writes its own opening line. Tools that call your webhooks are not called; a model writes a plausible result from the tool's definition instead. A conversation ends when the agent ends the call or transfers it, the caller hangs up, or it reaches max_caller_turns.

curl -X POST https://api.thunderphone.com/v1/agents/123/conversations/generate \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"count": 5, "dry_run": true}'

quote = requests.post(
    "https://api.thunderphone.com/v1/agents/123/conversations/generate",
    headers={"Authorization": "Bearer sk_live_YOUR_API_KEY"},
    json={"count": 5, "dry_run": True},
).json()
print(quote["estimate_cents"], quote["estimate"]["largest_reservation_cents"])
 
job = requests.post(
    "https://api.thunderphone.com/v1/agents/123/conversations/generate",
    headers={"Authorization": "Bearer sk_live_YOUR_API_KEY"},
    json={"count": 5, "quote_id": quote["quote_id"], "max_cost_cents": 150},
).json()

A dry run returns 200:

{
  "dry_run": true,
  "kind": "generate",
  "params": { "count": 5, "style": "normal", "target": null, "use_draft": true, "direction": "inbound", "max_caller_turns": 16, "max_moments": 3 },
  "estimate_cents": 61.4,
  "estimate": {
    "total_cents": 61.4,
    "total_usd": 0.61,
    "largest_reservation_cents": 9.814,
    "steps": [
      { "step": "personas", "model": "gpt-6-1-sol", "calls": 1, "input_tokens_per_call": 5200, "output_tokens_per_call": 2550, "usd_per_million_input": 2.5, "usd_per_million_output": 10.0, "cents": 3.85 }
    ],
    "notes": [],
    "expected_cents": null
  },
  "quote_id": "eyJvcmciOjQyLC…",
  "quote_expires_in_seconds": 900,
  "next_step": "Show this quote, then run again with the same parameters, without dry_run, with this quote_id and with max_cost_cents (the job never spends more)."
}

A real run returns 202 with the job and the estimate it was started on.

Import calls#

Turns the agent's recent finished calls into conversations. Calls with recording turned off are skipped, as are calls already imported and AI-caller simulations. At most two calls per caller are taken, spread over days and over the agent's prompt revisions, newest first.

Each call's caller turns are checked for hard moments: signals the call already stored (transcription seats that disagreed, a reply cut off, a gap of more than 3 seconds, the caller hanging up mid-flow, a flagged issue), and one model read of the call (the caller repeating or correcting themselves, the agent asking again, missing something, or not following an instruction, which it quotes). The flagged moments are kept, plus one ordinary moment so the set also measures what already works.

curl -X POST https://api.thunderphone.com/v1/agents/123/conversations/import \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"since_days": 30, "max_calls": 40, "dry_run": true}'

The responses match Generate conversations.

Refresh conversations#

After a prompt change that adds, removes or changes a behavior, some conversations no longer match what the agent would do: the agent still asks for a date of birth the new prompt no longer wants, say. A replay run on them tests the new prompt on histories it could not have produced. A refresh brings each active conversation up to the agent's current prompt revision as a new revision. The agent's own review reports when one is due: conversations.refresh_suggested and how many conversations it would update.

For each conversation, a refresh:

1. Compares behaviors. It diffs the rule inventories of the prompt the conversation was written for and the current one. A change in wording only rewrites nothing: the conversation is re-pointed to the new revision with the same rows.
2. Finds the out-of-date turns. One model read lists the agent turns that do something a removed behavior asked for, miss an added one, or follow the old version of a changed one. A conversation with none before its last example is re-pointed, and examples whose own reply changed get new answer keys.
3. Restores before it rewrites. When an earlier revision of the conversation was written for a prompt with these behaviors in their current state (a rule removed and later added back, for example), its rows are copied back, real caller audio included, as a restored revision.
4. Otherwise rewrites the affected span, from the first out-of-date agent turn to the last example's caller turn. Agent turns come from the same text pipeline a call uses, with the new prompt. A caller line that answered a removed question is dropped; the others are kept, with their real audio, when they still fit, and a simulated caller writes the rest. Rows after the last example are kept as they were.
5. Moves the examples to the new revision with rebuilt histories, and rewrites answer keys whose moment changed. An example whose caller turn is no longer in the conversation is retired with the reason, and comes back if a later refresh brings the turn back.

A conversation is retired when rewrites cover more than 60% of its agent turns, or when it is more than 8 prompt revisions behind and no refresh can bring it up to date. Retired conversations stay readable.

curl -X POST https://api.thunderphone.com/v1/agents/123/conversations/refresh \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dry_run": true}'

The responses match Generate conversations. A refresh's quote prices every conversation it reads as if it were rewritten, every model call at its output limit, so a refresh usually spends well under it. A conversation whose prompt revision is not known cannot be refreshed, since nothing tells what changed since it was written: it is quoted at 0, and the job leaves it as it is and reports it as skipped with that reason. That is a conversation imported from a call made before calls recorded their prompt revision, unless the agent has exactly one deployed revision with the call's prompt text, which the import then links it to. A refresh an API key, OAuth token or legacy token starts (MCP included) leaves test-split conversations, and conversations copied from held-out calls, out of its selection, its quote, its progress and its cost; naming one in conversation_ids returns 403 test_conversation, with nothing about the conversation. A person refreshes them from the dashboard, and that job's rows for them show API keys ids and counts only, without their outcome, and outcome_counts counts only the rows the caller can see. An active example whose history, reply or answer key a refresh rewrites goes back to pending_review with the label refreshed_needs_review, so the dashboard's validation runs do not grade against a key nobody reviewed; an example someone archived meanwhile stays archived. A refresh runs as a background job like generate and import; if its worker stops, it is resumed and skips the conversations it already did. It runs on the target prompt and the conversations frozen when it started: when the agent's prompt changes while it runs, the job ends failed, keeping what it refreshed.

Retrieve a job#

GET /v1/conversation-jobs/{job_id} reports a job's progress. Poll it until status is completed, stopped or failed.

Conversation job object#

{
  "id": 77,
  "agent_id": 123,
  "kind": "generate",
  "status": "completed",
  "params": { "count": 5, "style": "normal" },
  "estimate_cents": 61.4,
  "max_cost_cents": 150,
  "cost_cents": 48.9,
  "progress": { "done": 5, "total": 5, "step": "moments" },
  "error": "",
  "result": {
    "conversations": [
      { "id": 812, "split": "dev", "source": "simulated", "status": "active", "persona": { "name": "Dana" }, "end_reason": "agent_ended", "source_call_id": null, "example_ids": [5101, 5102] },
      { "id": 813, "agent_id": 123, "split": "test", "source": "simulated", "status": "active", "row_count": 14, "revision_count": 1, "cohort_number": 2, "hidden": true, "hidden_reason": "This item is in the agent's test split, whose content is hidden from API keys and coding agents. People can review it in the ThunderPhone dashboard." }
    ],
    "conversation_count": 2,
    "example_count": 2,
    "split_counts": { "dev": 1, "test": 1 },
    "errors": [],
    "hidden_error_count": 0
  },
  "created_at": "2026-10-19T16:39:58Z",
  "started_at": "2026-10-19T16:40:00Z",
  "completed_at": "2026-10-19T16:43:12Z"
}

The examples a job creates have status pending_review. Replay runs use them right away; validation runs use them once someone sets them active.

Errors#

MCP#

The MCP server exposes these as list_conversations, get_conversation, get_conversation_revisions, generate_conversations, import_conversations, refresh_conversations and get_conversation_job.

Related#