---
title: "Test Conversations"
description: "The conversations an agent is tested against: written by simulated callers for a new agent, or imported from its real calls, with their hardest moments turned into validation examples, and kept up to date as the prompt changes."
---

A test conversation is a whole conversation with one agent, kept so that
changes to the agent can be tested against it. Each one has a source:

| Source | Where it comes from |
|--------|---------------------|
| `production` | One of the agent's real calls, [imported](#import-calls) |
| `test_call` | One of the agent's builder test calls, imported with `include_test_calls` |
| `simulated` | [Generated](#generate-conversations): an invented caller talks to the agent |
| `adversarial` | Generated with `style: "adversarial"`: a caller who pushes on the prompt's weak spots |
| `manual` | Written by hand |

The hardest moments of each conversation become
[validation examples](/api-reference/validation-sets), each with an answer
key (the replies that would be acceptable at that moment, and what the agent
is expected to do). [Replay runs](/api-reference/replay-runs) replay those
moments against a changed configuration and judge the new replies against
the keys.

A conversation keeps every version of its turns as revisions, so the
version a run tested stays readable after the conversation changes. When a
prompt change adds, removes or changes a behavior, a
[refresh](#refresh-conversations) brings each conversation up to the new
prompt as a new revision.

## The agent iteration toolkit

These endpoints are part of the agent iteration toolkit, which is enabled
per organization while it is in beta. Without it they answer `403` with
`code: "feature_not_enabled"` and a `next_step`; ask
[support](mailto:support@thunderphone.com) to enable it.

### Dev and test splits

Each conversation is assigned once to the agent's **dev** split or its
**test** split, half and half by default. Imported calls are assigned by
their caller, so one caller's calls never sit in both splits; generated
conversations are assigned by their id, and adversarial ones always go to
dev.

The dev split is for tuning: try a change, replay the dev examples, read
the failures, repeat. The test split is for measuring a finished change
once. To keep it that way, API keys and OAuth tokens (and so MCP clients
and coding agents) never see a test-split conversation's content: they get
its id, split, source, status and counts, with `"hidden": true`. People
signed in to the dashboard see everything.

A test result spends its examples. Test conversations fill **cohorts**: each
new one joins the agent's open cohort, which seals when it holds 20
conversations, and the next one opens another. A [test replay
run](/api-reference/replay-runs#dev-and-test-splits) replays one sealed
cohort and spends it with its first result, so the next test run, for any
prompt, needs another. Importing calls or generating conversations fills new
cohorts; with none left, a test run is refused with `409 no_test_cohort`.

A production hold-out protects the moments taken from calls, their answer
keys and the test conversations. In an organization with the toolkit, a
call whose caller is in one of its agents' test splits is held out, on the
[calls API](/api-reference/calls) and MCP alike. An API key, an OAuth token
or a legacy token gets it as its `call_id`, its start time and
`"held_out": true` only, without its transcript, recording, history, grade
or report, on the call list and search, detail, transcript, audio, grade,
issue reports and exports; a write on it (grading, promoting it to a test
scenario, regenerating a turn) is refused with `403 held_out_call`. Views
across many calls (report summaries, issue clusters, investigations,
pipeline reports) leave held-out calls out of their counts and samples and
withhold text a model wrote from them. A test scenario promoted from a
held-out call is its id and `"held_out": true`, and so are the simulated
calls run from it. Splits are per agent, so a caller held out by one agent
can sit in another agent's dev split: a dev conversation or example copied
from their call reads like a test-split one, with `"held_out": true`. People
signed in to the dashboard see everything, and an admin can give one API key
`include_held_out_calls` (off by default, set only in the dashboard) for an
integration that needs every call; MCP never uses it. Leave held-out calls
out when tuning. One caller sits in one split of an agent: examples promoted
from a call follow the same caller rule as conversations.

When a call's recording is turned off, or the call is deleted, the
conversations imported from it go too, with their revisions, their
examples, the examples promoted straight from the call, the examples'
pinned caller audio and their replay results, and Core's replay cache for
the call. A purge that fails is retried until it finishes, so nothing
derived from the call outlives it. Until it has, a result of such an
example reads as its verdict and ids only, with `content_withheld:
"recording_disabled"`.

An agent runs one conversation job (generate, import or refresh) or one
[replay run](/api-reference/replay-runs#one-run-or-job-at-a-time) at a
time, so a refresh never rewrites the examples a replay is running on. A
job start while another job is going is refused with `409 job_active`
(`existing_job_id`), and while a replay run is going with `409
replay_run_active` (`existing_run_id`).

### `dry_run`, `quote_id` and `max_cost_cents`

Generating, importing and refreshing conversations spends model credits:
personas, simulated caller and agent turns, and reads of each conversation
to find its hard moments or its out-of-date turns. All three quote first.
Send the request with `"dry_run": true` and the response is a cost quote,
step by step, with a `quote_id`, and nothing runs. The quote counts every
request the run can make (its call counts are the run's upper bound) at a
typical size, without prompt-cache discounts.

A real run needs that `quote_id` and `max_cost_cents`, a cap in US cents
from 1 to 10,000. The `quote_id` is good for 15 minutes, for the same
agent and the same arguments, and for one job: it is used up once,
durably, in the transaction that creates the job, and a second run with it
is refused with `quote_used`. A start that is refused or fails before the
job exists (`409 job_active` or `replay_run_active`, a database error)
leaves it usable: retry with it. Once the job exists the quote is spent,
so a job that then fails or stops is repeated with a new dry run and its
new `quote_id`. It is refused when it is
missing, expired, forged or issued for other arguments, when the models or
prices changed, or when the run would now cost more than it quoted (the
prompt, the calls or the conversations changed since). A refresh's quote
also covers the prompt reviews it runs to read the prompts' behaviors, and
binds the target prompt and the conversations it selected: when either
changes before the run, it is refused with `quote_mismatch`.

The cap is hard. Before each request the job sets aside the most that
request can cost (its whole input at the uncached price plus its output
limit; an agent turn at the token budget the agent's model is held to) and
stops instead of sending one that does not fit (status `stopped`), keeping
what it already wrote. Because of that set-aside, a job stops with up to
one request's bound per worker unspent: the quote's
`largest_reservation_cents` is the largest, and to let a job finish, set
`max_cost_cents` to about `total_cents` plus twice
`largest_reservation_cents`. The job is charged what its requests really
cost, not what it set aside.

## Endpoints

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/v1/agents/{agent_id}/conversations` | List an agent's test conversations |
| `GET` | `/v1/conversations/{conversation_id}` | Retrieve one conversation with its current turns |
| `GET` | `/v1/conversations/{conversation_id}/revisions` | Every revision of a conversation: how and why it changed, with row diffs |
| `POST` | `/v1/agents/{agent_id}/conversations/generate` | Generate conversations with simulated callers |
| `POST` | `/v1/agents/{agent_id}/conversations/import` | Import the agent's recent calls |
| `POST` | `/v1/agents/{agent_id}/conversations/refresh` | Bring conversations up to the current prompt |
| `GET` | `/v1/conversation-jobs/{job_id}` | Follow a generate, import or refresh job |

## Conversation object

```json
{
  "id": 812,
  "agent_id": 123,
  "source": "simulated",
  "split": "dev",
  "status": "active",
  "retired_reason": "",
  "source_call_id": null,
  "persona": {
    "name": "Dana",
    "wants": "A plumber today for a leaking kitchen sink.",
    "knows": ["Lives at 14 Elm Street", "Free after 3 pm"],
    "traits": ["rambling"],
    "manner": "Friendly, gives more detail than asked.",
    "language": "en",
    "hangs_up_when": "A visit is booked or the agent cannot help.",
    "pushes_on": "",
    "tactics": [],
    "style": "normal"
  },
  "conversation_started_at": "2026-10-19T16:40:03Z",
  "current_revision": {
    "id": 990,
    "number": 1,
    "prompt_revision_id": 4182,
    "change_kind": "created",
    "row_count": 14,
    "audio_real_through": null,
    "created_at": "2026-10-19T16:40:31Z"
  },
  "revision_count": 1,
  "metadata": { "end_reason": "agent_ended" },
  "created_at": "2026-10-19T16:40:31Z",
  "updated_at": "2026-10-19T16:40:31Z",
  "rows": [
    { "role": "assistant", "content": "Northside Plumbing, how can I help?" },
    { "role": "user", "content": "Hi, my kitchen sink is leaking under the cabinet." }
  ]
}
```

| Field | Type | Description |
|-------|------|-------------|
| `source` | string | See the table above |
| `split` | string | `dev` or `test` |
| `status` | string | `active`, or `retired`: kept and readable, but left out of runs. Once a split has 50 real conversations, the next import retires its simulated ones (not adversarial ones); `retired_reason` says why |
| `source_call_id` | integer \| null | The [call](/api-reference/calls) an imported conversation came from |
| `persona` | object \| null | The simulated caller: who they are, what they want and know, how they talk, when they hang up, and for adversarial callers the passage they push on (`pushes_on`) and how (`tactics`) |
| `current_revision` | object | The version of the turns runs use now: its `number`, the prompt revision it matches, how it was made (`change_kind`: `created`, `refreshed`, `restored` or `manual`), its row count, and the last row whose caller audio is still the real recording (`audio_real_through`, `null` for text-only conversations) |
| `revision_count` | integer | How many revisions the conversation has |
| `metadata` | object | Details such as why a generated conversation ended (`end_reason`: `agent_ended`, `transferred`, `caller_hung_up`, `caller_gave_up`, `turn_cap` or `cost_cap`) |
| `rows` | object[] | The current revision's turns in order, as role and content plus any tool calls and results. Only on the single-conversation endpoint |

A test-split conversation read with an API key has only `id`, `agent_id`,
`split`, `source`, `status`, `row_count`, `revision_count`,
`cohort_number`, `hidden` and `hidden_reason`.

## List conversations

<CodeGroup>
```bash cURL
curl 'https://api.thunderphone.com/v1/agents/123/conversations?split=dev&status=active' \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY"
```
</CodeGroup>

| Param | Type | Description |
|-------|------|-------------|
| `split` | string | `dev` or `test` |
| `source` | string | One of the sources above |
| `status` | string | `active` or `retired` |
| `limit` | integer | 1 to 100, default 20 |
| `offset` | integer | Default 0 |

Returns a page (`count`, `limit`, `offset`, `next_offset`, `results`) of
conversations without `rows`, newest first.

## Retrieve a conversation

```bash
curl https://api.thunderphone.com/v1/conversations/812 \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY"
```

Returns the [conversation object](#conversation-object) with `rows`.

## Conversation revisions

`GET /v1/conversations/{conversation_id}/revisions` lists every revision,
newest first, with `limit` and `offset`. Nothing is deleted: a refresh adds
a revision, and the ones before it stay readable here.

```json
{
  "id": 1204,
  "number": 3,
  "parent_id": 1150,
  "prompt_revision_id": 4190,
  "change_kind": "refreshed",
  "change_note": "Refreshed for prompt revision 4190 (from 4182). Removed: Ask for the caller's date of birth before booking. 5 of 16 rows differ from the previous revision. Examples: 1 kept.",
  "behaviors": {
    "added": [],
    "removed": [{ "id": "b7a1c2d3e", "behavior": "Ask for the caller's date of birth before booking." }],
    "changed": [],
    "reworded": ["b4d1e09a2"]
  },
  "restored_from": null,
  "refresh": {
    "from_prompt_revision_id": 4182,
    "to_prompt_revision_id": 4190,
    "out_of_date": [
      { "turn": 3, "behavior": "b7a1c2d3e", "how": "does_removed", "caller_responds": true, "note": "Asks for the date of birth." }
    ],
    "rewrite": { "from_agent_turn": 3, "agent_turns_written": 3, "caller_lines_kept": 2, "caller_lines_written": 1, "caller_turns_dropped": { "4": "Answered the removed date-of-birth question." }, "end_reason": null, "tool_results": [] },
    "examples": { "repointed": [5101], "retired": [], "brought_back": [], "moment_changed": [5101], "answer_keys_rewritten": [5101] },
    "note": ""
  },
  "changed_rows": [6, 7, 8, 9, 10],
  "audio_real_through": 5,
  "created_by": "refresh_conversations",
  "created_at": "2026-10-19T18:02:44Z",
  "row_count": 16,
  "diff": [{ "op": "replace", "parent_rows": [6, 11], "rows": [6, 11], "parent_content": [], "content": [] }]
}
```

| Field | Description |
|-------|-------------|
| `change_kind` | `created`, `refreshed` (brought up to a new prompt revision), `restored` (rows copied back from an earlier revision) or `manual` |
| `behaviors` | Between the prompt revision the parent matched and this one's: behaviors `added`, `removed` and `changed` in meaning, and the ids `reworded` (wording only, which rewrites nothing) |
| `restored_from` | For a restore, the earlier revision the rows came from (`revision_id`, `number`, `prompt_revision_id`) |
| `refresh` | For a revision a refresh wrote: the prompt revisions it went from and to, the agent turns that were `out_of_date` (each with the behavior, `how`: `does_removed`, `misses_added` or `does_old_version`, and whether the caller's next line answered it), the `rewrite` when turns were rewritten, and the `examples` moved, retired, brought back or given new answer keys. `null` for other revisions |
| `changed_rows` | Indices of rows that differ from the parent |
| `diff` | The changed ranges of rows, each with `op` (`replace`, `insert` or `delete`), the parent's and the new row ranges, and the rows themselves |

The response also carries the `conversation`. For a test-split
conversation read with an API key, `results` is empty and `hidden` is
`true`.

## Generate conversations

For an agent with few or no calls. A model writes invented callers from the
agent's prompt and tools, each with a reason to call and a way of talking.
Each caller then talks to the agent turn by turn: the agent's replies come
from the same text pipeline a call uses, with the agent's real prompt,
tools and model, and the agent writes its own opening line. Tools that call
your webhooks are not called; a model writes a plausible result from the
tool's definition instead. A conversation ends when the agent ends the call
or transfers it, the caller hangs up, or it reaches `max_caller_turns`.

<CodeGroup>
```bash cURL
curl -X POST https://api.thunderphone.com/v1/agents/123/conversations/generate \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"count": 5, "dry_run": true}'
```

```python Python
quote = requests.post(
    "https://api.thunderphone.com/v1/agents/123/conversations/generate",
    headers={"Authorization": "Bearer sk_live_YOUR_API_KEY"},
    json={"count": 5, "dry_run": True},
).json()
print(quote["estimate_cents"], quote["estimate"]["largest_reservation_cents"])

job = requests.post(
    "https://api.thunderphone.com/v1/agents/123/conversations/generate",
    headers={"Authorization": "Bearer sk_live_YOUR_API_KEY"},
    json={"count": 5, "quote_id": quote["quote_id"], "max_cost_cents": 150},
).json()
```
</CodeGroup>

| Field | Type | Description |
|-------|------|-------------|
| `count` | integer | Conversations to write, 1 to 20. Default 5 |
| `style` | string | `normal` (default) or `adversarial`: callers who interrupt, refuse, answer a different question, give details in an odd format, ask for a person or try to steer the agent off its task, each aimed at a passage of the prompt |
| `target` | string | `changed_behaviors`: aim the callers at the passages that changed in the agent's last prompt edit |
| `use_draft` | boolean | Talk to the draft (default `true`) or the deployed configuration |
| `direction` | string | `inbound` (default: the caller rings the agent) or `outbound` |
| `max_caller_turns` | integer | 1 to 16, default 16 |
| `max_moments` | integer | Hard moments per conversation to turn into examples, 0 to 5. Default 3 |
| `dry_run` | boolean | Return the quote only |
| `quote_id` | string | The dry run's `quote_id`; required unless `dry_run` |
| `max_cost_cents` | integer | Hard spend cap, 1 to 10,000; required unless `dry_run` |

A dry run returns `200`:

```json
{
  "dry_run": true,
  "kind": "generate",
  "params": { "count": 5, "style": "normal", "target": null, "use_draft": true, "direction": "inbound", "max_caller_turns": 16, "max_moments": 3 },
  "estimate_cents": 61.4,
  "estimate": {
    "total_cents": 61.4,
    "total_usd": 0.61,
    "largest_reservation_cents": 9.814,
    "steps": [
      { "step": "personas", "model": "gpt-6-1-sol", "calls": 1, "input_tokens_per_call": 5200, "output_tokens_per_call": 2550, "usd_per_million_input": 2.5, "usd_per_million_output": 10.0, "cents": 3.85 }
    ],
    "notes": [],
    "expected_cents": null
  },
  "quote_id": "eyJvcmciOjQyLC…",
  "quote_expires_in_seconds": 900,
  "next_step": "Show this quote, then run again with the same parameters, without dry_run, with this quote_id and with max_cost_cents (the job never spends more)."
}
```

A real run returns `202` with the [job](#conversation-job-object) and the
estimate it was started on.

## Import calls

Turns the agent's recent finished calls into conversations. Calls with
recording turned off are skipped, as are calls already imported and
AI-caller simulations. At most two calls per caller are taken, spread over
days and over the agent's prompt revisions, newest first.

Each call's caller turns are checked for hard moments: signals the call
already stored (transcription seats that disagreed, a reply cut off, a gap
of more than 3 seconds, the caller hanging up mid-flow, a flagged issue),
and one model read of the call (the caller repeating or correcting
themselves, the agent asking again, missing something, or not following an
instruction, which it quotes). The flagged moments are kept, plus one
ordinary moment so the set also measures what already works.

<CodeGroup>
```bash cURL
curl -X POST https://api.thunderphone.com/v1/agents/123/conversations/import \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"since_days": 30, "max_calls": 40, "dry_run": true}'
```
</CodeGroup>

| Field | Type | Description |
|-------|------|-------------|
| `since_days` | integer | How far back to look, 1 to 365. Default 30 |
| `max_calls` | integer | Calls to import at most, 1 to 200. Default 40 |
| `include_test_calls` | boolean | Also import the agent's builder test calls. Default `false` |
| `max_moments` | integer | Hard moments per call to turn into examples, 0 to 5. Default 3 |
| `dry_run` | boolean | Return the quote only. Its `estimate.calls_selected` says how many calls would be imported |
| `quote_id` | string | The dry run's `quote_id`; required unless `dry_run` |
| `max_cost_cents` | integer | Hard spend cap, 1 to 10,000; required unless `dry_run` |

The responses match [Generate conversations](#generate-conversations).

## Refresh conversations

After a prompt change that adds, removes or changes a behavior, some
conversations no longer match what the agent would do: the agent still asks
for a date of birth the new prompt no longer wants, say. A replay run on
them tests the new prompt on histories it could not have produced. A
refresh brings each active conversation up to the agent's current prompt
revision as a new revision. The agent's own
[review](/api-reference/agents#review-a-prompt) reports when one is due:
`conversations.refresh_suggested` and how many conversations it would
update.

For each conversation, a refresh:

1. **Compares behaviors.** It diffs the rule inventories of the prompt the
   conversation was written for and the current one. A change in wording
   only rewrites nothing: the conversation is re-pointed to the new
   revision with the same rows.
2. **Finds the out-of-date turns.** One model read lists the agent turns
   that do something a removed behavior asked for, miss an added one, or
   follow the old version of a changed one. A conversation with none before
   its last example is re-pointed, and examples whose own reply changed get
   new answer keys.
3. **Restores before it rewrites.** When an earlier revision of the
   conversation was written for a prompt with these behaviors in their
   current state (a rule removed and later added back, for example), its
   rows are copied back, real caller audio included, as a `restored`
   revision.
4. **Otherwise rewrites the affected span**, from the first out-of-date
   agent turn to the last example's caller turn. Agent turns come from the
   same text pipeline a call uses, with the new prompt. A caller line that
   answered a removed question is dropped; the others are kept, with their
   real audio, when they still fit, and a simulated caller writes the rest.
   Rows after the last example are kept as they were.
5. **Moves the examples** to the new revision with rebuilt histories, and
   rewrites answer keys whose moment changed. An example whose caller turn
   is no longer in the conversation is retired with the reason, and comes
   back if a later refresh brings the turn back.

A conversation is retired when rewrites cover more than 60% of its agent
turns, or when it is more than 8 prompt revisions behind and no refresh can
bring it up to date. Retired conversations stay readable.

<CodeGroup>
```bash cURL
curl -X POST https://api.thunderphone.com/v1/agents/123/conversations/refresh \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dry_run": true}'
```
</CodeGroup>

| Field | Type | Description |
|-------|------|-------------|
| `conversation_ids` | integer[] | Refresh only these conversations of the agent, at most 500. Default: every active one |
| `from_prompt_revision_id` | integer | Refresh only the conversations written for this [prompt revision](/api-reference/prompt-revisions) |
| `use_draft` | boolean | Bring the conversations up to the draft prompt (default `true`) or the deployed one |
| `dry_run` | boolean | Return the quote only. Its `estimate.conversations` counts the `selected` conversations by how they stand: `current`, `no_behavior_change`, `behavior_changes`, `unknown` and `unrefreshable`, and lists the unrefreshable ones with the reason in `unrefreshable_conversations` (`id`, `reason`) |
| `quote_id` | string | The dry run's `quote_id`; required unless `dry_run` |
| `max_cost_cents` | integer | Hard spend cap, 1 to 10,000; required unless `dry_run` |

The responses match [Generate conversations](#generate-conversations).
A refresh's quote prices every conversation it reads as if it were
rewritten, every model call at its output limit, so a refresh usually
spends well under it. A conversation whose prompt revision is not known
cannot be refreshed, since nothing tells what changed since it was
written: it is quoted at 0, and the job leaves it as it is and reports it
as `skipped` with that `reason`. That is a conversation imported from a
call made before calls recorded their prompt revision, unless the agent
has exactly one deployed revision with the call's prompt text, which the
import then links it to. A refresh an API key, OAuth token or legacy token
starts (MCP included) leaves test-split conversations, and conversations
copied from held-out calls, out of its selection, its quote, its progress
and its cost; naming one in `conversation_ids` returns `403
test_conversation`, with nothing about the conversation. A person
refreshes them from the dashboard, and that job's rows for them show API
keys ids and counts only, without their outcome, and `outcome_counts`
counts only the rows the caller can see. An `active` example whose
history, reply or answer key a refresh rewrites goes back to
`pending_review` with the label `refreshed_needs_review`, so the
dashboard's validation runs do not grade against a key nobody reviewed;
an example someone archived meanwhile stays archived. A refresh runs
as a background job like generate and import; if its worker stops, it is
resumed and skips the conversations it already did. It runs on the target
prompt and the conversations frozen when it started: when the agent's
prompt changes while it runs, the job ends `failed`, keeping what it
refreshed.

## Retrieve a job

`GET /v1/conversation-jobs/{job_id}` reports a job's progress. Poll it
until `status` is `completed`, `stopped` or `failed`.

### Conversation job object

```json
{
  "id": 77,
  "agent_id": 123,
  "kind": "generate",
  "status": "completed",
  "params": { "count": 5, "style": "normal" },
  "estimate_cents": 61.4,
  "max_cost_cents": 150,
  "cost_cents": 48.9,
  "progress": { "done": 5, "total": 5, "step": "moments" },
  "error": "",
  "result": {
    "conversations": [
      { "id": 812, "split": "dev", "source": "simulated", "status": "active", "persona": { "name": "Dana" }, "end_reason": "agent_ended", "source_call_id": null, "example_ids": [5101, 5102] },
      { "id": 813, "agent_id": 123, "split": "test", "source": "simulated", "status": "active", "row_count": 14, "revision_count": 1, "cohort_number": 2, "hidden": true, "hidden_reason": "This item is in the agent's test split, whose content is hidden from API keys and coding agents. People can review it in the ThunderPhone dashboard." }
    ],
    "conversation_count": 2,
    "example_count": 2,
    "split_counts": { "dev": 1, "test": 1 },
    "errors": [],
    "hidden_error_count": 0
  },
  "created_at": "2026-10-19T16:39:58Z",
  "started_at": "2026-10-19T16:40:00Z",
  "completed_at": "2026-10-19T16:43:12Z"
}
```

| Field | Description |
|-------|-------------|
| `status` | `pending`, `running`, `completed`, `stopped` (at the cost cap; what it wrote is kept) or `failed` |
| `cost_cents` | What the job has spent so far, against `max_cost_cents` |
| `progress` | Done and total counts, the current step, and cost by step |
| `result.conversations` | What the job wrote. For API keys, a test-split conversation appears as the [hidden shape](#conversation-object) of a conversation, without its example count. For a refresh, each visible row also has its `outcome` (`current`, `repointed`, `restored`, `rewritten`, `retired`, `skipped` or `failed`), the new `revision_id`, how many examples were retired, brought back or given new answer keys, and a `reason` when there is one |
| `result.example_count` | Examples of the conversations the caller can see: for API keys, the dev split's |
| `result.outcome_counts` | A refresh job's outcomes, counted over the rows the caller can see |
| `result.errors` | Conversations or calls that failed without stopping the job, each with its `conversation_id`, `split` and `message`. For API keys, errors about test-split conversations, and conversations copied from held-out calls, are left out, and a call that failed before its conversation existed has a fixed message: the call id and the error's type |
| `result.hidden_error_count` | How many errors were left out for this caller |

The examples a job creates have status `pending_review`. Replay runs use
them right away; [validation runs](/api-reference/validation-sets) use them
once someone sets them `active`.

## Errors

| Status | Condition |
|--------|-----------|
| `400` | `validation_error`: an unknown field, a value of the wrong type or out of range, or a real run without `max_cost_cents` (the response includes `estimate_cents`) |
| `400` | `quote_required`, `quote_expired`, `quote_invalid`, `quote_mismatch` (another agent, kind or set of arguments, or the models or prices changed), `quote_outdated` (the run would now cost more than quoted) or `quote_used` (the quote already paid for a job): run the dry run again and send its `quote_id` |
| `503` | `quote_unverifiable`: the quote could not be recorded as used right now, so nothing ran; retry with it |
| `403` | `feature_not_enabled`: the organization does not have the agent iteration toolkit. `insufficient_scope`: an OAuth token without `testing:write` (`testing:read` to read) |
| `403` | For a refresh started by an API key, OAuth token or legacy token: `test_conversation`, a `conversation_ids` entry is a test-split conversation or one copied from a held-out call (refresh those from the dashboard) |
| `404` | The agent, conversation or job is not in this organization |
| `400` | For a refresh: `conversation_ids` that are not the agent's, more than 500 of them, or a `from_prompt_revision_id` that is not one of the agent's prompt revisions |
| `409` | `job_active`: the agent already has a job running (`existing_job_id`). `replay_run_active`: the agent has a [replay run](/api-reference/replay-runs) going (`existing_run_id`). Each agent runs one job or replay run at a time; the quote stays usable |

## MCP

The [MCP server](/guides/thunderphone-mcp-server) exposes these as
`list_conversations`, `get_conversation`, `get_conversation_revisions`,
`generate_conversations`, `import_conversations`, `refresh_conversations`
and `get_conversation_job`.

---

## Related

<CardGroup cols={2}>
  <Card title="Replay Runs" icon="rotate" href="/api-reference/replay-runs">
    Replay the examples against a changed configuration.
  </Card>
  <Card title="Validation Sets" icon="list-check" href="/api-reference/validation-sets">
    The examples and their answer keys.
  </Card>
</CardGroup>