---
title: "Replay Runs"
description: "Replay an agent's test moments under its draft or a changed configuration, judge each new reply against its answer key, and compare configurations side by side."
---
A replay run takes a sample of an agent's [validation examples](/api-reference/validation-sets),
replays each moment under a configuration, and judges the new reply against
the example's answer key. Use it to check a prompt edit before you deploy
it, and to compare configurations (product, audio tier, feedback mode,
voice, endpointing, language or a candidate prompt) on the same examples.
The examples come from the agent's [test conversations](/api-reference/conversations):
generated ones for a new agent, imported calls once it has some.
## How a moment is replayed
Each example is one caller turn with the conversation before it. The run
replays it with one of two engines:
- **Exact** replays the turn as a live call would. The turn's recorded
caller audio is played into a real call session in Core with the
configuration being tested, so transcription, end-of-speech detection,
the wait cover (an acknowledgement, ticks or silence) and the reply run
as in production. Exact replay needs a Bolt v2 configuration and a turn
with recorded caller audio. It reports latency to the first audio and
what covered the wait.
- **Text** sends the conversation's transcript to the agent's model with
the configuration's prompt and tools, and judges the reply text.
Generated conversations have no audio, so they always replay as text.
`engine: "auto"` (the default) uses exact replay where it can and text
everywhere else; each result says which engine ran and why. `engine:
"exact"` skips the examples that cannot be replayed exactly. On the dev
split, `auto` also falls back to text when Core cannot run an exact replay
(it predates the endpoint, or does not confirm the replay's budget); the
result says so and the run is `degraded`. On the test split each example's
engine is set before the run from the example's data (exact for recorded
caller audio, text for a text-only example), a configuration with no exact
path for an audio example is refused, `engine` must be omitted or `auto`
(`exact` or `text` is refused with `test_engine_per_example`), and
nothing falls back (see
[Dev and test splits](#dev-and-test-splits)). Pass rates are always
reported per engine.
A model judge reads each reply against the prompt and the example's answer
key and returns `pass`, `warn` or `fail`, with each issue's type, severity
(`warn` or `fail`), explanation and evidence. An example that could not be replayed or judged is `skipped` or
`error`; errors count against the pass rate.
## The agent iteration toolkit
These endpoints are part of the agent iteration toolkit, which is enabled
per organization while it is in beta. Without it they answer `403` with
`code: "feature_not_enabled"` and a `next_step`; ask
[support](mailto:support@thunderphone.com) to enable it.
### Dev and test splits
Every example belongs to the agent's dev split or its test split (see
[Dev and test splits](/api-reference/conversations#dev-and-test-splits)).
Tune on the dev split: `split: "dev"` (the default) returns every example's
replayed reply, the judge's verdict and the examples fixed and broken
against a baseline. Measure a finished change on the test split:
- a test result spends its examples. The test split is divided into
cohorts: each new test conversation joins the agent's open cohort, which
seals when it is full. A test run replays one sealed cohort, the oldest
unused one that has an example to replay (a cohort with none is passed
over, so it never blocks the next), whole: `sample_size` and `sample_seed` are ignored, so the
totals never narrow to a chosen example. The run claims its cohort when
it is created, before anything is queued, and its first result that
discloses anything spends it. A later test run, for any prompt, needs
another cohort; a run that ends before disclosing anything (refused, or
stopped by its cap before the first reply) hands its cohort back;
- with no unused cohort left, a test run is refused with
`409 no_test_cohort` and a `next_step`: import more calls or generate
conversations, which fill new cohorts;
- comparisons are dev-only: a request with `configs` on the test split is
refused with `400 comparison_dev_only`. Compare on dev, then run the test
split once for the configuration you pick, so the test result is never
the basis of the choice;
- each example's engine is set before the run, from the example's data: an
example with recorded caller audio replays exactly, a text-only
(simulated) one as text. Leave `engine` out: any value but `auto` is
refused with `400 test_engine_per_example`, before a cohort is claimed.
Totals come per engine;
- a test run never spends recorded caller audio on text. Exact replay
exists for Bolt v2 only, so when the cohort holds an example with
recorded caller audio and the configuration is anything else, the run is
refused with `409 test_needs_exact_replay` and a `next_step` before
anything is evaluated, and the cohort stays unused. Measure such a
configuration on the dev split, where its text replays of audio examples
are marked `degraded`. A cohort of text-only examples runs on text under
any configuration;
- a test run never falls back to text. Where its examples replay exactly,
its first exact replay runs alone, before anything is judged; when Core
has no exact replay or does not confirm its budget, the run ends `failed`
with `stop_reason: "exact_replay_unavailable"`, no results, and its
cohort unused. When Core's replay capacity stays busy through every
retry, it ends `failed` with `stop_reason: "core_busy"`, its cohort
unused too;
- with an API key or OAuth token (and so over MCP), a test run returns its
totals only: pass rate, verdict counts, failures by severity and kind,
latency and cost, with `"hidden": true`. People signed in to the
dashboard see every example.
On the dev split, an example copied from a held-out caller's call (their
caller sits in another agent's test split; see [held-out
calls](/api-reference/conversations#dev-and-test-splits)) reaches API keys
and OAuth tokens as its `example_id` and `"held_out": true` only, and is
left out of the summary and flips; the run says how many with
`held_out_example_count`.
A change made after reading test results is measured on a fresh cohort.
### One run or job at a time
An agent runs one replay run or one [conversation
job](/api-reference/conversations#retrieve-a-job) (generate, import or
refresh) at a time, so a refresh never rewrites the examples a replay is
running on. A start while a replay run is going is refused with `409
replay_run_active` (`existing_run_id`), and while a conversation job is
going with `409 conversation_job_active` (`existing_job_id`). Wait for it to
finish, then start again with the same `quote_id`. A run whose worker
stopped and did not come back within minutes does not block: the next
start replaces it.
### Data lifecycle
When a call's recording is turned off, or the call is deleted, the test
conversations imported from it, their examples, the examples promoted
straight from the call, those examples' replay results and Core's replay
cache for the call are removed. A purge that fails is retried until it
finishes, and the replay cache is swept once more an hour later, for a
replay that was still running. Until then a result of such an example
returns its `verdict` and ids only, with `content_withheld:
"recording_disabled"`, to every caller.
### PHI
Each example replays under its source call's PHI designation, the one the
call ran with; an example whose call recorded none replays as PHI. When any
sampled example, the deployed agent or the draft is PHI, every
configuration runs as PHI, and one that a PHI call cannot run (an
ElevenLabs voice, an experimental product, an audio or endpointing tier
without a BAA) is refused with `400` and the reason rather than replayed
under its label with something else.
### `dry_run`, `quote_id` and `max_cost_cents`
Replays and judging spend credits. Send the request with `"dry_run": true`
first: the response is the estimate, how many examples were sampled, the
`sample_seed` it drew, which engine each configuration would use, and a
`quote_id`, and nothing runs. For the test split it also gives the cohort
the run would spend (`test_cohort`: its `number` and `example_count`) and
how many unused cohorts are left (`test_cohorts_left`), or `409
no_test_cohort` when there is none.
A real run sends the same request again without `dry_run`, with that
`quote_id` and `sample_seed`, so it replays exactly the quoted sample. The
`quote_id` is good for 15 minutes, for this agent and request only (the
configurations and overrides, split, engine, configuration source, sample
size and seed or test cohort, baseline run), and for one run: it is used up
once, durably, in the transaction that creates the run, and a second run
with it is refused with `quote_used`. A start that is refused or fails
before the run exists, such as `409 replay_run_active` or a database
error, leaves it usable. Once the run exists the quote is spent, so a run
that then fails or stops is repeated with a new dry run and its new
`quote_id`. It is refused when it is missing,
expired or forged, when the request or the replay prices differ, or when
the run would now cost more than it quoted (the prompt or the examples
changed since). A test run quoted on a cohort that another test run
claimed meanwhile is refused with `quote_mismatch`: quote again.
The real run also needs `max_cost_cents`, a cap in US cents from 1 to 100,000,
shared by every configuration of a comparison. The cap is hard. Before
each request (a replay, then the judge's read of its reply) the run sets
aside the most that request can cost, and stops instead of sending one
that does not fit: the example is `skipped` with `skipped_reason:
"cost_cap"`, every example after it too, and the run's `stop_reason` is
`cost_cap`. What the requests really cost is then booked from their usage,
and that is what the run is charged. A text replay runs under a token
budget the agent's model is held to. An exact replay carries a hard budget
that Core enforces on every model and Decisions request of the replayed
turn, sized for a normal turn: each span's reply race and end-of-speech
checks, the hearing tier's fork, on an Express or Lite configuration one more reply
leg per span (the standard lane takes over a reply that is only a tool
call), one watchdog or failover leg, and one
re-run of the kind production takes (the reply race again, after a spoken
tool call on every leg, an empty reply or the language-switch guard, and
the watchdog's retry), counted in UTF-8 bytes; its transcript seats are
bounded by the turn's audio. The run sets aside that budget at the prices
of the models the configuration can reach, plus the seats:
about 85 cents to $1.25 per exact replay for a typical turn, against a real
cost of a few cents. A trace that does not
confirm the budget is booked at that set-aside; on the dev split the rest
of the run replays as text (`engine_fallback:
"core_did_not_enforce_replay_limits"`), and a test run is refused. A turn
that needs more than its budget (a second re-run, more retries) has the
extra request refused by Core and is an `error` with `error_code:
"replay_budget_exhausted"`, not a verdict, since its reply may not be what
a call would say; the summary counts it as `budget_exhausted`, outside the
pass rate. On the test split such a result keeps nothing of the replay and
spends no cohort, so a test run whose only results are such turns hands
its cohort back unused. A request that costs more than its set-aside stops the run
(`stop_reason: "cost_over_bound"`).
Because of the set-aside, a run stops with up to one request's bound per
worker unspent. A set-aside is released when its request ends, and each
worker replays one example at a time, so a cap only needs the expected
spend plus one set-aside per concurrent worker. The dry run's
`largest_reservation_cents` is the largest single request,
`concurrent_workers` how many examples the run replays at once, and
`suggested_max_cost_cents` the cap that lets it finish:
`estimated_cost_cents` plus `concurrent_workers` times
`largest_reservation_cents`. With exact replays, that is well above what
the run will be charged, and a cap below it can stop the run early.
`upper_bound_cents` is the most the run can spend, every request at its
bound, and `min_max_cost_cents` is the smallest cap that starts anything.
Replays reuse stored work, so a run usually costs less than its estimate.
## Endpoints
| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/v1/agents/{agent_id}/replay-runs` | Start a replay run or a comparison, or estimate one |
| `GET` | `/v1/agents/{agent_id}/replay-runs` | The agent's 20 most recent replay runs |
| `GET` | `/v1/replay-runs/{run_id}` | One run with its results |
| `GET` | `/v1/replay-run-groups/{group_id}` | A comparison: each configuration side by side |
## Start a replay run
```bash cURL
curl -X POST https://api.thunderphone.com/v1/agents/123/replay-runs \
-H "Authorization: Bearer sk_live_YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"split": "dev", "sample_size": 20, "dry_run": true}'
```
```python Python
estimate = requests.post(
"https://api.thunderphone.com/v1/agents/123/replay-runs",
headers={"Authorization": "Bearer sk_live_YOUR_API_KEY"},
json={"split": "dev", "sample_size": 20, "dry_run": True},
).json()
run = requests.post(
"https://api.thunderphone.com/v1/agents/123/replay-runs",
headers={"Authorization": "Bearer sk_live_YOUR_API_KEY"},
json={
"split": "dev",
"sample_size": 20,
"sample_seed": estimate["sample_seed"],
"quote_id": estimate["quote_id"],
"max_cost_cents": 200,
},
).json()
```
### Request fields
| Field | Type | Description |
|-------|------|-------------|
| `split` | string | `dev` (default) or `test` |
| `engine` | string | `auto` (default), `exact` or `text` |
| `config_source` | string | Start from the agent's `draft` (default) or its `deployed` configuration |
| `overrides` | object | Changes to that configuration for this run; see [Overrides](#overrides). Not with `configs` |
| `label` | string | Your name for the run, up to 100 characters |
| `configs` | object[] | 2 to 4 configurations, each `{label, overrides}`, replayed on the same examples. The first is the baseline. Starts a [comparison](#compare-configurations); dev split only |
| `sample_size` | integer | Examples to replay, 1 to 200. Default 20. Ignored on the test split, which replays one whole test cohort |
| `sample_seed` | integer | The same seed draws the same examples. A dry run returns the seed it drew with. Ignored on the test split |
| `max_cost_cents` | integer | Spend cap, 1 to 100,000; required unless `dry_run`. A hard cap; it only needs the expected spend plus the dry run's `largest_reservation_cents` per concurrent worker, which the dry run returns as `suggested_max_cost_cents` |
| `dry_run` | boolean | Return the estimate and a `quote_id` only |
| `quote_id` | string | The `quote_id` of a dry run of this same request; required unless `dry_run`, and good for one run. Send the dry run's `sample_seed` with it |
| `baseline_run_id` | integer | An earlier replay run of this agent on the same split; the run reports the examples fixed and broken against it |
Examples are drawn by weight: recent conversations, real calls, conversations
that match the newest prompt revisions, and conversations that have been
rewritten least come first. Examples from calls whose recording was later
turned off are left out, and removed (see [Data
lifecycle](#data-lifecycle)).
### Overrides
| Field | Description |
|-------|-------------|
| `prompt` | A candidate prompt instead of the configuration's. The run records it as a `candidate` [prompt revision](/api-reference/prompt-revisions) without changing the draft |
| `product` | The product tier, as the agent's `product` field |
| `thinking_level` | `base` or `extra` |
| `storm_feedback_mode` | `acknowledgement`, `tick` or `none`: what the caller hears while a slow reply is prepared |
| `voice` | A voice name |
| `language` | The language the call runs in |
| `audio_tier` | The Bolt v2 audio tier: `lean`, `standard` or `hearing`. Plain Bolt only; a configuration on another product that sets `lean` or `hearing` is refused with `400`, and `standard` passes |
| `endpointing` | The Bolt v2 endpointing tier, how long the caller must stay quiet before the reply request: `faster`, `fast`, `normal` or `patient` |
| `intelligence` | The Bolt v2 intelligence tier, the model that writes the replies: `express`, `lite` or `standard`. Plain Bolt on v2 only; a configuration on another product that sets `express` or `lite` is refused with `400`, and `standard` passes |
An override applies where the agent's own field would, so it passes
through the same composition and voice rules a call does. `null` means the
configuration's own value.
### Responses
A dry run returns `200`:
```json
{
"dry_run": true,
"split": "dev",
"engine": "auto",
"phi_mode": "phi",
"sample_seed": 1802233117,
"max_cost_cents": null,
"example_count": 20,
"configs": [
{ "label": "", "overrides": {}, "engines": { "exact": 14, "text": 6, "skipped": 0 }, "estimated_cost_cents": 128.94, "upper_bound_cents": 1881.97 }
],
"estimated_cost_cents": 128.94,
"upper_bound_cents": 1881.97,
"largest_reservation_cents": 103.605,
"min_max_cost_cents": 104,
"concurrent_workers": 2,
"suggested_max_cost_cents": 337,
"judge_max_completion_tokens": 4000,
"notes": [
"Exact replays are estimated with an empty stage cache; the judge at its output bound.",
"max_cost_cents is a hard cap: each request reserves its upper bound before it is sent and is refused when that does not fit what is left, and the run then stops. A reservation is released when its request ends, and this run replays 2 examples at a time, so a cap only needs the expected spend plus one largest_reservation_cents per concurrent worker: set max_cost_cents to about suggested_max_cost_cents (337), estimated_cost_cents plus 2 times largest_reservation_cents. upper_bound_cents is the most the run can spend.",
"Both engines run under a token budget Core enforces (replay_limits): …"
],
"quote_id": "eyJvcmciOjQyLC…",
"quote_expires_in_seconds": 900,
"next_step": "Show this estimate, then run again with the same request and this sample_seed, without dry_run, with this quote_id and with max_cost_cents."
}
```
This is the estimator's answer for 14 examples with recorded caller audio
and 6 text-only ones, under a Bolt v2 draft with a 10 KB prompt, replayed as
PHI: each exact replay sets aside about a dollar (`largest_reservation_cents`)
and costs a few cents, so a cap of `estimated_cost_cents` plus one
set-aside for each of its two workers (`suggested_max_cost_cents`) lets
the run finish; `min_max_cost_cents` is only the floor.
A test-split dry run adds `test_cohort` (`{"number": 3, "example_count":
41}`) and `test_cohorts_left`.
A real run returns `201` with the [run](#replay-run-object), status
`pending`. A comparison returns `201` with the
[comparison](#compare-configurations). Poll until the status is `completed`
or `failed`.
## Replay run object
`GET /v1/replay-runs/{run_id}` returns the run with its results. Add
`include_trace=true` for Core's full trace of each exact replay instead of
its summary.
```json
{
"id": 9031,
"agent_id": 123,
"label": "shorter callback rule",
"split": "dev",
"engine": "auto",
"config_source": "draft",
"config_overrides": {},
"status": "completed",
"stale": false,
"degraded": false,
"stop_reason": null,
"error_message": null,
"compare_group_id": null,
"baseline_run_id": 9020,
"prompt_revision_id": 4182,
"prompt_hash": "5b0e…",
"sample_seed": 1802233117,
"sample_size": 20,
"max_cost_cents": 200,
"estimated_cost_cents": 93.0,
"cost_cents": 51.6,
"test_cohort": null,
"created_at": "2026-10-19T17:10:02Z",
"started_at": "2026-10-19T17:10:03Z",
"completed_at": "2026-10-19T17:14:40Z",
"summary": {
"total": 20,
"replayed": 20,
"phi_mode": "no_phi",
"verdicts": { "pass": 16, "warn": 2, "fail": 2, "error": 0, "skipped": 0 },
"budget_exhausted": 0,
"pass_rate": null,
"by_engine": {
"exact": { "verdicts": { "pass": 12, "warn": 1, "fail": 1, "error": 0, "skipped": 0 }, "budget_exhausted": 0, "pass_rate": 0.857 },
"text": { "verdicts": { "pass": 4, "warn": 1, "fail": 1, "error": 0, "skipped": 0 }, "budget_exhausted": 0, "pass_rate": 0.667 }
},
"degraded": false,
"failures": { "by_severity": { "fail": 2, "warn": 2 }, "by_kind": { "missed_instruction": 3, "two_questions": 1 } },
"engine_fallbacks": 0,
"latency_to_first_audio_s": { "p50": 0.94, "p90": 1.61 },
"latency_to_reply_audio_s": { "p50": 1.42, "p90": 2.37 },
"wait_cover_share": { "acknowledgement": 0.07, "ticks": 0.0, "silence": 0.93, "filler": 0.0, "none": 0.0 },
"fork_rate": 0.07,
"cost_cents": 51.6,
"cost_per_turn_cents": 2.58
},
"flips": { "baseline_run_id": 9020, "by_engine": { "exact": { "compared": 13, "fixed": [5102, 5110], "broken": [5131] }, "text": { "compared": 6, "fixed": [], "broken": [] } }, "not_comparable": 0 },
"results": [
{
"example_id": 5131,
"verdict": "fail",
"engine_used": "exact",
"engine_fallback": null,
"exact_unavailable": null,
"skipped_reason": null,
"error": null,
"expectation": "Confirm the callback number digit by digit before ending the call.",
"original_response": { "spoken_response": "Great, we'll call you back soon. Anything else?", "tool_calls": [] },
"replayed_output": { "spoken_response": "Thanks, we'll call you back. Goodbye.", "tool_calls": [] },
"instruction_verdict": { "status": "fail", "summary": "The agent ended the call without confirming the callback number.", "issues": [{ "severity": "fail", "type": "missed_instruction", "explanation": "The prompt asks to read the number back before ending the call.", "evidence": "Thanks, we'll call you back. Goodbye." }], "confidence": 0.9 },
"metrics": { "first_audio_s": 0.91, "reply_audio_s": 1.38, "wait_cover": "silence", "forked": false },
"cost_cents": 4.5,
"trace_summary": { "reply": "…", "wait_cover": "…", "end_of_speech": "…" }
}
]
}
```
| Field | Description |
|-------|-------------|
| `status` | `pending`, `running`, `completed` or `failed` |
| `stop_reason` | Why the run stopped early: `cost_cap` (the next request did not fit `max_cost_cents`), `cost_over_bound` (a request cost more than its set-aside), `core_limits_unsupported` (the text replay was not held to its token budget) or, for a test run, `exact_replay_unavailable` (Core could not replay its examples exactly under a confirmed budget) or `core_busy` (Core's replay capacity stayed busy); either test stop leaves the run `failed` and its cohort unused |
| `stale` | `true` while the run's worker has stopped; it resumes within minutes, or a new run for the agent replaces it |
| `degraded` | `true` when an exact replay fell back to text (dev runs only) or nothing could be judged |
| `test_cohort` | A test run's cohort: its `number`, and `spent` once a result disclosed anything. `null` on the dev split |
| `prompt_revision_id` | The [prompt revision](/api-reference/prompt-revisions) the run tested |
| `summary.pass_rate` | Passes over judged examples, errors included, when one engine replayed every example; `null` when exact and text both ran (read `summary.by_engine`) or before any is judged |
| `summary.by_engine` | Verdicts and pass rate per engine (`exact`, `text`); exact and text verdicts are never mixed into one number |
| `summary.budget_exhausted` | Exact replays that needed more than their budget (`error_code: "replay_budget_exhausted"`), counted apart from the verdicts and outside the pass rate; also per engine |
| `summary.phi_mode` | The PHI designation the configuration ran under |
| `summary.latency_to_first_audio_s` | p50 and p90 seconds from the caller's last speech to the agent's first audio, over exact replays |
| `summary.wait_cover_share` | Over exact replays, how often the caller waited on an acknowledgement, ticks, silence or filler |
| `flips` | Against `baseline_run_id`, per engine: how many examples both runs judged on that engine, and the ids now passing (`fixed`) and now failing (`broken`). An example judged on different engines counts in `not_comparable`. Dev runs only |
| `results[].engine_fallback` | Why an exact replay ran as text instead (dev runs only): `core_has_no_replay_endpoint`, `core_did_not_enforce_replay_limits` or `turn_audio_unreadable` |
| `results[].error_code` | `replay_budget_exhausted` when the replayed turn needed more than its budget and Core refused the extra request; the result is an `error`, not a verdict |
| `results[].exact_unavailable` | Why `auto` chose text: `configuration_not_bolt_v2` or `no_turn_audio` |
| `results[].skipped_reason` | `cost_cap` (or the run's other `stop_reason`), `configuration_not_bolt_v2` or `no_turn_audio` (with `engine: "exact"`), `core_busy` (Core's replay capacity stayed busy; nothing ran or was charged), `recording_disabled`, or `history_audio_missing` |
| `results[].content_withheld` | `recording_disabled` when the example's source call turned recording-disabled after the run judged it: the result keeps its `verdict` and ids, and its expectation, replies, instruction verdict, metrics and trace are empty, for every caller |
| `results[].original_response` | What the agent said and which tools it called at that moment originally (`spoken_response`, `tool_calls`) |
| `results[].replayed_output` | The same for the replay |
A test-split run read with an API key has no `flips` or `results`, and has
`"hidden": true`.
`GET /v1/agents/{agent_id}/replay-runs` returns `{"runs": [...]}`, the
agent's 20 most recent runs without results.
## Compare configurations
Send `configs` instead of `overrides` to replay the same examples under 2
to 4 configurations. One run is created per configuration; the first is the
baseline, and every other run reports its flips against it. Comparisons run
on the dev split only; on the test split they are refused with `400
comparison_dev_only`. Check the
configuration before blaming the prompt: some failures belong to the
setup, such as mishearing spelled letters on a lower audio tier, or a slow
reply, or one that cuts the caller off, from the endpointing tier.
```json
{
"configs": [
{ "label": "standard", "overrides": {} },
{ "label": "hearing", "overrides": { "audio_tier": "hearing" } }
],
"sample_size": 30,
"sample_seed": 77120394,
"quote_id": "",
"max_cost_cents": 400
}
```
The response, and `GET /v1/replay-run-groups/{group_id}`, is the comparison:
```json
{
"id": "0f6b4f1e-9a51-4f7e-8a43-1c2a5d3e6b70",
"agent_id": 123,
"split": "dev",
"engine": "auto",
"config_source": "draft",
"status": "running",
"stop_reason": null,
"sample_seed": 77120394,
"sample_size": 30,
"max_cost_cents": 400,
"cost_cents": 12.4,
"baseline_run_id": 9040,
"configs": [
{ "run_id": 9040, "label": "standard", "overrides": {}, "baseline": true, "status": "running", "prompt_revision_id": 4182, "summary": { "pass_rate": null } },
{ "run_id": 9041, "label": "hearing", "overrides": { "audio_tier": "hearing" }, "baseline": false, "status": "pending", "prompt_revision_id": 4182, "summary": { "pass_rate": null }, "flips": null }
],
"created_at": "2026-10-19T17:20:00Z"
}
```
Each configuration's `summary` has the fields of a
[run's summary](#replay-run-object), so pass rate, latency, wait cover and
cost per turn line up side by side. `GET /v1/replay-runs/{run_id}` for any
run of the comparison also returns the comparison as `group`.
## Errors
| Status | Condition |
|--------|-----------|
| `400` | An unknown field or a value out of range, an unsupported override, `max_cost_required` (a real run without `max_cost_cents`), `max_cost_below_minimum` (a `max_cost_cents` below the quote's `min_max_cost_cents`, which the response repeats: no replay could start), `no_examples` (the dev split has no examples yet), `configuration_refused` (a configuration a PHI replay cannot run, with the reason), `comparison_dev_only` (`configs` on the test split), `test_engine_per_example` (an `engine` other than `auto` on the test split), or a `baseline_run_id` that is not a run of this agent on the same split |
| `403` | `insufficient_scope`: an OAuth token without `testing:write` (`testing:read` to read runs) |
| `403` | `feature_not_enabled`: the organization does not have the agent iteration toolkit |
| `404` | The agent, run or comparison is not in this organization |
| `400` | `quote_required`, `quote_expired`, `quote_invalid`, `quote_mismatch` (another agent or request, the prices changed, or another test run took the quoted cohort), `quote_outdated` (the run would now cost more than quoted) or `quote_used` (the quote already paid for a run): run the dry run again and send its `quote_id` and `sample_seed` |
| `409` | `replay_run_active`: the agent already has a run going (`existing_run_id`). `conversation_job_active`: the agent has a conversation job going (`existing_job_id`). `no_test_cohort`: no unused test cohort is left, or none is full yet. `test_cohort_changed`: another test run claimed the cohort first. `test_cohort_empty`: the next cohort's conversations have no examples yet. `test_needs_exact_replay`: the test cohort holds recorded caller audio and the configuration has no exact path (anything but Bolt v2); nothing ran and the cohort stays unused, so measure it on the dev split |
| `503` | `quote_unverifiable`: the quote could not be recorded as used right now, so nothing ran; retry with it |
| `429` | `replay_run_limit`: the organization already has 8 replay runs going |
Every error body has a `detail`; most also have a `code` and a `next_step`.
## MCP
The [MCP server](/guides/thunderphone-mcp-server) exposes these as
`run_replays`, `compare_configs` and `get_replay_run`.
---
## Related
Where the examples come from.
The prompt text each run tested.