Replay Runs

Replay an agent

A replay run takes a sample of an agent's validation examples, replays each moment under a configuration, and judges the new reply against the example's answer key. Use it to check a prompt edit before you deploy it, and to compare configurations (product, audio tier, feedback mode, voice, endpointing, language or a candidate prompt) on the same examples.

The examples come from the agent's test conversations: generated ones for a new agent, imported calls once it has some.

How a moment is replayed#

Each example is one caller turn with the conversation before it. The run replays it with one of two engines:

• Exact replays the turn as a live call would. The turn's recorded caller audio is played into a real call session in Core with the configuration being tested, so transcription, end-of-speech detection, the wait cover (an acknowledgement, ticks or silence) and the reply run as in production. Exact replay needs a Bolt v2 configuration and a turn with recorded caller audio. It reports latency to the first audio and what covered the wait.
• Text sends the conversation's transcript to the agent's model with the configuration's prompt and tools, and judges the reply text. Generated conversations have no audio, so they always replay as text.

engine: "auto" (the default) uses exact replay where it can and text everywhere else; each result says which engine ran and why. engine: "exact" skips the examples that cannot be replayed exactly. On the dev split, auto also falls back to text when Core cannot run an exact replay (it predates the endpoint, or does not confirm the replay's budget); the result says so and the run is degraded. On the test split each example's engine is set before the run from the example's data (exact for recorded caller audio, text for a text-only example), a configuration with no exact path for an audio example is refused, engine must be omitted or auto (exact or text is refused with test_engine_per_example), and nothing falls back (see Dev and test splits). Pass rates are always reported per engine.

A model judge reads each reply against the prompt and the example's answer key and returns pass, warn or fail, with each issue's type, severity (warn or fail), explanation and evidence. An example that could not be replayed or judged is skipped or error; errors count against the pass rate.

The agent iteration toolkit#

These endpoints are part of the agent iteration toolkit, which is enabled per organization while it is in beta. Without it they answer 403 with code: "feature_not_enabled" and a next_step; ask support to enable it.

Dev and test splits#

Every example belongs to the agent's dev split or its test split (see Dev and test splits). Tune on the dev split: split: "dev" (the default) returns every example's replayed reply, the judge's verdict and the examples fixed and broken against a baseline. Measure a finished change on the test split:

• a test result spends its examples. The test split is divided into cohorts: each new test conversation joins the agent's open cohort, which seals when it is full. A test run replays one sealed cohort, the oldest unused one that has an example to replay (a cohort with none is passed over, so it never blocks the next), whole: sample_size and sample_seed are ignored, so the totals never narrow to a chosen example. The run claims its cohort when it is created, before anything is queued, and its first result that discloses anything spends it. A later test run, for any prompt, needs another cohort; a run that ends before disclosing anything (refused, or stopped by its cap before the first reply) hands its cohort back;
• with no unused cohort left, a test run is refused with 409 no_test_cohort and a next_step: import more calls or generate conversations, which fill new cohorts;
• comparisons are dev-only: a request with configs on the test split is refused with 400 comparison_dev_only. Compare on dev, then run the test split once for the configuration you pick, so the test result is never the basis of the choice;
• each example's engine is set before the run, from the example's data: an example with recorded caller audio replays exactly, a text-only (simulated) one as text. Leave engine out: any value but auto is refused with 400 test_engine_per_example, before a cohort is claimed. Totals come per engine;
• a test run never spends recorded caller audio on text. Exact replay exists for Bolt v2 only, so when the cohort holds an example with recorded caller audio and the configuration is anything else, the run is refused with 409 test_needs_exact_replay and a next_step before anything is evaluated, and the cohort stays unused. Measure such a configuration on the dev split, where its text replays of audio examples are marked degraded. A cohort of text-only examples runs on text under any configuration;
• a test run never falls back to text. Where its examples replay exactly, its first exact replay runs alone, before anything is judged; when Core has no exact replay or does not confirm its budget, the run ends failed with stop_reason: "exact_replay_unavailable", no results, and its cohort unused. When Core's replay capacity stays busy through every retry, it ends failed with stop_reason: "core_busy", its cohort unused too;
• with an API key or OAuth token (and so over MCP), a test run returns its totals only: pass rate, verdict counts, failures by severity and kind, latency and cost, with "hidden": true. People signed in to the dashboard see every example.

On the dev split, an example copied from a held-out caller's call (their caller sits in another agent's test split; see held-out calls) reaches API keys and OAuth tokens as its example_id and "held_out": true only, and is left out of the summary and flips; the run says how many with held_out_example_count.

A change made after reading test results is measured on a fresh cohort.

One run or job at a time#

An agent runs one replay run or one conversation job (generate, import or refresh) at a time, so a refresh never rewrites the examples a replay is running on. A start while a replay run is going is refused with 409 replay_run_active (existing_run_id), and while a conversation job is going with 409 conversation_job_active (existing_job_id). Wait for it to finish, then start again with the same quote_id. A run whose worker stopped and did not come back within minutes does not block: the next start replaces it.

Data lifecycle#

When a call's recording is turned off, or the call is deleted, the test conversations imported from it, their examples, the examples promoted straight from the call, those examples' replay results and Core's replay cache for the call are removed. A purge that fails is retried until it finishes, and the replay cache is swept once more an hour later, for a replay that was still running. Until then a result of such an example returns its verdict and ids only, with content_withheld: "recording_disabled", to every caller.

PHI#

Each example replays under its source call's PHI designation, the one the call ran with; an example whose call recorded none replays as PHI. When any sampled example, the deployed agent or the draft is PHI, every configuration runs as PHI, and one that a PHI call cannot run (an ElevenLabs voice, an experimental product, an audio or endpointing tier without a BAA) is refused with 400 and the reason rather than replayed under its label with something else.

dry_run, quote_id and max_cost_cents#

Replays and judging spend credits. Send the request with "dry_run": true first: the response is the estimate, how many examples were sampled, the sample_seed it drew, which engine each configuration would use, and a quote_id, and nothing runs. For the test split it also gives the cohort the run would spend (test_cohort: its number and example_count) and how many unused cohorts are left (test_cohorts_left), or 409 no_test_cohort when there is none.

A real run sends the same request again without dry_run, with that quote_id and sample_seed, so it replays exactly the quoted sample. The quote_id is good for 15 minutes, for this agent and request only (the configurations and overrides, split, engine, configuration source, sample size and seed or test cohort, baseline run), and for one run: it is used up once, durably, in the transaction that creates the run, and a second run with it is refused with quote_used. A start that is refused or fails before the run exists, such as 409 replay_run_active or a database error, leaves it usable. Once the run exists the quote is spent, so a run that then fails or stops is repeated with a new dry run and its new quote_id. It is refused when it is missing, expired or forged, when the request or the replay prices differ, or when the run would now cost more than it quoted (the prompt or the examples changed since). A test run quoted on a cohort that another test run claimed meanwhile is refused with quote_mismatch: quote again.

The real run also needs max_cost_cents, a cap in US cents from 1 to 100,000, shared by every configuration of a comparison. The cap is hard. Before each request (a replay, then the judge's read of its reply) the run sets aside the most that request can cost, and stops instead of sending one that does not fit: the example is skipped with skipped_reason: "cost_cap", every example after it too, and the run's stop_reason is cost_cap. What the requests really cost is then booked from their usage, and that is what the run is charged. A text replay runs under a token budget the agent's model is held to. An exact replay carries a hard budget that Core enforces on every model and Decisions request of the replayed turn, sized for a normal turn: each span's reply race and end-of-speech checks, the hearing tier's fork, on an Express or Lite configuration one more reply leg per span (the standard lane takes over a reply that is only a tool call), one watchdog or failover leg, and one re-run of the kind production takes (the reply race again, after a spoken tool call on every leg, an empty reply or the language-switch guard, and the watchdog's retry), counted in UTF-8 bytes; its transcript seats are bounded by the turn's audio. The run sets aside that budget at the prices of the models the configuration can reach, plus the seats: about 85 cents to $1.25 per exact replay for a typical turn, against a real cost of a few cents. A trace that does not confirm the budget is booked at that set-aside; on the dev split the rest of the run replays as text (engine_fallback: "core_did_not_enforce_replay_limits"), and a test run is refused. A turn that needs more than its budget (a second re-run, more retries) has the extra request refused by Core and is an error with error_code: "replay_budget_exhausted", not a verdict, since its reply may not be what a call would say; the summary counts it as budget_exhausted, outside the pass rate. On the test split such a result keeps nothing of the replay and spends no cohort, so a test run whose only results are such turns hands its cohort back unused. A request that costs more than its set-aside stops the run (stop_reason: "cost_over_bound").

Because of the set-aside, a run stops with up to one request's bound per worker unspent. A set-aside is released when its request ends, and each worker replays one example at a time, so a cap only needs the expected spend plus one set-aside per concurrent worker. The dry run's largest_reservation_cents is the largest single request, concurrent_workers how many examples the run replays at once, and suggested_max_cost_cents the cap that lets it finish: estimated_cost_cents plus concurrent_workers times largest_reservation_cents. With exact replays, that is well above what the run will be charged, and a cap below it can stop the run early. upper_bound_cents is the most the run can spend, every request at its bound, and min_max_cost_cents is the smallest cap that starts anything. Replays reuse stored work, so a run usually costs less than its estimate.

Endpoints#

Start a replay run#

curl -X POST https://api.thunderphone.com/v1/agents/123/replay-runs \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"split": "dev", "sample_size": 20, "dry_run": true}'

estimate = requests.post(
    "https://api.thunderphone.com/v1/agents/123/replay-runs",
    headers={"Authorization": "Bearer sk_live_YOUR_API_KEY"},
    json={"split": "dev", "sample_size": 20, "dry_run": True},
).json()
 
run = requests.post(
    "https://api.thunderphone.com/v1/agents/123/replay-runs",
    headers={"Authorization": "Bearer sk_live_YOUR_API_KEY"},
    json={
        "split": "dev",
        "sample_size": 20,
        "sample_seed": estimate["sample_seed"],
        "quote_id": estimate["quote_id"],
        "max_cost_cents": 200,
    },
).json()

Request fields#

Examples are drawn by weight: recent conversations, real calls, conversations that match the newest prompt revisions, and conversations that have been rewritten least come first. Examples from calls whose recording was later turned off are left out, and removed (see Data lifecycle).

Overrides#

An override applies where the agent's own field would, so it passes through the same composition and voice rules a call does. null means the configuration's own value.

Responses#

A dry run returns 200:

{
  "dry_run": true,
  "split": "dev",
  "engine": "auto",
  "phi_mode": "phi",
  "sample_seed": 1802233117,
  "max_cost_cents": null,
  "example_count": 20,
  "configs": [
    { "label": "", "overrides": {}, "engines": { "exact": 14, "text": 6, "skipped": 0 }, "estimated_cost_cents": 128.94, "upper_bound_cents": 1881.97 }
  ],
  "estimated_cost_cents": 128.94,
  "upper_bound_cents": 1881.97,
  "largest_reservation_cents": 103.605,
  "min_max_cost_cents": 104,
  "concurrent_workers": 2,
  "suggested_max_cost_cents": 337,
  "judge_max_completion_tokens": 4000,
  "notes": [
    "Exact replays are estimated with an empty stage cache; the judge at its output bound.",
    "max_cost_cents is a hard cap: each request reserves its upper bound before it is sent and is refused when that does not fit what is left, and the run then stops. A reservation is released when its request ends, and this run replays 2 examples at a time, so a cap only needs the expected spend plus one largest_reservation_cents per concurrent worker: set max_cost_cents to about suggested_max_cost_cents (337), estimated_cost_cents plus 2 times largest_reservation_cents. upper_bound_cents is the most the run can spend.",
    "Both engines run under a token budget Core enforces (replay_limits): …"
  ],
  "quote_id": "eyJvcmciOjQyLC…",
  "quote_expires_in_seconds": 900,
  "next_step": "Show this estimate, then run again with the same request and this sample_seed, without dry_run, with this quote_id and with max_cost_cents."
}

This is the estimator's answer for 14 examples with recorded caller audio and 6 text-only ones, under a Bolt v2 draft with a 10 KB prompt, replayed as PHI: each exact replay sets aside about a dollar (largest_reservation_cents) and costs a few cents, so a cap of estimated_cost_cents plus one set-aside for each of its two workers (suggested_max_cost_cents) lets the run finish; min_max_cost_cents is only the floor.

A test-split dry run adds test_cohort ({"number": 3, "example_count": 41}) and test_cohorts_left.

A real run returns 201 with the run, status pending. A comparison returns 201 with the comparison. Poll until the status is completed or failed.

Replay run object#

GET /v1/replay-runs/{run_id} returns the run with its results. Add include_trace=true for Core's full trace of each exact replay instead of its summary.

{
  "id": 9031,
  "agent_id": 123,
  "label": "shorter callback rule",
  "split": "dev",
  "engine": "auto",
  "config_source": "draft",
  "config_overrides": {},
  "status": "completed",
  "stale": false,
  "degraded": false,
  "stop_reason": null,
  "error_message": null,
  "compare_group_id": null,
  "baseline_run_id": 9020,
  "prompt_revision_id": 4182,
  "prompt_hash": "5b0e…",
  "sample_seed": 1802233117,
  "sample_size": 20,
  "max_cost_cents": 200,
  "estimated_cost_cents": 93.0,
  "cost_cents": 51.6,
  "test_cohort": null,
  "created_at": "2026-10-19T17:10:02Z",
  "started_at": "2026-10-19T17:10:03Z",
  "completed_at": "2026-10-19T17:14:40Z",
  "summary": {
    "total": 20,
    "replayed": 20,
    "phi_mode": "no_phi",
    "verdicts": { "pass": 16, "warn": 2, "fail": 2, "error": 0, "skipped": 0 },
    "budget_exhausted": 0,
    "pass_rate": null,
    "by_engine": {
      "exact": { "verdicts": { "pass": 12, "warn": 1, "fail": 1, "error": 0, "skipped": 0 }, "budget_exhausted": 0, "pass_rate": 0.857 },
      "text": { "verdicts": { "pass": 4, "warn": 1, "fail": 1, "error": 0, "skipped": 0 }, "budget_exhausted": 0, "pass_rate": 0.667 }
    },
    "degraded": false,
    "failures": { "by_severity": { "fail": 2, "warn": 2 }, "by_kind": { "missed_instruction": 3, "two_questions": 1 } },
    "engine_fallbacks": 0,
    "latency_to_first_audio_s": { "p50": 0.94, "p90": 1.61 },
    "latency_to_reply_audio_s": { "p50": 1.42, "p90": 2.37 },
    "wait_cover_share": { "acknowledgement": 0.07, "ticks": 0.0, "silence": 0.93, "filler": 0.0, "none": 0.0 },
    "fork_rate": 0.07,
    "cost_cents": 51.6,
    "cost_per_turn_cents": 2.58
  },
  "flips": { "baseline_run_id": 9020, "by_engine": { "exact": { "compared": 13, "fixed": [5102, 5110], "broken": [5131] }, "text": { "compared": 6, "fixed": [], "broken": [] } }, "not_comparable": 0 },
  "results": [
    {
      "example_id": 5131,
      "verdict": "fail",
      "engine_used": "exact",
      "engine_fallback": null,
      "exact_unavailable": null,
      "skipped_reason": null,
      "error": null,
      "expectation": "Confirm the callback number digit by digit before ending the call.",
      "original_response": { "spoken_response": "Great, we'll call you back soon. Anything else?", "tool_calls": [] },
      "replayed_output": { "spoken_response": "Thanks, we'll call you back. Goodbye.", "tool_calls": [] },
      "instruction_verdict": { "status": "fail", "summary": "The agent ended the call without confirming the callback number.", "issues": [{ "severity": "fail", "type": "missed_instruction", "explanation": "The prompt asks to read the number back before ending the call.", "evidence": "Thanks, we'll call you back. Goodbye." }], "confidence": 0.9 },
      "metrics": { "first_audio_s": 0.91, "reply_audio_s": 1.38, "wait_cover": "silence", "forked": false },
      "cost_cents": 4.5,
      "trace_summary": { "reply": "…", "wait_cover": "…", "end_of_speech": "…" }
    }
  ]
}

A test-split run read with an API key has no flips or results, and has "hidden": true.

GET /v1/agents/{agent_id}/replay-runs returns {"runs": [...]}, the agent's 20 most recent runs without results.

Compare configurations#

Send configs instead of overrides to replay the same examples under 2 to 4 configurations. One run is created per configuration; the first is the baseline, and every other run reports its flips against it. Comparisons run on the dev split only; on the test split they are refused with 400 comparison_dev_only. Check the configuration before blaming the prompt: some failures belong to the setup, such as mishearing spelled letters on a lower audio tier, or a slow reply, or one that cuts the caller off, from the endpointing tier.

{
  "configs": [
    { "label": "standard", "overrides": {} },
    { "label": "hearing", "overrides": { "audio_tier": "hearing" } }
  ],
  "sample_size": 30,
  "sample_seed": 77120394,
  "quote_id": "<quote_id of the dry run of this request>",
  "max_cost_cents": 400
}

The response, and GET /v1/replay-run-groups/{group_id}, is the comparison:

{
  "id": "0f6b4f1e-9a51-4f7e-8a43-1c2a5d3e6b70",
  "agent_id": 123,
  "split": "dev",
  "engine": "auto",
  "config_source": "draft",
  "status": "running",
  "stop_reason": null,
  "sample_seed": 77120394,
  "sample_size": 30,
  "max_cost_cents": 400,
  "cost_cents": 12.4,
  "baseline_run_id": 9040,
  "configs": [
    { "run_id": 9040, "label": "standard", "overrides": {}, "baseline": true, "status": "running", "prompt_revision_id": 4182, "summary": { "pass_rate": null } },
    { "run_id": 9041, "label": "hearing", "overrides": { "audio_tier": "hearing" }, "baseline": false, "status": "pending", "prompt_revision_id": 4182, "summary": { "pass_rate": null }, "flips": null }
  ],
  "created_at": "2026-10-19T17:20:00Z"
}

Each configuration's summary has the fields of a run's summary, so pass rate, latency, wait cover and cost per turn line up side by side. GET /v1/replay-runs/{run_id} for any run of the comparison also returns the comparison as group.

Errors#

Every error body has a detail; most also have a code and a next_step.

MCP#

The MCP server exposes these as run_replays, compare_configs and get_replay_run.

Related#