Test Scenarios & Suites

Saved regression scenarios, AI scenario generation, condition-matrix runs, and release-gate suites.

Where Simulations are ad-hoc, test scenarios are saved, repeatable test cases for an agent. Scenarios can be written by hand, generated by AI from the agent's prompt, or promoted from a real production call (optionally anchored to an issue report). Executing scenarios produces a test run (a simulation batch under the hood), optionally expanded across a matrix of audio/speech/behavior conditions. Suites bundle scenarios into a pass/fail release gate that can run on a schedule.

Endpoints

Method	Path	Description
`GET` / `POST`	`/v1/agents/{agent_id}/test-scenarios`	List / create scenarios
`GET` / `PATCH` / `DELETE`	`/v1/agents/{agent_id}/test-scenarios/{scenario_id}`	Manage one scenario
`POST`	`/v1/agents/{agent_id}/test-scenarios/generate`	AI-generate scenarios from the agent prompt
`POST`	`/v1/agents/{agent_id}/test-scenarios/refresh`	Re-generate stale scenarios after prompt changes
`POST`	`/v1/calls/{call_id}/promote-to-scenario`	Turn a real call into a scenario
`GET`	`/v1/agents/{agent_id}/test-runs`	List the agent's test runs (scenario batches)
`POST`	`/v1/agents/{agent_id}/test-runs/execute`	Execute scenarios (with optional condition matrix)
`GET`	`/v1/agents/{agent_id}/test-runs/{batch_id}`	Test-run detail with per-scenario results
`GET` / `POST`	`/v1/suites`	List / create suites
`GET` / `PATCH` / `DELETE`	`/v1/suites/{suite_id}`	Manage one suite (UUID id)

Scenario object

{
  "id": 311,
  "agent_id": 12,
  "title": "Caller asks for a refund past the window",
  "scenario_prompt": "You bought the product 45 days ago and demand a refund…",
  "labels": ["refunds", "edge-case"],
  "audio_conditions": { "noise_asset": "none" },
  "speech_style": { "pace": "natural", "disfluencies": "none", "accent": "default" },
  "caller_behavior": { "long_silence_seconds": 0, "dtmf_sequence": "", "topic_change": "" },
  "telephony": { "voicemail_answer": false },
  "direction": "inbound",
  "source": "ai",
  "prompt_hash": "d41d8…",
  "source_call_id": null,
  "source_issue_trace_id": null,
  "source_turn_index": null,
  "latest_verdict": "pass",
  "is_stale": false,
  "created_at": "2026-04-20T18:24:10.113Z",
  "updated_at": "2026-04-20T18:24:10.113Z"
}

Field	Type	Description
`id`	integer	Scenario id
`title`	string	≤ 200 chars
`scenario_prompt`	string	≤ 4000 chars — the role-play brief for the test caller
`labels`	array of string	Up to 10 labels, ≤ 50 chars each
`audio_conditions`	object	`noise_asset` ∈ `none/cafe/street/car/office`; `snr_db` ∈ `0/5/10/20` (required when a noise asset is set)
`speech_style`	object	`pace` ∈ `natural/slow/fast`; `disfluencies` ∈ `none/light`; `accent` ∈ `default/en-US/en-GB/en-IN/es-MX/en-AU/fr-CA`
`caller_behavior`	object	`long_silence_seconds` 0–15; `dtmf_sequence` (≤ 12 of `0-9*#`); `topic_change` (≤ 500 chars)
`telephony`	object	`voicemail_answer` boolean — answer as a voicemail machine
`direction`	string	`inbound` or `outbound`
`source`	string	`manual`, `ai`, or `production` (promoted from a call)
`prompt_hash`	string	Hash of the agent prompt the scenario was generated against
`source_call_id` / `source_issue_trace_id` / `source_turn_index`	mixed \| null	Provenance for promoted scenarios
`latest_verdict`	string	`pass`, `fail`, or `unknown` — from the most recent run
`is_stale`	boolean	True for `ai`/`production` scenarios whose `prompt_hash` no longer matches the agent prompt

Create / update / delete

POST requires title + scenario_prompt (other fields optional, defaults above) and returns 201. PATCH accepts any subset. DELETE returns 204.

AI-generate scenarios

cURL

curl -X POST https://api.thunderphone.com/v1/agents/12/test-scenarios/generate \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"count": 6, "include_edge_cases": true, "locale": "en"}'

Field	Type	Default	Description
`count`	integer	6	1–12 scenarios
`include_edge_cases`	boolean	`true`	Mix in adversarial cases
`locale`	string	`en`	Language of the generated briefs

Returns 201 with {"scenarios": [ … ]} (source: "ai"), or 502 if generation failed.

Refresh stale scenarios

After you change the agent's prompt, AI scenarios go is_stale: true. POST /test-scenarios/refresh with {"scenario_ids": [...]} (≤ 50) or {"only_stale": true} (or an empty body to refresh all) re-generates their briefs against the current prompt and returns {"scenarios": [...]}. 502 on generation failure.

Promote a call to a scenario

Builds one regression scenario from a real call's transcript, grade, and (optionally) a filed issue.

cURL

curl -X POST https://api.thunderphone.com/v1/calls/987654321/promote-to-scenario \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"issue_trace_id": "5f9a…", "failing_turn_index": 7}'

Field	Type	Required	Description
`issue_trace_id`	UUID	no	Anchor the scenario to an issue report on this call
`failing_turn_index`	integer ≥ 0	no	Transcript index of the agent turn that went wrong — focuses generation on a ±3-turn window

Returns 201 with the new scenario (source: "production"). If the same call (+ same issue) was already promoted, returns 200 with the existing scenario instead of creating a duplicate. 400 when the turn index is out of range or not an agent turn.

Test runs (execute scenarios)

cURL

curl -X POST https://api.thunderphone.com/v1/agents/12/test-runs/execute \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "channel": "web",
    "scenario_ids": [311, 312],
    "matrix": {
      "audio_conditions": [{ "noise_asset": "none" }, { "noise_asset": "cafe", "snr_db": 10 }]
    },
    "consent_to_charge": true
  }'

Field	Type	Required	Description
`channel`	string	yes	`web` (browser-style calls against the agent) or `telephony` (real calls through a phone number)
`phone_number_id`	integer	for `telephony`	Which of your numbers to exercise
`scenario_ids`	array of integer	no	≤ 50 — omit to run all of the agent's scenarios
`matrix`	object	no	Condition axes to expand: `audio_conditions` (≤ 5), `speech_style` (≤ 7), `caller_behavior` (≤ 8), `telephony` (≤ 2) — each scenario runs once per combination. Total expanded run count is capped; oversize matrices return `400`
`stagger_seconds`	integer	no	0–60, default 5
`consent_to_charge`	boolean	yes	Must be `true`

Returns 201 with the test-run object. 422 when no scenarios match; 404 for a missing phone number.

Test-run object

Test runs are returned by GET /test-runs (array, limit ≤ 200 default 50, offset) without per-run details, and by GET /test-runs/{batch_id} with a runs array:

{
  "id": 201,
  "agent_id": 12,
  "channel": "web",
  "phone_number_id": null,
  "phone_number_display": null,
  "status": "in_progress",
  "total": 4,
  "completed": 1,
  "failed": 0,
  "pass_count": 1,
  "fail_count": 0,
  "matrix_dimensions": { "audio_conditions": [ … ] },
  "estimated_minutes": 8,
  "estimated_cost_cents": 64,
  "agent_prompt_hash": "d41d8…",
  "error_message": "",
  "requested_by_id": 7,
  "started_at": "2026-04-20T18:24:09.011Z",
  "completed_at": null,
  "created_at": "2026-04-20T18:24:09.011Z",
  "updated_at": "2026-04-20T18:24:09.011Z",
  "runs": [
    {
      "id": 7110,
      "batch_id": 201,
      "scenario_id": 311,
      "scenario_title": "Caller asks for a refund past the window",
      "scenario_prompt": "…",
      "scenario_labels": ["refunds"],
      "condition_dimensions": { "audio_conditions": { "noise_asset": "cafe", "snr_db": 10 } },
      "test_call_run_id": 7110,
      "call_id": 987654321,
      "status": "completed",
      "verdict": "pass",
      "grade_score": 92,
      "duration_seconds": 54,
      "transcript_excerpt": "…",
      "transcript": [ { "role": "user", "content": "…", "timestamp": null } ],
      "grade": { /* grade object */ },
      "recording_url": "https://…",
      "error_message": "",
      "started_at": "2026-04-20T18:24:10.113Z",
      "completed_at": "2026-04-20T18:25:04.822Z"
    }
  ]
}

Per-run verdict is pass / fail / unknown (derived from the grade when the run completes). call_id is the public call id — link it to GET /v1/calls/{call_id}.

Suites (release gates)

A suite is a named set of scenarios for one agent plus pass criteria. Suite runs are recorded against the suite; scheduled runs fire automatically (hourly / daily / weekly), and the dashboard's Simulations page can trigger them on demand.

Suite object

{
  "id": "b4e7c1a9-…",
  "name": "Release gate — support agent",
  "agent_id": 12,
  "agent_name": "Customer Support Agent",
  "scenario_ids": [311, 312],
  "scenarios": [
    { "id": 311, "title": "Caller asks for a refund past the window", "labels": ["refunds"], "is_stale": false }
  ],
  "criteria": { "min_pass_rate": 100, "zero_critical_failures": true },
  "schedule": {
    "enabled": true,
    "frequency": "daily",
    "timezone": "America/Los_Angeles",
    "hour": 6,
    "minute": 0,
    "weekday": 0,
    "next_run_at": "2026-04-21T13:00:00+00:00",
    "last_run_at": null
  },
  "last_green_at": null,
  "created_at": "2026-04-20T18:24:10.113Z",
  "updated_at": "2026-04-20T18:24:10.113Z",
  "runs": [
    {
      "id": "9d2f…",
      "suite_id": "b4e7c1a9-…",
      "batch_id": 201,
      "baseline_run_id": null,
      "trigger": "schedule",
      "status": "running",
      "verdict": "pending",
      "pass_rate": null,
      "critical_failure_count": 0,
      "regression_count": 0,
      "results": [],
      "regressions": [],
      "accepted_at": null,
      "completed_at": null,
      "created_at": "2026-04-20T18:24:10.113Z"
    }
  ]
}

Create / update a suite

cURL

curl -X POST https://api.thunderphone.com/v1/suites \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Release gate — support agent",
    "agent_id": 12,
    "scenario_ids": [311, 312],
    "criteria": { "min_pass_rate": 100, "zero_critical_failures": true },
    "schedule": { "frequency": "daily", "timezone": "America/Los_Angeles", "hour": 6 }
  }'

Field	Type	Required on create	Description
`name`	string	yes	≤ 200 chars
`agent_id`	integer	yes
`scenario_ids`	array of integer	yes	1–50 scenarios; every one must belong to the suite's agent
`criteria.min_pass_rate`	integer	no	0–100, default 100
`criteria.zero_critical_failures`	boolean	no	Default `true`
`schedule`	object \| null	no	`frequency` ∈ `hourly/daily/weekly` (required inside the object), `timezone` (IANA, default UTC), `hour` 0–23, `minute` 0–59, `weekday` 0–6 (weekly). Pass `null` to remove the schedule

POST /v1/suites returns 201; PATCH /v1/suites/{suite_id} accepts any subset; DELETE returns 204. List and detail responses embed the suite's most recent runs (up to 50). A run's verdict becomes pass only when the pass rate meets min_pass_rate and (if enabled) no critical-severity failures occurred; regressions lists scenarios that passed in the baseline run but fail now.

Simulations

The run/batch primitives test runs are built on.

Issue Reports

Promote issues into regression scenarios and track regression_guarded.

Agents

Scenario staleness tracks the agent's prompt.

Calls

Every scenario run produces a call log.