--- title: "Test Scenarios & Suites" description: "Saved regression scenarios, AI scenario generation, condition-matrix runs, and release-gate suites." --- Where [Simulations](/api-reference/test-calls) are ad-hoc, **test scenarios** are saved, repeatable test cases for an agent. Scenarios can be written by hand, generated by AI from the agent's prompt, or **promoted from a real production call** (optionally anchored to an issue report). Executing scenarios produces a **test run** (a [simulation batch](/api-reference/test-calls#batch-object) under the hood), optionally expanded across a matrix of audio/speech/behavior conditions. **Suites** bundle scenarios into a pass/fail release gate that can run on a schedule. ## Endpoints | Method | Path | Description | |--------|------|-------------| | `GET` / `POST` | `/v1/agents/{agent_id}/test-scenarios` | List / create scenarios | | `GET` / `PATCH` / `DELETE` | `/v1/agents/{agent_id}/test-scenarios/{scenario_id}` | Manage one scenario | | `POST` | `/v1/agents/{agent_id}/test-scenarios/generate` | AI-generate scenarios from the agent prompt | | `POST` | `/v1/agents/{agent_id}/test-scenarios/refresh` | Re-generate stale scenarios after prompt changes | | `POST` | `/v1/calls/{call_id}/promote-to-scenario` | Turn a real call into a scenario | | `GET` | `/v1/agents/{agent_id}/test-runs` | List the agent's test runs (scenario batches) | | `POST` | `/v1/agents/{agent_id}/test-runs/execute` | Execute scenarios (with optional condition matrix) | | `GET` | `/v1/agents/{agent_id}/test-runs/{batch_id}` | Test-run detail with per-scenario results | | `GET` / `POST` | `/v1/suites` | List / create suites | | `GET` / `PATCH` / `DELETE` | `/v1/suites/{suite_id}` | Manage one suite (UUID id) | ## Scenario object ```json { "id": 311, "agent_id": 12, "title": "Caller asks for a refund past the window", "scenario_prompt": "You bought the product 45 days ago and demand a refund…", "labels": ["refunds", "edge-case"], "audio_conditions": { "noise_asset": "none" }, "speech_style": { "pace": "natural", "disfluencies": "none", "accent": "default" }, "caller_behavior": { "long_silence_seconds": 0, "dtmf_sequence": "", "topic_change": "" }, "telephony": { "voicemail_answer": false }, "direction": "inbound", "source": "ai", "prompt_hash": "d41d8…", "source_call_id": null, "source_issue_trace_id": null, "source_turn_index": null, "latest_verdict": "pass", "is_stale": false, "created_at": "2026-04-20T18:24:10.113Z", "updated_at": "2026-04-20T18:24:10.113Z" } ``` | Field | Type | Description | |-------|------|-------------| | `id` | integer | Scenario id | | `title` | string | ≤ 200 chars | | `scenario_prompt` | string | ≤ 4000 chars — the role-play brief for the test caller | | `labels` | array of string | Up to 10 labels, ≤ 50 chars each | | `audio_conditions` | object | `noise_asset` ∈ `none/cafe/street/car/office`; `snr_db` ∈ `0/5/10/20` (required when a noise asset is set) | | `speech_style` | object | `pace` ∈ `natural/slow/fast`; `disfluencies` ∈ `none/light`; `accent` ∈ `default/en-US/en-GB/en-IN/es-MX/en-AU/fr-CA` | | `caller_behavior` | object | `long_silence_seconds` 0–15; `dtmf_sequence` (≤ 12 of `0-9*#`); `topic_change` (≤ 500 chars) | | `telephony` | object | `voicemail_answer` boolean — answer as a voicemail machine | | `direction` | string | `inbound` or `outbound` | | `source` | string | `manual`, `ai`, or `production` (promoted from a call) | | `prompt_hash` | string | Hash of the agent prompt the scenario was generated against | | `source_call_id` / `source_issue_trace_id` / `source_turn_index` | mixed \| null | Provenance for promoted scenarios | | `latest_verdict` | string | `pass`, `fail`, or `unknown` — from the most recent run | | `is_stale` | boolean | True for `ai`/`production` scenarios whose `prompt_hash` no longer matches the agent prompt | ### Create / update / delete `POST` requires `title` + `scenario_prompt` (other fields optional, defaults above) and returns `201`. `PATCH` accepts any subset. `DELETE` returns `204`. ### AI-generate scenarios ```bash cURL curl -X POST https://api.thunderphone.com/v1/agents/12/test-scenarios/generate \ -H "Authorization: Bearer sk_live_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"count": 6, "include_edge_cases": true, "locale": "en"}' ``` | Field | Type | Default | Description | |-------|------|---------|-------------| | `count` | integer | 6 | 1–12 scenarios | | `include_edge_cases` | boolean | `true` | Mix in adversarial cases | | `locale` | string | `en` | Language of the generated briefs | Returns `201` with `{"scenarios": [ … ]}` (`source: "ai"`), or `502` if generation failed. ### Refresh stale scenarios After you change the agent's prompt, AI scenarios go `is_stale: true`. `POST /test-scenarios/refresh` with `{"scenario_ids": [...]}` (≤ 50) or `{"only_stale": true}` (or an empty body to refresh **all**) re-generates their briefs against the current prompt and returns `{"scenarios": [...]}`. `502` on generation failure. ### Promote a call to a scenario Builds one regression scenario from a real call's transcript, grade, and (optionally) a filed issue. ```bash cURL curl -X POST https://api.thunderphone.com/v1/calls/987654321/promote-to-scenario \ -H "Authorization: Bearer sk_live_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"issue_trace_id": "5f9a…", "failing_turn_index": 7}' ``` | Field | Type | Required | Description | |-------|------|----------|-------------| | `issue_trace_id` | UUID | no | Anchor the scenario to an [issue report](/api-reference/issue-reports) on this call | | `failing_turn_index` | integer ≥ 0 | no | Transcript index of the **agent** turn that went wrong — focuses generation on a ±3-turn window | Returns `201` with the new scenario (`source: "production"`). If the same call (+ same issue) was already promoted, returns `200` with the existing scenario instead of creating a duplicate. `400` when the turn index is out of range or not an agent turn. --- ## Test runs (execute scenarios) ```bash cURL curl -X POST https://api.thunderphone.com/v1/agents/12/test-runs/execute \ -H "Authorization: Bearer sk_live_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "channel": "web", "scenario_ids": [311, 312], "matrix": { "audio_conditions": [{ "noise_asset": "none" }, { "noise_asset": "cafe", "snr_db": 10 }] }, "consent_to_charge": true }' ``` | Field | Type | Required | Description | |-------|------|----------|-------------| | `channel` | string | yes | `web` (browser-style calls against the agent) or `telephony` (real calls through a phone number) | | `phone_number_id` | integer | for `telephony` | Which of your numbers to exercise | | `scenario_ids` | array of integer | no | ≤ 50 — omit to run **all** of the agent's scenarios | | `matrix` | object | no | Condition axes to expand: `audio_conditions` (≤ 5), `speech_style` (≤ 7), `caller_behavior` (≤ 8), `telephony` (≤ 2) — each scenario runs once per combination. Total expanded run count is capped; oversize matrices return `400` | | `stagger_seconds` | integer | no | 0–60, default 5 | | `consent_to_charge` | boolean | yes | Must be `true` | Returns `201` with the [test-run object](#test-run-object). `422` when no scenarios match; `404` for a missing phone number. ### Test-run object Test runs are returned by `GET /test-runs` (array, `limit` ≤ 200 default 50, `offset`) without per-run details, and by `GET /test-runs/{batch_id}` with a `runs` array: ```json { "id": 201, "agent_id": 12, "channel": "web", "phone_number_id": null, "phone_number_display": null, "status": "in_progress", "total": 4, "completed": 1, "failed": 0, "pass_count": 1, "fail_count": 0, "matrix_dimensions": { "audio_conditions": [ … ] }, "estimated_minutes": 8, "estimated_cost_cents": 64, "agent_prompt_hash": "d41d8…", "error_message": "", "requested_by_id": 7, "started_at": "2026-04-20T18:24:09.011Z", "completed_at": null, "created_at": "2026-04-20T18:24:09.011Z", "updated_at": "2026-04-20T18:24:09.011Z", "runs": [ { "id": 7110, "batch_id": 201, "scenario_id": 311, "scenario_title": "Caller asks for a refund past the window", "scenario_prompt": "…", "scenario_labels": ["refunds"], "condition_dimensions": { "audio_conditions": { "noise_asset": "cafe", "snr_db": 10 } }, "test_call_run_id": 7110, "call_id": 987654321, "status": "completed", "verdict": "pass", "grade_score": 92, "duration_seconds": 54, "transcript_excerpt": "…", "transcript": [ { "role": "user", "content": "…", "timestamp": null } ], "grade": { /* grade object */ }, "recording_url": "https://…", "error_message": "", "started_at": "2026-04-20T18:24:10.113Z", "completed_at": "2026-04-20T18:25:04.822Z" } ] } ``` Per-run `verdict` is `pass` / `fail` / `unknown` (derived from the grade when the run completes). `call_id` is the public call id — link it to [`GET /v1/calls/{call_id}`](/api-reference/calls). --- ## Suites (release gates) A **suite** is a named set of scenarios for one agent plus pass criteria. Suite runs are recorded against the suite; scheduled runs fire automatically (hourly / daily / weekly), and the dashboard's Simulations page can trigger them on demand. ### Suite object ```json { "id": "b4e7c1a9-…", "name": "Release gate — support agent", "agent_id": 12, "agent_name": "Customer Support Agent", "scenario_ids": [311, 312], "scenarios": [ { "id": 311, "title": "Caller asks for a refund past the window", "labels": ["refunds"], "is_stale": false } ], "criteria": { "min_pass_rate": 100, "zero_critical_failures": true }, "schedule": { "enabled": true, "frequency": "daily", "timezone": "America/Los_Angeles", "hour": 6, "minute": 0, "weekday": 0, "next_run_at": "2026-04-21T13:00:00+00:00", "last_run_at": null }, "last_green_at": null, "created_at": "2026-04-20T18:24:10.113Z", "updated_at": "2026-04-20T18:24:10.113Z", "runs": [ { "id": "9d2f…", "suite_id": "b4e7c1a9-…", "batch_id": 201, "baseline_run_id": null, "trigger": "schedule", "status": "running", "verdict": "pending", "pass_rate": null, "critical_failure_count": 0, "regression_count": 0, "error_message": "", "results": [], "regressions": [], "accepted_at": null, "completed_at": null, "created_at": "2026-04-20T18:24:10.113Z" } ] } ``` ### Create / update a suite ```bash cURL curl -X POST https://api.thunderphone.com/v1/suites \ -H "Authorization: Bearer sk_live_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Release gate — support agent", "agent_id": 12, "scenario_ids": [311, 312], "criteria": { "min_pass_rate": 100, "zero_critical_failures": true }, "schedule": { "frequency": "daily", "timezone": "America/Los_Angeles", "hour": 6 } }' ``` | Field | Type | Required on create | Description | |-------|------|--------------------|-------------| | `name` | string | yes | ≤ 200 chars | | `agent_id` | integer | yes | | | `scenario_ids` | array of integer | yes | 1–50 scenarios; every one must belong to the suite's agent | | `criteria.min_pass_rate` | integer | no | 0–100, default 100 | | `criteria.zero_critical_failures` | boolean | no | Default `true` | | `schedule` | object \| null | no | `frequency` ∈ `hourly/daily/weekly` (required inside the object), `timezone` (IANA, default UTC), `hour` 0–23, `minute` 0–59, `weekday` 0–6 (weekly). Pass `null` to remove the schedule | `POST /v1/suites` returns `201`; `PATCH /v1/suites/{suite_id}` accepts any subset; `DELETE` returns `204`. List and detail responses embed the suite's most recent runs (up to 50). A run's `verdict` becomes `pass` only when the pass rate meets `min_pass_rate` and (if enabled) no critical-severity failures occurred; `regressions` lists scenarios that passed in the baseline run but fail now. Suite runs include an `error_message` string with the batch-level dispatch failure reason; it is not nullable, and an empty string means there was no dispatch failure. Each object in `results` also includes a non-nullable `error_message` with that scenario's dispatch failure reason, or an empty string when no dispatch failure was recorded. --- ## Related The run/batch primitives test runs are built on. Promote issues into regression scenarios and track `regression_guarded`. Scenario staleness tracks the agent's prompt. Every scenario run produces a call log.