---
title: "Test an agent end-to-end (API)"
description: "Run one-shot simulations, parallel scenario batches, and release-gate suites through the ThunderPhone API so agent regressions are caught before customers hear them."
---

<Note>
  Prefer the dashboard? The same capability lives in **Simulations**
  (`/dashboard/simulations`), including AI scenario generation — see
  [Simulate a call](/guides/simulate-a-call). This page covers the
  programmatic path.
</Note>

Iterating on an AI agent means iterating on its prompt, its tools,
and the way it handles edge cases. The **simulations API** runs real
calls against an agent using a scenario prompt you supply. Targeting
an agent creates a bot-to-bot run; targeting a phone number creates a
SIP loopback run. Every run produces a real call log with
transcript, grading, and billing, so you see exactly how the agent
behaves and what it costs.

Use it for:

- Pre-deploy smoke tests after every prompt edit
- Regression suites wired into CI (hook `test-call.completed` webhook
  → fail the build if score drops)
- Stress-testing concurrency limits

## One-shot: single run

```bash
curl -X POST https://api.thunderphone.com/v1/simulations \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "target_type":     "agent",
    "target_id":       12,
    "direction":       "outbound",
    "scenario_prompt": "You are a polite caller asking about refund policy for order 12345.",
    "consent_to_charge": true
  }'
```

Fields:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `target_type` | string | yes | `agent` or `phone_number` |
| `target_id` | integer | yes | The agent id (or phone number id) |
| `direction` | string | no | `outbound` (default; test caller places) or `inbound` (test caller answers) |
| `scenario_prompt` | string | no | Drives what the test bot says |
| `language` / `primary_language` | string | no | Language for the test caller; unsupported codes are rejected |
| `simulator_product` | string | no | `testing` (default) or `spark` for a more human-like simulated caller, such as warm-transfer consult tests |
| `consent_to_charge` | boolean | **yes** | Must be `true`. The estimate bills both the selected agent and the simulated caller at a flat per-minute rate, plus any telephony leg |
| `target_number` | string | no | E.164 override for the remote side; otherwise the platform test number is used |

`mode` is read-only and derived from `target_type`: `agent` produces
`mode="bot"`, while `phone_number` produces `mode="sip"`.

The response is a [simulation run object](/api-reference/test-calls#test-call-run-object)
in `status="queued"`. Poll until `status` becomes `completed` or
`failed`; once `call_id` is set, load the transcript via
[`GET /v1/calls/{call_id}/transcript`](/api-reference/calls#get-transcript).

## Batches: parallel scenarios

Run N scenarios concurrently — useful for regression suites that
hit every known edge case in parallel:

```bash
curl -X POST https://api.thunderphone.com/v1/simulations/batches \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "target_type":     "agent",
    "target_id":       12,
    "direction":       "outbound",
    "run_count":       5,
    "stagger_seconds": 2,
    "scenario_prompts": [
      "Ask about refund policy.",
      "Ask for hours of operation.",
      "Complain about a delayed shipment.",
      "Ask to speak with a human.",
      "Ask an unrelated trivia question."
    ],
    "consent_to_charge": true
  }'
```

Response carries a `run_ids` list of child run ids. Fetch batch
status:

```bash
curl https://api.thunderphone.com/v1/simulations/batches/{batch_id} \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY"
```

`run_count` is capped at 20; `stagger_seconds` spaces out the spawn
to avoid hammering the agent (0–60 s).

## Hook it into CI

Create a release gate suite on the **Simulations** page
(`/dashboard/simulations`) — pick the agent, add scenarios by hand or
click **Generate scenarios with AI** to draft them from the agent's
prompt (with an optional edge-case pass), and group them into a suite.
A suite pins its scenarios and agent, plus a minimum pass rate and an
optional zero-critical-failures rule. Passing runs become the accepted
baseline; later pass→fail transitions are returned as regressions.

Use an [organization API key](/api-reference/developer-api-keys) in CI.
This script triggers the suite, polls until grading and comparison are
complete, and exits nonzero unless the verdict is `pass`:

```bash
#!/usr/bin/env bash
set -euo pipefail

: "${THUNDERPHONE_API_KEY:?Set THUNDERPHONE_API_KEY}"
: "${THUNDERPHONE_ORG_ID:?Set THUNDERPHONE_ORG_ID}"
: "${THUNDERPHONE_SUITE_ID:?Set THUNDERPHONE_SUITE_ID}"

base="https://api.thunderphone.com/v1/orgs/${THUNDERPHONE_ORG_ID}/suites/${THUNDERPHONE_SUITE_ID}"
auth="Authorization: Bearer ${THUNDERPHONE_API_KEY}"

run_id="$(curl --fail --silent --show-error -X POST "${base}/run" \
  -H "$auth" -H "Content-Type: application/json" -d '{}' | jq -r '.id')"

deadline=$((SECONDS + 1800))
while (( SECONDS < deadline )); do
  result="$(curl --fail --silent --show-error \
    "${base}/runs/${run_id}" -H "$auth")"
  status="$(jq -r '.status' <<<"$result")"
  if [[ "$status" == "completed" ]]; then
    jq . <<<"$result"
    [[ "$(jq -r '.verdict' <<<"$result")" == "pass" ]]
    exit
  fi
  sleep 10
done

echo "ThunderPhone suite timed out" >&2
exit 1
```

`POST /v1/orgs/{org_id}/suites/{suite_id}/run` returns `202` with the
run id. `GET /v1/orgs/{org_id}/suites/{suite_id}/runs/{run_id}` returns
`status`, `verdict`, `pass_rate`, `critical_failure_count`, and the
baseline `regressions` list. Both endpoints bind the URL organization
to the API key's organization.

## Run a suite on a schedule

Open the agent's **Simulate** tab, select **Release gate suites**, and create or
edit a suite. Turn on **Run on a schedule**, choose **Frequency** and
**Timezone**, then set **Minute past the hour**, **Local time**, or **Day** as
shown. Select **Save suite**. Unchecking **Run on a schedule** removes the
dashboard schedule.

### Via API

PATCH the suite to add or replace its schedule. See [Suites (release
gates)](/api-reference/test-scenarios#suites-release-gates) for the complete
suite object and endpoints.

```bash
curl -X PATCH https://api.thunderphone.com/v1/suites/{suite_id} \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "schedule": {
      "enabled": true,
      "frequency": "daily",
      "timezone": "America/Chicago",
      "hour": 6,
      "minute": 30
    }
  }'
```

`frequency` can be `hourly`, `daily`, or `weekly`. Use an IANA timezone.
Hourly schedules use `minute`; daily schedules use `hour` and `minute`;
weekly schedules also use `weekday`, where Monday is `0` and Sunday is `6`.
The suite response includes `next_run_at` and `last_run_at`.

Times follow the selected timezone's daylight-saving changes. Scheduled runs
appear in the suite's run history and use its current agent, scenarios,
criteria, and accepted baseline. Each generated test call emits
`test-call.completed`; there is no suite-level completion webhook. Scheduled
calls are billed at the same simulation rate as manual suite runs and record
`trigger: "schedule"` on the suite run.

To pause a schedule without changing its timing, PATCH the complete existing
schedule object with `"enabled": false`. `frequency` is required; omitted
timezone and time fields reset to their defaults, so include the existing
values. Send `"schedule": null` to remove the schedule.

## Patterns

### Per-prompt regression corpus

Maintain a JSON file of `{name, scenario_prompt, expected_outcome}`
tuples. On every prompt change, run the full set as a batch; diff the
transcripts and grades against the previous run.

### Per-release smoke test

A single batch of five happy-path scenarios you run after every
deploy. Latency-sensitive, so keep `stagger_seconds: 0`.

### Latency benchmarking

Run identical scenarios against different product tiers (`spark`,
`bolt`, `storm-base`). Compare the `call.graded` scores and the
`duration_seconds` from each resulting call log.

---

## Next steps

<CardGroup cols={2}>
  <Card title="Test calls reference" icon="flask" href="/api-reference/test-calls">
    Every query parameter, status code, and batch shape.
  </Card>
  <Card title="AI grading" icon="chart-line" href="/api-reference/calls#ai-call-grading">
    Auto-score every test run to track quality over time.
  </Card>
  <Card title="Issue reports" icon="triangle-exclamation" href="/api-reference/issue-reports">
    Flag specific tests for human review.
  </Card>
  <Card title="test-call.completed webhook" icon="bolt" href="/webhooks/events#test-call-completed">
    Stream results into your CI / Slack / PagerDuty.
  </Card>
</CardGroup>