Test an agent end-to-end (API)

Run one-shot simulations, parallel scenario batches, and release-gate suites through the ThunderPhone API so agent regressions are caught before customers hear them.

Iterating on an AI agent means iterating on its prompt, its tools, and the way it handles edge cases. The simulations API runs real calls against an agent using a scenario prompt you supply. Targeting an agent creates a bot-to-bot run; targeting a phone number creates a SIP loopback run. Every run produces a real call log with transcript, grading, and billing, so you see exactly how the agent behaves and what it costs.

Use it for:

• Pre-deploy smoke tests after every prompt edit
• Regression suites wired into CI (hook test-call.completed webhook → fail the build if score drops)
• Stress-testing concurrency limits

One-shot: single run#

curl -X POST https://api.thunderphone.com/v1/simulations \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "target_type":     "agent",
    "target_id":       12,
    "direction":       "outbound",
    "scenario_prompt": "You are a polite caller asking about refund policy for order 12345.",
    "consent_to_charge": true
  }'

Fields:

mode is read-only and derived from target_type: agent produces mode="bot", while phone_number produces mode="sip".

The response is a simulation run object in status="queued". Poll until status becomes completed or failed; once call_id is set, load the transcript via GET /v1/calls/{call_id}/transcript.

Batches: parallel scenarios#

Run N scenarios concurrently — useful for regression suites that hit every known edge case in parallel:

curl -X POST https://api.thunderphone.com/v1/simulations/batches \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "target_type":     "agent",
    "target_id":       12,
    "direction":       "outbound",
    "run_count":       5,
    "stagger_seconds": 2,
    "scenario_prompts": [
      "Ask about refund policy.",
      "Ask for hours of operation.",
      "Complain about a delayed shipment.",
      "Ask to speak with a human.",
      "Ask an unrelated trivia question."
    ],
    "consent_to_charge": true
  }'

Response carries a run_ids list of child run ids. Fetch batch status:

curl https://api.thunderphone.com/v1/simulations/batches/{batch_id} \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY"

run_count is capped at 20; stagger_seconds spaces out the spawn to avoid hammering the agent (0–60 s).

Hook it into CI#

Create a release gate suite on the Simulations page (/dashboard/simulations) — pick the agent, add scenarios by hand or click Generate scenarios with AI to draft them from the agent's prompt (with an optional edge-case pass), and group them into a suite. A suite pins its scenarios and agent, plus a minimum pass rate and an optional zero-critical-failures rule. Passing runs become the accepted baseline; later pass→fail transitions are returned as regressions.

Use an organization API key in CI. This script triggers the suite, polls until grading and comparison are complete, and exits nonzero unless the verdict is pass:

#!/usr/bin/env bash
set -euo pipefail
 
: "${THUNDERPHONE_API_KEY:?Set THUNDERPHONE_API_KEY}"
: "${THUNDERPHONE_ORG_ID:?Set THUNDERPHONE_ORG_ID}"
: "${THUNDERPHONE_SUITE_ID:?Set THUNDERPHONE_SUITE_ID}"
 
base="https://api.thunderphone.com/v1/orgs/${THUNDERPHONE_ORG_ID}/suites/${THUNDERPHONE_SUITE_ID}"
auth="Authorization: Bearer ${THUNDERPHONE_API_KEY}"
 
run_id="$(curl --fail --silent --show-error -X POST "${base}/run" \
  -H "$auth" -H "Content-Type: application/json" -d '{}' | jq -r '.id')"
 
deadline=$((SECONDS + 1800))
while (( SECONDS < deadline )); do
  result="$(curl --fail --silent --show-error \
    "${base}/runs/${run_id}" -H "$auth")"
  status="$(jq -r '.status' <<<"$result")"
  if [[ "$status" == "completed" ]]; then
    jq . <<<"$result"
    [[ "$(jq -r '.verdict' <<<"$result")" == "pass" ]]
    exit
  fi
  sleep 10
done
 
echo "ThunderPhone suite timed out" >&2
exit 1

POST /v1/orgs/{org_id}/suites/{suite_id}/run returns 202 with the run id. GET /v1/orgs/{org_id}/suites/{suite_id}/runs/{run_id} returns status, verdict, pass_rate, critical_failure_count, and the baseline regressions list. Both endpoints bind the URL organization to the API key's organization.

Run a suite on a schedule#

Open the agent's Simulate tab, select Release gate suites, and create or edit a suite. Turn on Run on a schedule, choose Frequency and Timezone, then set Minute past the hour, Local time, or Day as shown. Select Save suite. Unchecking Run on a schedule removes the dashboard schedule.

Via API#

PATCH the suite to add or replace its schedule. See Suites (release gates) for the complete suite object and endpoints.

curl -X PATCH https://api.thunderphone.com/v1/suites/{suite_id} \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "schedule": {
      "enabled": true,
      "frequency": "daily",
      "timezone": "America/Chicago",
      "hour": 6,
      "minute": 30
    }
  }'

frequency can be hourly, daily, or weekly. Use an IANA timezone. Hourly schedules use minute; daily schedules use hour and minute; weekly schedules also use weekday, where Monday is 0 and Sunday is 6. The suite response includes next_run_at and last_run_at.

Times follow the selected timezone's daylight-saving changes. Scheduled runs appear in the suite's run history and use its current agent, scenarios, criteria, and accepted baseline. Each generated test call emits test-call.completed; there is no suite-level completion webhook. Scheduled calls are billed at the same simulation rate as manual suite runs and record trigger: "schedule" on the suite run.

To pause a schedule without changing its timing, PATCH the complete existing schedule object with "enabled": false. frequency is required; omitted timezone and time fields reset to their defaults, so include the existing values. Send "schedule": null to remove the schedule.

Patterns#

Per-prompt regression corpus#

Maintain a JSON file of {name, scenario_prompt, expected_outcome} tuples. On every prompt change, run the full set as a batch; diff the transcripts and grades against the previous run.

Per-release smoke test#

A single batch of five happy-path scenarios you run after every deploy. Latency-sensitive, so keep stagger_seconds: 0.

Latency benchmarking#

Run identical scenarios against different product tiers (spark, bolt, storm-base). Compare the call.graded scores and the duration_seconds from each resulting call log.

Next steps#