Developer cookbook

Test an agent end-to-end (API)

Run one-shot simulations, parallel scenario batches, and release-gate suites through the ThunderPhone API so agent regressions are caught before customers hear them.

Iterating on an AI agent means iterating on its prompt, its tools, and the way it handles edge cases. The simulations API runs real calls against an agent using a scenario prompt you supply. Targeting an agent creates a bot-to-bot run; targeting a phone number creates a SIP loopback run. Every run produces a real call log with transcript, grading, and billing, so you see exactly how the agent behaves and what it costs.

Use it for:

• Pre-deploy smoke tests after every prompt edit
• Regression suites wired into CI (hook test-call.completed webhook → fail the build if score drops)
• Stress-testing concurrency limits

One-shot: single run#

curl -X POST https://api.thunderphone.com/v1/simulations \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "target_type":     "agent",
    "target_id":       12,
    "direction":       "outbound",
    "scenario_prompt": "You are a polite caller asking about refund policy for order 12345.",
    "consent_to_charge": true
  }'

Fields:

mode is read-only and derived from target_type: agent produces mode="bot", while phone_number produces mode="sip".

The response is a simulation run object in status="queued". Poll until status becomes completed or failed; once call_id is set, load the transcript via GET /v1/calls/{call_id}/transcript.

Batches: parallel scenarios#

Run N scenarios concurrently — useful for regression suites that hit every known edge case in parallel:

curl -X POST https://api.thunderphone.com/v1/simulations/batches \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "target_type":     "agent",
    "target_id":       12,
    "direction":       "outbound",
    "run_count":       5,
    "stagger_seconds": 2,
    "scenario_prompts": [
      "Ask about refund policy.",
      "Ask for hours of operation.",
      "Complain about a delayed shipment.",
      "Ask to speak with a human.",
      "Ask an unrelated trivia question."
    ],
    "consent_to_charge": true
  }'

Response carries a run_ids list of child run ids. Fetch batch status:

curl https://api.thunderphone.com/v1/simulations/batches/{batch_id} \
  -H "Authorization: Bearer sk_live_YOUR_API_KEY"

run_count is capped at 20; stagger_seconds spaces out the spawn to avoid hammering the agent (0–60 s).

Hook it into CI#

Create a release gate suite on the Simulations page (/dashboard/simulations) — pick the agent, add scenarios by hand or click Generate scenarios with AI to draft them from the agent's prompt (with an optional edge-case pass), and group them into a suite. A suite pins its scenarios and agent, plus a minimum pass rate and an optional zero-critical-failures rule. Passing runs become the accepted baseline; later pass→fail transitions are returned as regressions.

Use an organization API key in CI. This script triggers the suite, polls until grading and comparison are complete, and exits nonzero unless the verdict is pass:

#!/usr/bin/env bash
set -euo pipefail
 
: "${THUNDERPHONE_API_KEY:?Set THUNDERPHONE_API_KEY}"
: "${THUNDERPHONE_ORG_ID:?Set THUNDERPHONE_ORG_ID}"
: "${THUNDERPHONE_SUITE_ID:?Set THUNDERPHONE_SUITE_ID}"
 
base="https://api.thunderphone.com/v1/orgs/${THUNDERPHONE_ORG_ID}/suites/${THUNDERPHONE_SUITE_ID}"
auth="Authorization: Bearer ${THUNDERPHONE_API_KEY}"
 
run_id="$(curl --fail --silent --show-error -X POST "${base}/run" \
  -H "$auth" -H "Content-Type: application/json" -d '{}' | jq -r '.id')"
 
deadline=$((SECONDS + 1800))
while (( SECONDS < deadline )); do
  result="$(curl --fail --silent --show-error \
    "${base}/runs/${run_id}" -H "$auth")"
  status="$(jq -r '.status' <<<"$result")"
  if [[ "$status" == "completed" ]]; then
    jq . <<<"$result"
    [[ "$(jq -r '.verdict' <<<"$result")" == "pass" ]]
    exit
  fi
  sleep 10
done
 
echo "ThunderPhone suite timed out" >&2
exit 1

POST /v1/orgs/{org_id}/suites/{suite_id}/run returns 202 with the run id. GET /v1/orgs/{org_id}/suites/{suite_id}/runs/{run_id} returns status, verdict, pass_rate, critical_failure_count, and the baseline regressions list. Both endpoints bind the URL organization to the API key's organization.

Patterns#

Per-prompt regression corpus#

Maintain a JSON file of {name, scenario_prompt, expected_outcome} tuples. On every prompt change, run the full set as a batch; diff the transcripts and grades against the previous run.

Per-release smoke test#

A single batch of five happy-path scenarios you run after every deploy. Latency-sensitive, so keep stagger_seconds: 0.

Latency benchmarking#

Run identical scenarios against different product tiers (spark, bolt, storm-base). Compare the call.graded scores and the duration_seconds from each resulting call log.

Next steps#