Validation sets: replay real call turns before deployment
Turn real ThunderPhone call moments into reusable validation examples, review expected behavior, replay them against an agent draft, and catch regressions before deploy.
A validation example freezes one agent turn from a real call: the conversation leading up to it, the relevant caller audio, what the agent actually did, and the ground truth for what it should do. Replaying a set against the current draft catches regressions without placing another call.
Use the organization-wide Validation sets page
(/dashboard/validation) to build and review the dataset. Use an individual
agent's Validation tab to run the set and inspect run history.
Add examples from call history
- Choose Add examples
Pick the agent whose behavior the example should validate. The workspace shows up to 25 recent non-simulation calls for that agent.
- Select a call and agent turn
Read the transcript, hover the agent turn you want to preserve, and select the checklist action. The conversation through that turn and its caller audio spans are copied into the validation dataset.
- Review the expectation
ThunderPhone drafts a description of correct behavior. Edit it so it is specific and testable, and optionally edit the accepted example response and expected tool calls.
- Save or activate
Save draft leaves the example in Needs review. Save & activate includes it in future validation runs.
Promoting the same turn twice reuses the existing example rather than creating two copies.
Example kinds and statuses
Kinds describe why an example exists:
| Kind | Use |
|---|---|
| Failure | A real response that exposed behavior you want to prevent. |
| Golden | A known-good response worth preserving. |
| Manual | A manually curated case that is neither category. |
Statuses control the lifecycle:
| Status | Meaning |
|---|---|
| Needs review | Ground truth is still being reviewed; excluded from runs. |
| Active | Included in validation runs. |
| Stale | The prompt changed since the example was frozen; review its expectation. |
| Incompatible | The example cannot be replayed with the current configuration. |
| Archived | Preserved for history but excluded from runs. |
Browse and refine the dataset
Filter the organization dataset by agent, status, or kind. Selecting an example shows:
- Frozen conversation context and playable caller audio spans.
- The source call and original response.
- Ground-truth expectation, accepted spoken response, and tool calls.
- The latest replay beside the original and expected behavior.
- The latest verdict and judge explanations, when available.
Choose Edit ground truth to revise the expectation or accepted response. Choose Replay against draft to test only that example without starting a full run. Replays use the agent's current draft, not necessarily its deployed configuration.
Run a validation set
From the agent's Validation tab, start a run over every active example. Each example receives one of these verdicts:
- Pass — the response satisfies the expectation and caller audio.
- Warn — the response is usable but one check found uncertainty.
- Fail — it contradicts the audio or misses required behavior.
- Skipped — required caller audio could not be loaded; it is excluded from the pass-rate denominator.
- Error — replay or grading could not complete.
The run summary reports pass, warning, failure, and skipped counts, the pass rate, estimated cost, and any regression from pass to fail since the preceding completed run. Only one validation run per agent can be active at a time.
Deploy with validation evidence
When an agent has validation examples, the deploy dialog reports whether the latest completed run matches the current draft. You can:
- Deploy with the displayed pass rate and regression count.
- Run validation first if no matching run exists.
- Deliberately deploy anyway when the result has been reviewed.
Validation is evidence and a deployment check, not an automatic production traffic split. For live comparisons, use Experiments (A/B testing).
API automation
The Validation Sets API covers promoting turns, managing examples, streaming pinned audio, individual replays, batch runs, run results, deploy-time status, and the organization overview.