Calls

Validation Sets

Frozen real-call turns replayed against your draft configuration, graded automatically before you deploy.

Where Test Scenarios simulate whole calls end-to-end, a validation set is a collection of frozen single turns from real calls. Each validation example pins the exact conversation history and caller audio of one agent turn, plus an expectation of how the agent should respond. Running the set replays every example against the agent's current draft configuration and grades each response with two automated judges — one listens to the pinned caller audio for contradictions, the other checks the response against the agent's prompt and the example's expectation. The result is a pass rate and a list of regressions you can review before deploying.

Endpoints#

Example object#

{
  "id": 42,
  "agent_id": 12,
  "kind": "failure",
  "status": "pending_review",
  "labels": ["refunds"],
  "expectation": "Confirms the refund window and offers to escalate.",
  "accepted_responses": [{ "spoken_response": "…", "tool_calls": [] }],
  "original_response": { "spoken_response": "…", "tool_calls": [] },
  "target_turn_index": 7,
  "source_call_id": 903212880123,
  "source_issue_id": null,
  "prompt_hash": "d41d8…",
  "is_stale_prompt": false,
  "audio_span_indices": [3, 5, 7],
  "expectation_proposal_error": null,
  "last_verdict": "pass",
  "last_run_id": 9,
  "context": [ { "role": "user", "content": "…" } ],
  "created_at": "2026-07-22T18:24:10.113Z",
  "updated_at": "2026-07-22T18:24:10.113Z"
}

Promote a call turn#

POST /v1/calls/{call_id}/validation-examples with turn_index (the agent turn to freeze), kind, and optionally source_issue_id. Returns the created example with 201. The turn's history and caller audio are snapshotted immediately, so the example survives the source call being deleted. Promoting the same turn twice returns 409 with existing_id.

List examples#

GET /v1/agents/{agent_id}/validation-examples returns { "examples": [...] } (without context).

GET /v1/validation-examples returns the same shape across every agent in the organization, newest first, with agent_name added to each row. Optional query filters: agent_id, status, kind.

Manage one example#

GET returns the full example. PATCH accepts any subset of expectation, accepted_responses, labels, kind, and status (allowed transitions: pending_review → active|archived, active → archived, archived → active). DELETE removes the example and its pinned audio.

Caller audio#

GET /v1/validation-examples/{example_id}/audio/{span_index} streams the pinned WAV for one of the example's audio_span_indices.

Replay one example#

POST /v1/validation-examples/{example_id}/replay (empty body) replays the frozen turn against the agent's current draft configuration and returns:

{
  "replayed_output": { "spoken_response": "…", "tool_calls": [] },
  "replay_metadata": { "audio_spans_expected": 3, "audio_spans_loaded": 3, "skipped": false },
  "raw_prompt_hash": "ab12…",
  "cached": false
}

If any pinned audio span fails to load, skipped is true and the result is never graded or counted. Identical replays (same draft config) are served from cache with cached: true.

Validation runs#

POST /v1/agents/{agent_id}/validation-runs starts a batch run over the agent's active examples (body is optional; config_source: "draft" is the only supported value). Returns the run with 201. One run per agent may be in flight (409 with existing_id otherwise), and concurrent runs per organization are capped (429). Starting a run with no active examples returns 400.

{
  "id": 9,
  "agent_id": 12,
  "config_source": "draft",
  "status": "completed",
  "prompt_hash": "ab12…",
  "pass_count": 11,
  "warn_count": 1,
  "fail_count": 2,
  "skipped_count": 1,
  "total_count": 15,
  "pass_rate": 0.786,
  "regressions": [{ "example_id": 42, "previous": "pass", "current": "fail" }],
  "estimated_cost_cents": 8,
  "error_message": null,
  "created_at": "2026-07-22T18:24:10.113Z",
  "started_at": "2026-07-22T18:24:11.001Z",
  "completed_at": "2026-07-22T18:26:44.913Z",
  "triggered_by_name": "Alec Bell"
}

status is pending, running, completed, or failed. pass_rate excludes skipped examples from the denominator. regressions lists examples that passed in the previous completed run but failed in this one.

Run detail#

GET /v1/validation-runs/{run_id} returns the run plus results, one per graded example:

{
  "results": [
    {
      "id": 71,
      "example_id": 42,
      "example_expectation": "Confirms the refund window…",
      "example_kind": "failure",
      "verdict": "fail",
      "replayed_output": { "spoken_response": "…", "tool_calls": [] },
      "replay_metadata": { "audio_spans_expected": 3, "audio_spans_loaded": 3, "skipped": false },
      "audio_verdict": { "contradiction": true, "explanation": "…" },
      "instruction_verdict": { "followed": false, "explanation": "…" },
      "created_at": "2026-07-22T18:25:02.113Z"
    }
  ]
}

verdict combines both judges: pass, warn, fail, or skipped (audio unavailable). Each judge verdict includes an explanation you can surface to reviewers.

List runs#

GET /v1/agents/{agent_id}/validation-runs returns { "runs": [...] } — the 20 most recent runs, newest first.

Validation status#

GET /v1/agents/{agent_id}/validation-status powers the deploy-confirmation check:

{
  "active_example_count": 15,
  "latest_run": { "…": "run object or null" },
  "latest_run_matches_draft": true
}

latest_run_matches_draft is true when the latest completed run was executed against the current draft prompt — i.e. the pass rate on screen reflects what you are about to deploy.

Org overview#

GET /v1/validation-sets returns one row per agent in the organization that has validation examples, for a cross-agent dashboard view:

{
  "agents": [
    {
      "agent_id": 12,
      "agent_name": "Support line",
      "active_count": 15,
      "pending_count": 2,
      "archived_count": 1,
      "latest_run": { "…": "run object or null" },
      "latest_run_matches_draft": true
    }
  ]
}

Agents without any validation examples are omitted. Rows are sorted by agent name.