How to test a voice agent before it answers real calls

Test a voice agent as a layered, stateful calling system: verify deterministic rules and tool contracts first, run scripted and adversarial conversations next, exercise the real audio and telephony path, then release to a limited audience with explicit rollback conditions. A transcript that looks reasonable is not sufficient evidence. The agent must also hear the right input, take turns at usable moments, preserve corrected state, call external systems safely, recover from failures, and hand off or stop when it reaches a boundary.

Define what can fail

A voice agent crosses several components during every turn. Audio moves through a phone or browser connection. Speech is detected and transcribed. A conversation policy selects words or an action. External tools may read or mutate business data. Synthesized audio returns to the caller. Events and call records move to downstream systems.

Testing only the response text skips most of that path. A useful failure model includes:

Perception: Did the system capture the caller's audio and interpret important entities correctly?
Timing: Did it wait long enough for the caller, respond without an unexplained pause, and handle interruptions coherently?
Policy: Did it follow scope, disclosure, confirmation, and escalation rules?
State: Did corrections replace old values, and did later turns use the authoritative version?
Actions: Did it select the right tool, send valid arguments, and avoid duplicate side effects?
Telephony: Did dialing, DTMF, voicemail, transfer, hangup, and disconnect behavior work on the intended call path?
Integration: Were call events accepted, authenticated, deduplicated, and processed downstream?
Recovery: When a dependency failed, did the agent make a safe and truthful choice?

Rank failures by consequence and reversibility before writing test cases. A slightly awkward sentence and an unconfirmed cancellation are not equal defects. Release gates should emphasize the latter.

Build a test ladder

The most efficient sequence moves from cheap, isolated checks to complete calls. Each layer answers a different question.

1. Validate configuration and contracts

Check prompt assembly, required variables, tool schemas, destination numbers, routing rules, and knowledge attachments without placing a call. Parse structured configuration and reject missing fields early. Confirm that secrets do not appear in prompts, logs, or tool descriptions.

For each tool, validate required arguments, allowed values, authorization rules, timeouts, and error mapping. Mutating tools should accept an idempotency key or enforce equivalent deduplication in application logic. These checks catch errors that a conversational simulation might encounter only rarely.

2. Unit-test deterministic policy code

If application code decides whether a field is valid, a call may transfer, or an action requires confirmation, test those branches directly. Keep deterministic business rules out of subjective transcript grading where possible.

Useful examples include date constraints, office-hour routing, retry limits, consent state, escalation eligibility, and the mapping from tool errors to recovery choices. The expected result should be exact: allowed or denied, destination A or B, retry or stop.

3. Run scenario conversations

A call simulation exercises multi-turn behavior without asking a human to repeat the same script. Define the caller's goal, starting facts, behavioral variations, dependency responses, and assertions. The scenario should specify what must happen, what must never happen, and which outcomes are acceptable alternatives.

For example, a rescheduling scenario could require the agent to identify the existing booking, ask for a new time, check availability, confirm the change, and update the booking once. It could forbid claiming success before the update response and permit a handoff if the lookup fails twice.

Do not make every simulated caller cooperative. Add callers who answer two questions at once, change their mind, correct a name, ask an unrelated question, remain silent, or interrupt a confirmation. Variation exposes whether the policy understands the task or merely follows the happy-path wording.

4. Test with a human microphone

Human testing reveals timing defects that text-based scenarios cannot. Ask testers to use ordinary speech rather than reading polished scripts. They should pause, restart sentences, use filler words, speak over the agent, and call from a noisy but realistic environment.

Record observations at the event level. “The conversation felt slow” is a clue; “the agent waited after the final transcript before initiating the availability lookup” is actionable evidence. Mark when audio began and ended, when the turn was finalized, when a tool started and returned, and when response audio began.

5. Exercise the telephony path

A browser test proves the conversational stack can use a microphone. It does not prove the production phone path. Place end-to-end calls through the intended ingress and egress configuration, using numbers and destinations you are authorized to test.

Cover answer, busy, rejection, no answer, voicemail, early hangup, DTMF, transfer acceptance, transfer failure, and carrier disconnect. If the deployment uses SIP, inspect signaling as well as audio: final response codes, dialog establishment, re-INVITEs when applicable, RTP flow, and BYE handling. A 200 OK response to an INVITE does not prove that both parties received intelligible audio.

Network impairment tests can introduce delay, jitter, reordering, and packet loss in a controlled environment. The goal is not to make every degraded call sound perfect. It is to confirm that the system fails legibly, avoids unsafe actions based on partial input, and terminates or recovers according to policy.

6. Release narrowly and observe

After preproduction tests pass, expose the agent to a bounded, low-risk call path: internal callers, a dedicated test number, or a deliberately limited traffic segment. Define who can stop the rollout and what evidence triggers that decision.

Do not use early production traffic to discover whether cancellation, emergency escalation, or access control works. Those paths need direct testing first. The limited release is for distribution shift: phrasing, environments, and caller goals that the designed scenarios did not capture.

Write scenarios as testable contracts

A scenario needs more than a sample transcript. The exact dialogue is allowed to vary, while the business invariants should remain stable.

A practical scenario record contains:

name: caller corrects appointment date before confirmation
starting state: customer identity verified; no date selected
caller behavior: says Thursday, then corrects it to Tuesday
dependency fixtures: Tuesday available; Thursday available
must happen: final confirmation says Tuesday; one booking is created
must not happen: Thursday booking; two mutation requests
acceptable recovery: human handoff if the correction cannot be resolved

Use fixtures for external systems so failures are repeatable. Return success, validation errors, empty results, slow responses, timeouts, and unknown outcomes deliberately. A suite that depends on a live calendar or CRM can become nondeterministic for reasons unrelated to the agent.

Keep a separate small set of true end-to-end tests against sandbox or test accounts. Contract fixtures prove policy behavior; live integrations prove that authentication, schemas, and network boundaries still match reality.

Grade facts and behavior separately

Automated graders are useful, but one overall “good call” score hides the failure mode. Split evaluation into assertions that can be measured directly and judgments that need a rubric.

Direct assertions include:

the required disclosure occurred before collecting specified information;
a tool was or was not invoked;
arguments matched the caller's corrected values;
a mutation happened once;
the call reached a permitted final disposition;
the agent did not state success after an error response.

Rubric judgments include whether the agent was concise, whether a clarification was understandable, and whether the handoff summary was useful. Define each rubric level with observable behavior. Avoid asking a grader whether the call “felt professional” without saying what that means.

Call scoring should preserve the supporting turn or event for every failure. A number without evidence is difficult to debug and easy to optimize in the wrong direction.

Test the audio behaviors that change meaning

Voice-specific cases are not cosmetic. They can change the requested action.

Interruption: Start speaking during a long response and during a critical confirmation. The expected behavior may differ.
Endpointing: Pause mid-sentence, then continue. Check whether the system steals the turn or waits indefinitely. The engineering details are explained in turn-taking and interruptions.
Corrections: Replace a date, quantity, name, or destination after it has entered state.
Spelling and digits: Exercise similar-sounding letters, repeated digits, extensions, and leading zeros where the use case permits them.
Background speech: Confirm that a nearby voice does not silently authorize an action.
Silence: Test silence before an answer, during dependency work, and after a question.
Language changes: If multiple languages are configured, switch at predictable and awkward points in the call.
Long turns: Give extra context before the actual request and verify that the key intent is retained.

Use synthetic audio for repeatability and human callers for natural variation. Neither replaces the other.

Verify tools, events, and downstream state

The call is not complete just because the farewell sounded correct. Compare the transcript and tool trace with the authoritative downstream record. If a booking was changed, verify the booking. If a lead was routed, verify its owner and disposition.

Test delayed, duplicated, and out-of-order event delivery. Webhook consumers should acknowledge quickly, queue work, deduplicate by event identifier, and make their own processing idempotent. The guide to call webhooks covers the delivery model in detail.

Also test partial success. The call action may succeed while the analytics event fails, or the event may arrive while an enrichment step is unavailable. Observability should distinguish the customer-facing outcome from secondary processing.

Turn failures into regression coverage

Every material defect found in review or limited release should produce the smallest scenario that reproduces it. Add that case to regression testing, fix the responsible layer, and rerun the relevant suite.

Store the versions of the prompt, tools, knowledge set, configuration, and runtime alongside the result. Otherwise the same scenario name can silently exercise a different system next week. Keep pass criteria under change control, too; weakening an assertion should be reviewed like a code change.

Use A/B testing only after each variant independently clears safety and correctness gates. An experiment can compare two acceptable behaviors. It should not decide whether an unsafe behavior is permissible because it improves a business metric.

On ThunderPhone

ThunderPhone documents browser microphone tests, AI-caller simulations, bot-to-bot and SIP-loopback calls, reusable scenarios, graded call logs, regression suites with minimum pass-rate gates, CI execution, and live-traffic experiments. Simulations are billable real calls, and the interface shows the charge before a run.

A practical release gate

Before connecting a production number, require evidence that high-risk scenarios pass, required tool and policy assertions hold, the real phone path carries two-way audio, transfer and failure outcomes are understood, and a human handoff remains available where policy requires it. Name the person who can pause the release and preserve the traces needed to reproduce a failure.

The exact threshold is a business decision, but it should be written before the final test run. Moving a gate after seeing a failure turns a release criterion into a negotiation.

FAQ

How many scenarios does a voice agent need?

There is no universal count. Start from distinct risks, states, tools, and failure modes. Add variations where audio or caller behavior could change the outcome, then keep material production failures as permanent regression cases.

Can text transcripts replace test calls?

No. They are useful for policy coverage, but they omit speech recognition, timing, synthesis, interruption, audio transport, SIP behavior, and carrier outcomes. At least some tests must traverse the same call path as production.

Should test calls write to real business systems?

Most scenario tests should use deterministic fixtures or dedicated test accounts. Maintain a small end-to-end set against sandbox systems to catch authentication and schema drift. Never let a routine test mutate uncontrolled production records.

What should happen when a test is flaky?

First identify whether the variability is an expected property of audio and conversation or an uncontrolled dependency. Preserve the trace, isolate the layer, and make external responses deterministic where possible. Do not simply rerun until it passes.