Voice agent regression testing

Edit one line of a voice agent's prompt and you have changed every call it will take. The instruction you added to fix Tuesday's refund complaint sits in the same context as the rules for disclosures, transfers, and confirmations — and the model re-weighs all of them together on every turn. Regression testing is the discipline that catches the behaviors you did not mean to change: rerun a standing set of scenarios after every edit, compare against an accepted baseline, and block the deploy when something that used to pass now fails.

This guide goes deep on the regression loop specifically: why voice agents regress so easily, how to design a suite, the trade-offs between replaying frozen real-call turns and running live simulations, wiring the gate into CI, and keeping the suite trustworthy over time. For the full pre-launch testing ladder — configuration checks, telephony, limited release — see the broader guide to testing voice agents.

Why a one-line edit changes distant behaviors

Conventional software changes are local: a function edit affects its callers. A voice agent's prompt is global context. Several mechanisms turn a small edit into a distant regression:

• Instruction interaction. Prompts are interpreted as a whole. A new rule ("always offer the callback option") can outcompete an older one ("confirm identity before discussing the account") in situations where both apply, without either being deleted.
• Attention and emphasis shifts. Adding, reordering, or rewording sections changes what the model treats as salient. Behaviors that survived on emphasis rather than explicit instruction quietly decay — see the prompt-engineering guide for why placement carries meaning.
• Tool and knowledge ripple. Renaming a tool argument, tightening a schema, or attaching a new document changes which action looks appropriate several turns downstream of any turn you tested by hand.
• Nondeterminism. The same configuration produces varied wording call to call, so a behavior can degrade from "always" to "usually" — invisible in one manual test call, obvious across a suite.

The consequence: manually re-testing the case you just fixed proves almost nothing. The risk lives in the neighbors — the scenarios you were not thinking about while editing.

Anatomy of a regression suite

A regression suite is a fixed set of scenarios with expectations, run against every candidate configuration and compared to a baseline.

Scenarios describe the caller and the conditions: goal, starting facts, temperament, and the awkward variations that break happy paths — corrections, interruptions, two answers at once, an off-topic detour. Draw them from three sources: the agent's core call types, its riskiest behaviors (mutations, transfers, disclosures), and every material production failure you have seen. A real call that exposed a gap should be promoted into a permanent scenario, not fixed and forgotten.

Expectations must tolerate valid variation. Two passing calls can use different wording and turn counts, so grade meaning, required actions, and outcomes — the booking was made once, the disclosure preceded data collection, the transfer went to the right destination — not an exact script. Distinguish must happen, must never happen, and acceptable alternative outcomes, and keep call scoring evidence attached so a failure points at a turn, not just a number.

Gates turn results into decisions. Two thresholds cover most needs: a minimum pass rate across the suite, and a zero-tolerance rule for critical scenarios where "usually fine" is not fine — compliance disclosures, double-charging, unsafe claims. A useful suite also tracks transitions: a run that passes 90% is read differently when the failures are the same two flaky scenarios as last week versus two scenarios that passed yesterday. Pass→fail transitions against the accepted baseline are the regression signal; the aggregate rate is just the summary.

Treat the suite itself as configuration under change control. Weakening an expectation to get a release through should be reviewed like a code change, because it is one.

Three replay mechanisms, and what each one catches

Regression checks differ in how much of the real system they exercise per unit of cost and time.

Frozen real-call turns (validation sets). Freeze one agent turn from a real call — the conversation up to that point, the caller's audio, what the agent actually did, and the ground truth for what it should do — then replay the frozen context against the new configuration and grade the response. Because input is pinned, a changed response isolates the configuration change; nothing about the caller varied. This is the cheapest and most surgical mechanism: no call is placed, dozens of examples run quickly, and each example encodes exactly one judgment. Its limit is reach — it validates single-turn decisions in frozen contexts, not whether a full conversation still flows, and examples go stale when the intended behavior changes.

Simulated conversations. A call simulation has an AI caller improvise a full conversation within a scenario, producing a normal call log — transcript, recording, grading. Simulations exercise multi-turn state, tool sequencing, and recovery, and they scale to whole suites without human callers. They cost real money where simulations bill as real calls, and they reintroduce variation: the simulated caller phrases things differently run to run, which is realistic but means a failure needs reading before blaming the edit.

Bot-to-bot versus loopback paths. Simulations can connect directly to the agent (bot-to-bot) or dial through a phone number (a SIP loopback), which additionally exercises the telephony leg — routing, audio transport, and number configuration. Loopback runs are the slowest and most complete; use them where the change could plausibly touch the call path, and rely on bot-to-bot for prompt- and tool-level regression volume.

A practical layering, cheapest first: validation-set replay on every draft edit; a bot-to-bot scenario suite before deploy; a small loopback set before releases that touch telephony or on a schedule. Human test calls and limited rollout remain the final layers — covered in the broader testing guide.

Wiring regression into CI

Voice agents deserve the same merge-gate treatment as code:

1. Pin the suite. A release gate should reference a fixed set of scenarios, its target agent, a minimum pass rate, and any zero-critical-failure rule — so the same evidence standard applies to every candidate.
2. Trigger on change. Run the suite from CI whenever prompt, tool, or knowledge configuration changes, using a scoped API credential. Completion webhooks let the pipeline react without polling — the delivery mechanics are covered in how webhooks work for calls.
3. Fail the build on the verdict. The pipeline should read a machine verdict — pass rate, critical failures, regressions versus baseline — and exit nonzero on failure. A human reading transcripts is review, not a gate.
4. Run scenarios in parallel, with care. Batching shortens the loop, but stagger call starts so the suite does not compete with itself for the agent's concurrent capacity.
5. Budget for it. Where simulated calls bill as real calls, a suite run has a real price. That is a feature — it forces suites to stay curated — but the number belongs in the testing budget, not as a surprise.

Record what each run tested: prompt version, tools, knowledge set. A green run against yesterday's draft is not evidence about today's.

Keeping the suite honest

Suites rot in two directions. They go stale — the intended behavior changed, but expectations did not, so the suite blocks a correct improvement or, worse, keeps validating a workflow the business abandoned. And they go soft — flaky scenarios get rerun until green, expectations get loosened under deadline pressure, and the gate stops meaning anything.

Maintenance practices that hold up:

• Re-review frozen examples when the prompt changes. An example whose ground truth predates the current intent is a false alarm generator; platforms that flag examples as stale after prompt changes make this tractable.
• Retire deliberately, never silently. Archive an obsolete scenario with a reason, keeping history; do not delete failures to make a run green.
• Promote every material production failure. The smallest scenario that reproduces the defect enters the suite before the fix ships, and stays.
• Investigate flakes; do not rerun them away. Distinguish legitimate conversational variation from an uncontrolled dependency, and pin the input where possible.
• Separate regression from experimentation. The suite defends known-good behavior; comparing two acceptable behaviors on live traffic is A/B testing, run only after each variant clears the gate.

On ThunderPhone

ThunderPhone implements this loop end to end. Validation sets freeze real agent turns — conversation context, caller audio, and an editable ground-truth expectation — from recent call history; examples are classified as failure, golden, or manual, move through review before activating, and are flagged stale when the prompt changes. Replaying a set against the current draft grades each example pass, warn, or fail, and the run summary reports the pass rate, estimated cost, and any pass→fail regressions since the preceding completed run; the deploy dialog then shows whether the latest run matches the draft being deployed.

For full conversations, simulations put an AI caller on a real call with the agent — bot-to-bot when targeting the agent, SIP loopback when targeting a phone number — producing a normal graded call log. Scenarios can be drafted by AI from the agent's own prompt (with an optional edge-case pass), grouped into suites with a minimum pass rate and an optional zero-critical-failures rule; passing runs become the accepted baseline and later pass→fail transitions are returned as regressions. Suites run from CI via API with a machine-readable verdict, batches run scenarios in parallel with configurable stagger, a completion webhook streams results to pipelines, and a real call's detail page can promote the conversation into a reusable test case. Simulations are billable real calls, and the interface shows the charge before a run.

FAQ

How is regression testing different from general voice-agent testing?

General testing asks whether the agent works at all — configuration, conversation, telephony, integrations, rollout, per the testing guide. Regression testing asks a narrower question after every change: did anything that previously met expectations get worse? It is a standing gate, not a launch project.

Do I need to re-run tests for a "trivial" prompt edit?

Yes. Prompt edits are global by nature — wording and placement shift how every instruction is weighed — and the cheapest mechanisms (replaying frozen turns, a small bot-to-bot suite) make the check inexpensive relative to a client-facing regression.

Replayed turns or simulations — which should I rely on?

Both, at different frequencies. Frozen-turn replay is fast, cheap, and surgical — ideal on every draft edit — but validates single turns. Simulations exercise full conversations and tools — the pre-deploy gate — at real-call cost. Loopback runs add the telephony path when it could be affected.

What pass rate should gate a release?

There is no universal number; it depends on how much valid variation your expectations tolerate and the cost of each failure class. What matters is structure: a written threshold set before the run, a zero-tolerance rule for critical scenarios, and attention to pass→fail transitions against baseline rather than the aggregate alone.

Who should see regression results?

The team that edits the agent, at minimum — and for agencies operating agents on behalf of client businesses, the discipline is what makes client-facing reporting safe: changes clear the gate first, and clients see the resulting record through a client portal rather than discovering regressions on live calls.