Voice agent observability: catching failures that never throw an error
An AI phone agent almost never fails the way ordinary software fails. The process stays up, the health check stays green, and the call connects — while the conversations themselves quietly get worse. A prompt edit makes the agent verbose, a knowledge update removes the answer to a common question, a tool starts timing out mid-call, or a model change shifts how the agent handles an edge case. None of these raise an exception. They appear, days later, as a drifting success rate, a growing share of early hang-ups, or a slow rise in handle time. Voice agent observability is the discipline of instrumenting the system so an operator sees that drift early and can trace a moving number to its cause.
This guide covers what an observability surface for voice agents needs to show and why. Its two companions cover the adjacent layers: live call monitoring for watching a single conversation as it happens, and AI call quality assurance for judging finished conversations against a standard.
Why silent failure is the default
Conventional monitoring assumes failures announce themselves: a request returns an error, a queue backs up, a process crashes. Voice agents break that assumption because the failing component — the conversation — has no error channel. An agent that misquotes business hours completes the call successfully by every infrastructure measure. An agent that talks over callers still produces a transcript. An agent whose booking tool silently rejects one class of request still says goodbye politely.
The result is that the interesting failures are distributional, not binary. The system does not go from working to broken; the proportion of good outcomes shifts. Detecting a shift requires three things a typical uptime dashboard does not have: a definition of a good outcome per call, a time series of outcome proportions, and a way to move from a suspicious aggregate to the individual calls behind it.
It also requires honesty about what an outcome measure means. A "successful call" judged by an automated grader is a judgment, not a fact, and it inherits the grader's blind spots. Observability tells you that something moved and where; confirming why usually ends in reading transcripts and listening to audio, which is why every number on the surface should link to its underlying calls.
Separate failure classes before averaging anything
The single most damaging design mistake in a voice agent dashboard is one blended failure number. A call can end badly for reasons with completely different owners:
- Infrastructure errors. The call failed to connect, dropped, or hit a platform fault. These are the platform's or the telephony path's responsibility, and their diagnosis runs through signaling and media evidence — see where the milliseconds go for how much machinery sits under one call.
- No conversation. The call connected but the caller disconnected before any real exchange — a wrong number, an immediate hang-up, an answering machine. These often say more about traffic quality or caller expectations than about the agent.
- Unsuccessful conversations. A real conversation happened and the agent did not accomplish what it should have. This is the class that prompt, knowledge, and tool changes actually influence.
- Successful conversations. The agent did its job, as judged by whatever standard the team has defined.
Blending these produces numbers that mislead in both directions. A carrier problem drags down a "success rate" that no prompt edit can fix; a rising share of instant hang-ups masks a genuine conversation-quality regression because the denominator is inflated with calls the agent never had a chance to handle.
The fix is denominators chosen per question. Conversation quality should be computed over conversations — with infrastructure errors excluded from the denominator entirely, and early disconnects reported as their own class rather than counted as agent failures. Infrastructure health should be computed over all call attempts. When one number must move for an operator to act, it should be unambiguous which team acts.
Attribute every number to an agent and a configuration
A fleet average is where regressions go to hide. Most organizations run several agents — different use cases, languages, or phone numbers — and a defect usually ships to one of them. If the surface reports only organization-wide totals, a broken agent handling a fifth of the traffic moves the topline a few points, which reads as noise.
Per-agent attribution turns that into a findable signal: a table of calls, failures, issues, and average outcome per agent makes the outlier visible in one sort. The same logic extends in time — a regression has a start, and a chart bucketed by hour or day lets an operator line the inflection up against a deploy, a prompt edit, or a traffic change. Configuration versioning completes the loop: when the agent's prompt and tools are versioned per deploy, "what changed at 2 p.m. Tuesday" has an answer. That is also the boundary where observability hands off to regression testing — once a cause is confirmed, the failing case should become a permanent test, as covered in how to test voice agents.
Attribution has a second axis: turn type. A latency or failure number blended across tool-calling turns and plain conversational turns hides which path regressed. Where the platform exposes it, split by whether a tool ran, by language, and by call direction.
Drill-down is not optional
Every aggregate on the surface should answer the question "which calls?" A failure count that cannot be expanded into a list of failed calls — each with its end reason and a path into the transcript and recording — is a rumor, not a measurement. In practice, the working loop looks like: notice the moved number, open the recent failures behind it, read three transcripts, and either identify the pattern or escalate to a structured review. The faster that loop runs, the smaller a regression's blast radius.
End reasons deserve their own mention. "Caller hung up", "no answer", and "call failed" are different diagnoses, and a surface that collapses them into "failed" forces operators to rediscover the distinction call by call.
Be honest about sampling
At volume, computing detailed conversation metrics over every call in a window gets expensive, and platforms reasonably compute some measures from a sample of recent calls. Sampling is fine; silent sampling is not. A surface that presents a sampled statistic as if it covered the whole window invites operators to over-read small movements — a few unlucky calls in the sample can look like a trend.
A good surface states when it is sampling and how many calls the sample covers, so the reader can weigh the number accordingly. The same honesty applies to graded outcomes: if success is judged by an automated grader, the surface should say so, and the team should periodically check the grader against human judgment — the calibration problem covered in depth in AI call quality assurance.
Dashboards find causes; alerts find out about problems
A dashboard nobody is looking at detects nothing. The watching should be delegated to alert rules on the same metrics the dashboard shows — success rate below a floor, failure rate above a ceiling, call volume collapsing or spiking — with notifications where the team already works. The dashboard's job is the investigation after the page: scoping to the window and agent, reading the trend, and drilling into calls.
Between threshold alerts and manual review sits a third layer: automatic issue detection, which reads individual conversations at scale and promotes recurring problems into named, triageable patterns. Observability says the success rate dipped; issue detection says fifty calls hit the same missing-answer failure. The layers are complementary, and call analytics sits above both, asking slower business questions of the accumulated data.
On ThunderPhone
ThunderPhone's Observability page (under Quality & Testing in the dashboard) scopes everything by a time window — from the last hour to the last 30 days — and an agent filter. It shows summary cards for calls, success rate (AI-graded, with infrastructure errors excluded from the denominator), unsuccessful calls, early disconnects, errors, in-progress calls, and average duration; volume and success-rate charts bucketed hourly or daily; a per-agent table of calls, failed calls, issues raised, and average grade; and a recent-failed-calls list with end reasons and a one-click path into each call's transcript and recording. On very busy windows, detailed metrics are sampled from the most recent calls, and a notice says how many calls the sample covers. The current window exports as JSON, and alert rules on success rate, failure rate, and call volume deliver to email or Slack.
FAQ
How is observability different from call analytics?
Observability is operational: continuous, near-real-time, and tied to system health — is the fleet working right now, and what changed? Call analytics asks slower business questions of accumulated data: why people call, how outcomes trend by segment, what to staff for. They often read the same underlying records at different tempos.
What metrics should a voice agent dashboard show first?
Call volume, a conversation success rate with infrastructure errors excluded from its denominator, failure counts split by class (error, no conversation, unsuccessful), average duration, and a per-agent breakdown of all of the above — each with a drill-down into the underlying calls.
Why exclude infrastructure errors from the success-rate denominator?
Because they answer a different question with a different owner. A carrier outage should page whoever owns telephony; it should not make the prompt look worse. Blending the classes means every investigation starts by re-separating them manually.
Can automated grading be trusted as the success signal?
It is the only signal that scales to every call, and it is a judgment rather than ground truth. Use it for trends and triage, keep the drill-down path to transcripts and recordings, and calibrate it periodically against human review — the practice covered in AI call quality assurance.