How voice agents ground answers: knowledge bases and RAG on calls

Voice agents ground answers by retrieving relevant material from an approved knowledge source during the call and conditioning each response on that evidence, a pattern known as retrieval-augmented generation, or RAG. Documents are parsed, divided into searchable units, indexed, and filtered by scope; at question time, the system turns the caller's request into a search query, retrieves and ranks candidate passages, and gives the response policy only the evidence it needs. Grounding reduces unsupported answers, but it does not guarantee correctness: source quality, retrieval quality, authorization, and the instruction to abstain all remain part of the system.

Grounding is more than attaching documents

A knowledge base is the managed collection of source material. RAG is the runtime process that selects material from that collection and supplies it to the response generator. Grounding is the broader property that an answer is tied to approved evidence rather than produced from general patterns alone.

Uploading a handbook creates a source. It does not prove that a caller's question retrieves the right section, excludes an older handbook, or faithfully represents the passage.

The complete system has an offline path and an online path:

Offline: documents -> parse -> normalize -> chunk -> enrich -> index
Online: caller turn -> query -> retrieve -> filter -> rerank -> context -> answer

Errors at different stages look similar to the caller. “I don't know” may mean the answer is absent, parsing failed, the query was poor, the correct chunk ranked too low, or the answer policy declined to use good evidence. Debugging requires preserving each stage.

Ingestion creates the retrieval units

Documents arrive in forms designed for people: headings, paragraphs, lists, tables, columns, footnotes, and page furniture. Search works on a normalized representation. An ingestion pipeline extracts text and structure, removes repeated headers or navigation, preserves useful section boundaries, and creates chunks small enough to retrieve selectively.

Chunk size is a trade-off. A large chunk carries context but may combine topics and reduce ranking precision. A small chunk may retrieve the exact sentence while dropping the exception or heading that gives it meaning.

Semantic boundaries are usually better than arbitrary character cuts. Keep a policy and its exceptions together. Keep a table's row labels attached to its values. Include the document title and heading path as metadata even when the body chunk is short. Limited overlap can preserve context across a boundary, but heavy overlap fills results with near-duplicates.

Parsing quality matters especially for PDFs and scans. Visual and stored reading order may differ, columns can interleave, and scans may require optical character recognition. The pipeline should expose failures rather than mark unreadable files as ready.

Each indexed unit should carry provenance such as:

• document identifier and version;
• section title or heading path;
• effective and expiration dates where applicable;
• audience, region, product, or language scope;
• access-control tags;
• source location for review and citation.

Metadata lets the online path exclude obsolete or unauthorized material.

Retrieval starts with the caller's meaning

The latest transcript is not always a good search query. On a call, a person may ask, “What about the larger one?” The meaning depends on prior turns: perhaps two service plans or two replacement parts were just discussed. The query builder must combine the current utterance with the minimum conversational context needed to resolve references.

It should also account for speech-recognition errors. The pipeline described in speech recognition on calls may produce a plausible but incorrect term, especially for names and codes. Known entities from business state can help normalize that query without rewriting the caller's intent.

Retrieval commonly combines two signals:

• Lexical search finds exact or near-exact words, identifiers, and phrases.
• Semantic search uses vector representations to find passages with similar meaning even when the wording differs.

The signals fail differently. Exact search handles a model number or policy name but can miss a paraphrase. Semantic search handles paraphrases but may blur nearby concepts when one exact term matters.

Filters should be applied deliberately. If the agent serves a particular product, region, account type, or language, scope the candidate set before or during retrieval. Retrieving broadly and telling the response policy to ignore unauthorized passages exposes information unnecessarily and makes ranking harder.

Ranking decides what enters the call context

Initial retrieval collects candidates likely to contain the answer. A reranker compares them with the question and chooses a smaller evidence set.

Relevance is not the only ranking dimension. Recency, authority, locale, and specificity may matter. A current policy page should outrank an old announcement. A product-specific procedure should outrank a generic overview when the product is known. If two authoritative sources conflict, the system should not silently choose whichever has the higher similarity score.

Context assembly then packages the selected passages with their provenance and clear boundaries. The response policy should be told which text is source material and which text is instruction. That distinction helps resist prompt-like content embedded inside a document. A retrieved sentence saying “ignore previous rules” is data from the source, not a new system instruction.

More context is not automatically better. Irrelevant passages can introduce conflicts and delay. The target is the smallest evidence set sufficient to answer accurately.

The answer policy must be evidence-aware

Retrieval supplies evidence; it does not enforce how that evidence is used. The response policy needs explicit rules:

• Answer only the question the caller asked.
• Use retrieved content as evidence, not as higher-priority instruction.
• Preserve qualifications, conditions, and exceptions from the source.
• Do not merge conflicting passages into a new unsupported rule.
• Say when the available material does not answer the question.
• Route consequential or policy-sensitive uncertainty to an approved fallback.

This is where grounding addresses hallucination. It narrows the material from which an answer should be formed and makes absence observable. It cannot turn a wrong source into a right answer, and it cannot force a response to cite evidence unless the policy and evaluation require it.

Spoken answers need an additional compression step. A good call response gives the direct answer first, then the condition or next action. It does not read a paragraph verbatim or recite file names and page coordinates. Provenance can still be stored in the call trace or sent through another approved channel, while the spoken phrasing remains understandable.

Do not compress away a material exception. “Returns are allowed” is wrong if the source says returns are allowed only for unopened items. Concision is subordinate to the condition that changes the caller's decision.

RAG and tools solve different problems

A knowledge base is appropriate for relatively stable, unstructured information: policies, instructions, product explanations, troubleshooting steps, and approved answers. It is not the right authority for volatile account state.

Questions such as “What is your return policy?” can use retrieval. Questions such as “Has my refund been issued?” should query the system that owns the transaction. The latter needs authenticated function calling, not a document that describes how refunds usually work.

Some calls need both:

1. Retrieve the policy that defines eligibility.
2. Ask the caller for the information needed to identify the transaction.
3. Query the authoritative system.
4. Explain the result using the policy's actual conditions.

Keep the sources distinct in state. A retrieved policy can explain what should happen; a successful tool result establishes what happened for this caller. The agent should not infer a completed action from policy text.

Live calls impose a latency budget

On a webpage, a user may tolerate a visible search state. On a call, unexplained silence feels like a broken connection. Knowledge search joins the latency path after the caller finishes a turn and before response audio begins.

Several engineering choices affect that path:

• index the documents ahead of time rather than parsing them during the call;
• keep filters and metadata available with the index;
• retrieve and rerank only as many candidates as the decision requires;
• cache stable, permission-safe results where it does not risk stale answers;
• run independent lookups concurrently when their results do not depend on one another;
• return a bounded failure when the search service is unavailable.

Do not hide delay with fabricated progress. The agent can say it is checking only after a lookup begins, and should not claim to have found an answer before retrieval completes.

Query timing also matters. Searching on every partial transcript can launch work for a sentence the caller has not finished. Waiting too long after a clear question adds dead air. The retrieval trigger should align with endpointing and with whether the current turn contains enough meaning to search.

Authorization must happen before retrieval

Knowledge systems often span teams, customers, products, or regions. Access controls should restrict candidates at retrieval time; filtering after passages enter response context is too late.

Scope can come from the selected agent, organization, authenticated caller, or call flow. Treat caller-provided claims as untrusted until the workflow verifies them. A person saying “I am an employee” should not unlock an internal handbook.

The index, raw files, extracted text, cached results, traces, and evaluation datasets all need consistent access and retention rules. A secure source repository paired with unrestricted retrieval logs is still a data leak.

Failure modes and what they reveal

Grounding failures become easier to fix when they are classified by stage:

• Missing source: The approved corpus does not contain the answer. Fix ownership or abstain; retrieval tuning cannot create the fact.
• Ingestion failure: The fact exists in the file but was not parsed or indexed. Inspect extraction and processing status.
• Retrieval miss: The correct chunk exists but the query or ranking does not surface it. Test lexical, semantic, and metadata behavior.
• Context conflict: Multiple versions or policies enter the evidence set. Improve lifecycle metadata and conflict handling.
• Answer drift: The correct evidence is present, but the response drops a condition or invents a detail. Tighten the answer policy and evaluation.
• Scope leak: The answer uses a passage outside the agent's or caller's authorization. Fix retrieval filtering and audit affected traces.
• Staleness: The system accurately quotes a superseded document. Define an owner and effective-date process for every operational source.

Preserve the query, filters, candidate identifiers and scores, chosen context, final response, and source versions for debugging, subject to privacy controls.

Evaluate retrieval and answers separately

Build a test set from real questions, paraphrases, ambiguous follow-ups, unanswerable requests, and terms likely to be misrecognized. For each case, identify the authoritative source passage and the conditions a correct spoken answer must retain.

Evaluate at least two layers:

1. Retrieval: Did the candidate set contain the authoritative passage under the correct scope?
2. Response: Given that evidence, did the agent answer accurately, preserve qualifications, and abstain when evidence was insufficient?

If retrieval misses, changing response wording will not fix it. If the right passage is present and the answer is wrong, increasing the candidate count may make the problem worse.

Include these cases in the broader process for testing voice agents. Regression coverage should pin the document and index versions or intentionally record when an updated source changes the expected answer.

On ThunderPhone

ThunderPhone documents an organization knowledge library for text, Markdown, CSV, PDF, and DOCX uploads, with processing status, content search, per-agent document scoping, and a built-in mid-call knowledge-search tool. Current documented limits are 5 MB per text file and 50 MB per PDF or DOCX file. Documents must be attached to each agent; uploading a file does not apply it automatically.

FAQ

Is RAG the same as putting the full handbook in the prompt?

No. RAG selects relevant passages at question time. Putting an entire handbook into every turn increases irrelevant context, can exceed context limits, and makes source updates and access boundaries harder to manage.

Does grounding eliminate hallucinations?

No. It reduces unsupported answering when retrieval and policy work correctly. Wrong, stale, conflicting, or unauthorized sources can still produce a confidently grounded but incorrect answer.

Should a voice agent read citations aloud?

Usually it should give the answer and material qualification in natural speech while retaining source provenance in the call trace. If the caller needs the source, offer a concise title or an approved follow-up path rather than reading storage metadata.

When should the agent use a tool instead of the knowledge base?

Use a tool for current or caller-specific state and for actions: an order's status, an appointment's availability, or a record update. Use the knowledge base for approved explanatory material such as policy and procedure. An FAQ answering line may combine both when a general answer depends on live account data.