When Should an AI Assistant Say 'I Don't Know'?

An AI assistant should say "I don't know" whenever it lacks enough reliable information to support an answer. In a RAG system, that usually means no sufficiently relevant source material was retrieved, the available passages are weak or contradictory, or the answer cannot be justified from the retrieved evidence. In those situations, refusing to answer or explicitly signalling uncertainty is better than generating something that merely sounds plausible.

This matters especially in knowledge systems used for legal, public-sector or business information. A language model can almost always produce a fluent response, even when the required facts are missing. Without safeguards, that creates a system that sounds authoritative while sometimes guessing. A well-designed assistant therefore treats "no reliable answer available" as a valid output rather than as a failure.

Language models don't automatically know when knowledge is missing

A language model does not behave like a database where an answer is either present or absent. It generates text by predicting what is likely to come next based on the prompt and the context it receives. That means generation can continue even when the underlying evidence is poor. Suppose a user asks about a specific clause in an internal agreement, but that agreement is not present in the RAG index — the model may still produce a legal-sounding answer, because the question resembles patterns it has seen elsewhere.

This is where hallucination emerges: not necessarily because the model is deliberately inventing information, but because it continues generating despite lacking sufficient factual grounding. A RAG system therefore needs more than good retrieval — see also RAG explained: how to build an AI that doesn't lie. It also needs explicit behaviour for cases where retrieval fails.

Refusal should start before generation

The first safeguard usually sits in the retrieval layer. A RAG pipeline retrieves passages that are semantically related to the user's question, and these passages are typically ranked using a relevance score or another retrieval metric. If every retrieved passage is only weakly related to the query, that is a strong indication that the system does not have enough evidence.

You can therefore introduce a threshold: only questions with sufficiently relevant retrieval results are passed to the language model. If that threshold is not met, the application does not need to ask the model for a free-form answer at all — it can return a controlled response such as "I cannot answer this reliably based on the available sources." This is often safer than relying on the language model itself to decide whether it has enough information.

There is no universal threshold. Set it too high and the assistant refuses useful questions; set it too low and irrelevant passages enter the prompt, increasing the chance of hallucination. The correct threshold needs to be evaluated against real questions from the intended use case, not copied from a handbook.

Detect when retrieval found no useful context

A common RAG design mistake is to always send the top retrieval results to the model, regardless of their actual quality. A vector search will almost always return something — the top five results are therefore not necessarily relevant; they may simply be the five least irrelevant passages in the index.

The application should distinguish between useful context retrieved, ambiguous or incomplete context retrieved, and no useful context retrieved. The third category should not result in a normal confident answer. The middle category can produce a more nuanced response: the assistant explains that the available sources answer only part of the question and identifies which information could not be established — often more useful than a binary system that either answers everything or refuses the entire request.

Verify that citations actually support the answer

Citations alone do not guarantee reliability. An AI system can cite a document that is generally related to the subject while the specific claim in the answer is not supported by that document — a pattern examined in more depth in Why RAG without citations is worthless. This happens when the model generates a plausible statement first and then associates it with the closest available source.

For systems where traceability matters, citation verification should therefore be an explicit step: does a supporting source passage exist, does that passage actually support the claim, and does the generated answer add facts that are absent from the retrieved context? Some of these checks can be rule-based; others can be handled through a second model pass that compares individual statements with the retrieved evidence. If a material claim cannot be grounded, the answer should be revised, qualified or rejected.

Confidence is not a magical reliability score

AI systems often use the term confidence score, but the concept can be misleading. A language model generally does not provide a single objective number that tells you whether an answer is true. In a RAG architecture, "confidence" is more often constructed from several separate signals: retrieval relevance, the number of passages supporting the same answer, contradictions between sources, whether the answer follows directly from the retrieved text, and whether citations genuinely support the claims.

These signals can be combined into decision logic: strong retrieval combined with verified grounding may permit an answer, while weak retrieval or unsupported claims should trigger uncertainty or refusal. This is more reliable than simply asking the model "how confident are you?" — language models can sound highly confident while being wrong.

Sometimes the assistant should refuse only part of the answer

Many user questions contain multiple components, and those components may not all be equally answerable. Imagine a user asks: "What does our employment policy say about working from home, and how much home-working allowance do I receive?" The knowledge base may contain the remote-working policy but no information about an allowance. A good assistant should not reject the entire question, but it should also not invent an amount — it explains the policy that is supported by the available sources and clearly states that no allowance amount was found.

This pattern is particularly useful for long or compound questions. Technically, the system can split a request into individual claims or sub-questions and evaluate whether sufficient evidence exists for each one. The result is an assistant that exposes the boundary of its knowledge instead of hiding it.

A system that never refuses is predictably unreliable

An assistant that always provides an answer may initially appear more helpful. In practice, that design makes the system harder to trust, because users can no longer tell the difference between answers backed by strong evidence and responses that are primarily plausible language generation. The consequences increase with the stakes: an incorrect answer in an internal product catalogue may be inconvenient, but an unsupported answer about legal documents, procedures, public policy or compliance requirements can influence real decisions.

A system that never refuses also hides weaknesses in the knowledge base. Missing documents, poor chunking and outdated indexes remain unnoticed because the language model continuously fills the gaps with generated text. Refusals therefore provide useful diagnostic information: if users repeatedly ask questions for which the system cannot find sufficient evidence, that reveals exactly where the knowledge base needs improvement.

Monitor why answers are refused

A reliable RAG system should record not only that a question was refused, but also the reason: no documents retrieved, relevance score below the required threshold, conflicting sources, required information absent from retrieved passages, or citation verification failed. If many questions fail because retrieval returns nothing useful, indexing, document coverage or chunking may need improvement. If retrieval works but grounding checks frequently fail, the problem may lie in prompting or answer generation. Refusal therefore serves both as a safety mechanism and as an operational feedback loop.

A trustworthy AI assistant should not try to answer every question at any cost. Build explicit checks around retrieval quality, missing evidence and citation support, and let the system partially or fully refuse when those checks fail. "I don't know based on the available sources" is not a weakness in a RAG system; in many cases, it is evidence that the system has been designed correctly.