The hard part of RAG is teaching it to say “I don’t know”

A retrieval system that breaks is easy to fix. A retrieval system that answers confidently from nothing is the expensive one, because nobody files a ticket about it.

Ask a question the documents do not cover and most systems will answer anyway. The retrieval step returns the closest passages it can find, which on an unrelated question are still passages, and the model does what it was trained to do: writes a fluent answer from the material in front of it. The output looks exactly like a correct answer. There is no signal in the formatting, the tone or the confidence that anything went wrong.

People notice eventually. What they do next is the real damage: they stop trusting the system, go back to searching the shared drive, and the project quietly becomes something nobody uses.

Refusal is a retrieval problem first

The obvious fix is to tell the model to say when it does not know. That helps less than expected, because the model cannot tell. It sees passages that were retrieved and has no way to judge whether they were retrieved because they were relevant or because something had to come back.

Refusal only becomes possible once retrieval is good enough that an empty result is meaningful. That means a few things in practice.

Documents get split along their own structure rather than at a fixed character count. A table cut down the middle retrieves as nonsense. A clause severed from its heading loses the thing that made it findable.

Search runs both ways. Semantic matching finds passages that say the same thing in different words, which is the whole point. Keyword matching catches the exact clause reference, part number or policy code that embeddings routinely miss, and those are disproportionately what people actually search for.

Candidates get reranked before they reach the model. Retrieving twenty and narrowing to four beats retrieving four, consistently, and the additional cost is small enough not to argue about.

Once retrieval is doing its job, a low-scoring result genuinely means the corpus does not cover the question, and a threshold becomes something you can act on.

What a refusal should look like

Not an apology. A statement about the corpus rather than about the system.

“The documents available to me do not cover this” is useful, because it tells the reader where to go next. “I’m sorry, I’m not able to help with that” is not, because it sounds like a limitation of the assistant and invites the reader to rephrase and try again until something comes back.

The second version trains people to work around the safeguard. We have watched it happen.

Citations are the check, not the decoration

Every answer carries its sources, linked to the passage rather than the document. One click to verify, not a sixty-page PDF and a search box.

The reason is behavioural. A system where verification is cheap gets verified occasionally, which is enough to keep people calibrated about how much to trust it. A system where verification means finding the right page yourself never gets verified at all, and trust becomes a matter of habit rather than evidence.

Where it hands over

On a patient-facing system we built for a US healthcare team, the assistant books appointments, takes payments and routes people to the right specialist. Anything approaching clinical judgement escalates to a person.

That line was drawn before any code was written, which is the only point at which it can be drawn properly. Afterwards there is always a specific case that seems fine to handle automatically, and each one is individually reasonable.

How to know whether it works

Build the evaluation set from questions collected from the people who will use the system, before the build. Include the ones the documents do not answer, because those are the cases you are actually trying to control.

A test set written by the developer proves the developer built what they intended. That is a different and much easier question than whether the thing works.