Your RAG has sources. Why are the answers still wrong?

Start with a specific bad answer.

“The RAG is hallucinating” describes a symptom. A useful investigation begins with the question, the answer, the retrieved material and the expected behaviour. Keep those together: the output alone may hide where the failure started.

Separate the failures.

1. The system retrieved the wrong material.

Check whether the relevant source was available and whether the retrieved passages were actually relevant to the question. If the evidence is wrong, a well-written answer can still fail.

2. The answer is not supported by the material.

A reference is useful only if it supports the associated claim. Review what the response says against the passages actually provided. Citation presence and citation correctness are different checks.

3. The answer exceeds the product’s capabilities.

The system may describe actions the product cannot perform. In our sister-concern RAG case, an answer claimed users could build a Ferrari using the app. That example made the missing boundary immediately visible.

Choose changes from the evidence.

Possible changes include the source material, retrieval approach, context supplied to the model, answer instructions or handling of unanswerable questions. These are investigation areas, not a diagnosis for every RAG system.

Record the original failure and repeat representative checks after a change. Include ordinary questions, previously failing questions and questions the system should not try to answer.

Define what “fixed” means.

Agree the expected behaviour before treating an improvement as a completed recovery. That might include using relevant sources, keeping claims within the product’s actual capabilities and acknowledging when the material does not support an answer.

If these failures are blocking your product, explore our AI system and RAG rescue service.

Read the sister-concern RAG rescue →