RAG applications
Know whether retrieval or generation failed
See whether the answer failed because the source was missing, the wrong content was retrieved, the context was weak, or the model ignored it.
- Question
- How long do I have to exchange an item?
- Retrieved
- returns_policy.md · 3 chunks
- Context
- Covers refunds, not exchanges
- Answer
- Not supported by the retrieved context
What you can do
Three checks between question and answer
Find the knowledge that is missing
Group questions the Knowledge Base cannot answer and see which sources need to be added or updated.
See whether retrieval or generation failed
Inspect the documents, chunks, and context supplied to the model, then compare the final answer against that evidence.
Fix it and run the same question again
Update the source, retrieval, or prompt, then re-run the scenario and compare the result.
Question to answer
Where a RAG answer loses its source
The question retrieves the wrong evidence, the answer drifts past it, the source is fixed — and the same question re-runs until the answer cites the context.
Question
exchange window?
Retrieved · 3 chunks
returns_policy.md
Answer unsupported
context covers refunds only
Source updated
exchange section added
Re-run · supported
answer cites context