vault / agents / rag.md

RAG

Agents & LLMs 48 words 3 outgoing 3 backlinks

Retrieval-augmented generation searches a document collection, then writes an answer using the retrieved context. Keep retrieval and answer quality separate: finding the right document does not guarantee the model uses it, and a plausible answer does not prove the document was found.

Read the full example: related post.

LINKS IN THIS NOTE

[[ Retrieval recall ]] Ask whether the expected sources reached the retrieved context. When the answer is wrong, this helps distinguish missing evidence from poor use of available evidence. The expected source set must itself be checked; a score against an incomplete reference can... [[ Groundedness ]] Check whether the answer’s claims are supported by the retrieved sources. This differs from usefulness: an answer can stay within its evidence and still fail to answer the question. Read groundedness alongside relevance and retrieval recall, rather than reducing all... [[ Evaluation-driven development ]] Compare relevance, retrieval recall, groundedness and latency against a baseline. Inspect examples as well as scores. In my current workflow, evaluation produces a report for engineers rather than an automatic deployment gate. That is a description of this system, not...