vault / agents / golden-dataset.md

Golden datasets

Agents & LLMs 53 words 2 outgoing 2 backlinks

A golden set pairs questions with expected answers and source references. In the design described in the post, simulated personas propose document-anchored candidates, but people approve them before they count. Generated text is a proposal, not ground truth, and changing documents can make a once-good record stale.

Read the full example: related post.

LINKS IN THIS NOTE

[[ Human review ]] People approve proposed evaluation records and handle doubtful feedback cases. Keep those jobs explicit: approving a reference answer is different from deciding which owner should investigate an incident. The triage workflow routes missing or conflicting evidence to review instead of... [[ Evaluation-driven development ]] Compare relevance, retrieval recall, groundedness and latency against a baseline. Inspect examples as well as scores. In my current workflow, evaluation produces a report for engineers rather than an automatic deployment gate. That is a description of this system, not...