vault / agents / human-review.md

Human review

Agents & LLMs 53 words 2 outgoing 2 backlinks

People approve proposed evaluation records and handle doubtful feedback cases. Keep those jobs explicit: approving a reference answer is different from deciding which owner should investigate an incident. The triage workflow routes missing or conflicting evidence to review instead of treating a confident explanation as a decision.

Read the full example: related post.

LINKS IN THIS NOTE

[[ Golden datasets ]] A golden set pairs questions with expected answers and source references. In the design described in the post, simulated personas propose document-anchored candidates, but people approve them before they count. Generated text is a proposal, not ground truth, and changing... [[ Feedback triage ]] A thumbs-down starts an investigation, not a diagnosis. The workflow I built replays the question, collects evidence, distinguishes a content gap from a configuration issue, and routes the case to an owner. Routing is ordinary code; the model proposes the...