vault / agents / online-eval.md

Online evaluation

Agents & LLMs 51 words 3 outgoing 1 backlinks

Offline examples help compare changes under controlled conditions. Production feedback shows questions and failures the reference set missed. The two complement each other. Online signals need investigation: a thumbs-down tells us that something went wrong, not whether retrieval, content, configuration or the answer caused it.

Read the full example: related post.

LINKS IN THIS NOTE

[[ Evaluation-driven development ]] Compare relevance, retrieval recall, groundedness and latency against a baseline. Inspect examples as well as scores. In my current workflow, evaluation produces a report for engineers rather than an automatic deployment gate. That is a description of this system, not... [[ Observability ]] For feedback investigation, record enough of the request to reconstruct what happened: the question, retrieval, response and relevant configuration. A trace helps connect those steps. Dashboards can show a pattern, but a concrete replay is often what tells us which... [[ Feedback triage ]] A thumbs-down starts an investigation, not a diagnosis. The workflow I built replays the question, collects evidence, distinguishes a content gap from a configuration issue, and routes the case to an owner. Routing is ordinary code; the model proposes the...