vault / mlops / serving.md

Model serving

ML platform 54 words 3 outgoing 5 backlinks

A trained model is not yet a service. A shared serving path handles the repeated work around it: authentication, tracing, deployment conventions and orchestration. Keep the supported path narrow and explain failures clearly. The post distinguishes the interface I built from general design advice I would apply now.

Read the full example: related post.

LINKS IN THIS NOTE

[[ SDK design ]] An SDK is an interface for model authors. Sensible defaults reduce repeated configuration, but hiding every detail makes debugging harder. Make the ordinary path small, leave important behaviour inspectable, and write errors that suggest the next action. These are design... [[ Distributed traces ]] A trace follows a request through the services and steps it touches. It lets a slow or failed prediction be inspected as one chain rather than unrelated log lines. In a serving interface, tracing is useful as a shared platform... [[ P95 latency ]] P95 describes the point below which 95% of measured request times fall. It says more about slow requests than an average, but still needs a workload and measurement window. A load-test result is not a promise about production traffic. Bounded...