vault / exp / uplift.md

Uplift modeling

Experimentation 64 words 2 outgoing 0 backlinks

Model who changes behaviour because of the message, not who converts anyway. Split the audience: persuade them (target these), sure things (wasted spend), sleeping dogs (leave alone - the message backfires). Gotchas: you never observe the counterfactual, so evaluate with uplift or Qini curves on a held-out randomized group, not accuracy; and a strong outcome model is not an uplift model.

Reference: scikit-uplift docs.

LINKS IN THIS NOTE

[[ A/B testing ]] Random assignment creates comparable groups; a shared metric defines what is compared. A p-value describes how surprising the data would be under a specified null hypothesis. It is not the probability that the hypothesis is true. Decide the measurement and... [[ Thompson sampling ]] Reach for it when traffic is live and a fixed A/B test would keep sending users to a losing arm: sample each arm’s posterior, serve the highest draw, and let uncertainty supply the exploration. In the production pattern from the...