vault / exp / ab-testing.md

A/B testing

Experimentation 53 words 2 outgoing 4 backlinks

Random assignment creates comparable groups; a shared metric defines what is compared. A p-value describes how surprising the data would be under a specified null hypothesis. It is not the probability that the hypothesis is true. Decide the measurement and testing plan before interpreting a favourable result.

Read the full example: related post.

LINKS IN THIS NOTE

[[ Metric semantic layer ]] A metric definition includes more than a formula: its population, event, time window and owner matter too. The platform I describe stored those definitions as metadata and used a BigQuery query engine to compute them. Reusing one definition helps only... [[ Multi-armed bandits ]] A bandit shifts allocation while learning which option performs better. That trades some fixed-test simplicity for less traffic spent on weak options. In the coupon example, Thompson sampling updates a traffic split in a batch job; request handling uses that...