A/B Test Hypothesis & KPIs

Motivation

make sure you are solving a real problem
Chicken and egg
You don’t know everything, you assume.
it’s easy to make up a story that fits your findings as opposed to doing the hard work of understanding what’s going on.
Scientific approach

What is it

A good hypothesis is a statement about what you believe to be true.

Experiments do not fail - hypotheses are proven wrong

Do’s and Don’ts

Clear Hypotheses Lead to Clear Learning

Try to prove or disapprove your hypothesis by investigating the data.

Not everything needs an A/B test

Write your hypothesis! It helps to make it clear and concise.

What will we learn if we win? What will we learn if we fail?

Test bold hypothesis. Stuff you believe that will move the needle, your traffic is expensive.

The hypothesis should be correlated to one of the company’s main KPI

Execution

Sometimes a poor implementation keeps a good idea from succeeding, conversely, a great idea can succeed in spite of a poor implementation)
Motivation: It’s the minimal thing you can do in order to prove/ disregard your hypothesis (MVP).
Make sure your execution is bold: visible and prominent.
Make sure you do it you have enough people that will be exposed to the test.
Go back to your hypothesis, check if it answers your hypothesis.

Define your KPI

‌‌Define your KPI - What are you optimizing for? What is the top-line metric? how would you define success?
Choose the main KPI which is one of the company goals: WAPP, conversion to paying, Collection, TROI.
Choose a proxy KPI: Predicts the main KPI, and is correlated to the main KPI. It allows you to wait less time for results. (for example number of activities in 3 days.)
“Execution proxy”: Sometimes execution and predicted will be the same. (just for reject)
Sensitive to changes so you can detect a change. But not too much: because then it will take its tool long to be stable: can change easily. (WAPP vs DAPP, more than 10 communication events)
the conversion rate of the proxy is similar or higher to the main KPI conversion rate. show correlation (for example, the intent is a proxy to conversion to paying with high accuracy)
Check AA, above which lift you can be sure it's not too fluctuate (in given sample size)
See if you can use a proxy KPI that other people already proved (Content event, 4d4w, Intent etc..).

Define your minimum desired sample size

in order to define your minimum sample size for the test you need

Don’t conclude anything if you haven’t reached your minimum sample size / used our test simulator for getting results with confidence.

as bigger the change will be the smaller traffic you'll need

Recent Posts