A/B tests
Show visitors one of several variants, and see which one gets more of them to do what you hoped.
An experiment has a name and two to eight variants. Ask which variant a visitor gets, show it, and choose the event that counts as success. The app then compares the variants and says whether the difference is likely real.
In a web page
With the script on the page:
<script>
snekot('experiment', 'pricing-page', ['control', 'annual-first'], (variant) => {
if (variant === 'annual-first') document.body.classList.add('annual-first');
});
</script>
The callback gets the variant as soon as it is known. Once the script has loaded, the call also returns a promise of it. If the answer cannot be had, because the visitor is offline or the request is blocked, it is the first variant, so list the one you show today first.
Asking again on the same page costs nothing: the first answer is reused.
How a visitor keeps their variant
The variant is not stored anywhere. It is worked out each time from the experiment’s name and who is asking, so the same person always gets the same one:
- a visitor the site remembers is the same person on any day, from any network;
- a signed-in person, set with
setUserId, is their user id, on any device; - anyone else is the day’s visitor, the same one their page views are counted under. They keep their variant on every page of a visit, and until the day ends. Tomorrow they count as someone new and may get another.
So for a test that runs longer than a day on visitors who are not signed in, turn on remember, or accept that some people will see both variants on different days. Nothing extra is kept on the visitor’s device either way.
Keep the list of variants the same, in the same order, everywhere the experiment is asked for. Changing the list deals everyone again.
Avoiding a flash
The answer takes one request, so a page that draws the first variant and then swaps will flicker. Hide what differs until the callback runs, or decide on the server.
From a server
POST https://api.snekot.co/v1/experiment with a private or public key:
curl -X POST https://api.snekot.co/v1/experiment \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"experiment":"pricing-page","variants":["control","annual-first"],"user_id":"user-123"}'
{ "experiment": "pricing-page", "variant": "annual-first" }
| Field | |
|---|---|
experiment |
Required. The name, up to 80 characters. |
variants |
Required. Two to eight names. The first is the one the others are measured against. |
user_id |
The person, when you know them. |
ip, user_agent |
For someone with no id, from a server: their address and user agent. They tie the person to their page views, and are not stored. |
A server must send a user_id, or an ip and user_agent. Every SDK has the call. With the TypeScript SDK:
const variant = await events.experiment({ experiment: 'pricing-page', variants: ['control', 'annual-first'], user_id: user.id });
Exposures
Being shown a variant is recorded once a visit, the first time the experiment is asked for. Exposures are not events: they do not appear in the feed and do not count against the month.
An experiment appears in the app, under Experiments (under Charts on a phone), the first time anyone asks for it. A project holds up to 50.
Reading the result
Choose the experiment’s goal in the app: any event, or a page view. For each variant you then see how many people were shown it, how many of them did the goal afterwards, and the rate.
A person counts once, for the variant they saw first, and converts if they did the goal after seeing it. For the goal to be tied to an anonymous visitor it has to come from the same browser, or from your server with the visitor’s ip and user_agent; see Sending events.
Under the numbers is one sentence:
- Not enough data yet. A variant has fewer than 100 visitors, or there are too few conversions to compare.
- No clear difference yet. The gap is one that chance alone produces often.
- One variant converts better. A gap this wide would be rare if the variants were really the same, so it is likely real.
The comparison is a two-proportion z-test of each variant against the first, at 5%. With more than two variants that 5% is shared between the comparisons, so each has to clear a stricter bar.
Looking every day and stopping the first time a winner is named makes a chance result likelier. Decide how long the test runs before it starts.
The same numbers are available with a read key at GET /v1/experiments, and to assistants.