Free tools

Plan and read your A/B test

Before the test: how many visitors you need, and how long it will take. After: the probability that version B really does better than version A.

Before the test: how many visitors?

Orders ÷ visitors of the page, over the last few weeks.

The smallest lift worth the test. 20%: going from 2% to 2.4%.

On the tested page, both versions combined.

The starting values are an example. Replace them with yours: the result follows every keystroke.

Result

Visitors per version

22,000

That is 44,000 visitors in total.

Enough to see version B go from 2% to 2.4%.

Test length
147 days

More than 60 days: at this traffic, this test will not reach a verdict. Test a bolder change, or a busier page.

After the test: who wins?

Orders or add-to-carts: the same measure on both sides. The starting values are an example.

Result

Probability that B beats A

94%

No verdict yet

The gap may still be due to chance. At this rate, you need about 22,000 visitors per version to see a lift of 20%.

Rate of A
2%
Rate of B
2.75%
Gap of B over A
+37.5%

How to read the result

  1. The number of visitors per version is the volume to gather before looking at the result. Stopping earlier, as soon as B seems to win, is the first cause of false winners.

  2. The probability that B does better is only a verdict from 95% (or at 5% and below: A wins then). In between, nobody knows yet.

  3. The smaller the lift to detect, the more visitors you need: seeing +5% takes about sixteen times more traffic than seeing +20%.

The formula

  • Visitors per version: two-sided test on two proportions, 95% threshold, 80% power
  • Probability that B wins: Beta posterior on each rate (uniform prior), compared through a normal approximation
  • Verdict: not before 100 visitors per version and 10 conversions in total

This is the method of the A/B tests in SellNext: the tool and the app give the same result on the same numbers. The number of visitors is rounded up.

What the tool leaves out

The tool does not see how your test ran: neither its length (count at least 7 days), nor an uneven split between A and B, nor orders attributed to the wrong version. It does not judge a price test: a higher price converts less and can still earn more.

Frequently asked questions

How long should an A/B test run?
At least a full week, even with a lot of traffic: Sunday does not buy like Tuesday. Then until you have gathered the calculated number of visitors. After two months without a result, the test will not reach a verdict.
Can I look at the results along the way?
Yes, but without deciding. The more often you look, the more likely you are to land on a gap due to chance. Set the volume first, and decide once it is reached.
What should I test first?
What every visitor sees: the title and the image at the top of the product page, the text of the buy button. One change at a time, otherwise you will not know which one mattered.

Go from the math to your store

SellNext starts from your product and prepares your brand, your Shopify store and your ads. You get profitability, A/B tests and stock tracking there, calculated on your real numbers.

Build my brand

No credit card to start