Do you have enough traffic to run your experiment?

Let’s say you run a small business. You sell widgets online for the low price of $2000 a widget. You purchase Google search ads for the keyword “widgets” to drive traffic to your site. These work well, costing $5 per click which seems high but is actually cost effective because 1 in 10 people who click on your ad end up purchasing a widget. You have a budget of $1000/month for ads. Can you A/B test your ad landing page effectively? 

Luckily you know of Evan and his awesome A/B tools. Using his Sample Size calculator you figure out that if you want to detect an improvement of 50% you need 6 months of data to test two different landing pages. And if you want to detect an improvement of just 10% you need almost 6 years of data. 

Now this doesn’t mean you shouldn’t run experiments. But you should be aware that with outcomes that have a small chance of happening you need either a really big improvement or lots of data to be able to detect it with confidence.

If this interests you I highly recommend reading some of his A/B testing articles.