A/B testing is an essential part of an impactful marketing campaign. With every component of your campaigns that you can test, landing pages, imagery, headlines, value props, etc., you'll be able to refine your approach and make smarter, more effective use of your resources.
Knowing when it's time to stop an A/B test, however, is not always easy. Stop the test too soon, and you may end up with biased, inaccurate results that lead you down the wrong path. Stop it too late, and you'll have wasted time that you could have used to implement the winning variant or, better yet, run another A/B test.
To help you find just the right stopping place, we've created this quick guide and calculator to help you determine the optimal sample size you should plan to reach in your next A/B test.
Refining your marketing with A/B testing
A/B testing compares two versions of a webpage, email, or any other marketing asset to determine which performs better. It involves creating two versions of the same asset, changing one element (headline, image, call-to-action) in one version, and then randomly showing each version to a subset of your audience. The test results are then analyzed to determine which version performed better regarding engagement, conversion rates, or other key metrics.
But how big of an audience do you need to participate in your test before you can come to a reliable conclusion? Enter the statistically significant sample size.
Calculating your minimum sample size
Sample size refers to the number of individuals or observations included in a study or experiment. It is the number of participants or data points that are selected from a larger population to represent that population in a research study. Sample sizes are used in experiments where it isn't feasible to analyze all the data available, so you calculate a portion of the data to work on. The sample size is crucial because it affects the accuracy and reliability of the results obtained from the study. A larger sample size generally leads to more accurate and reliable results, while a smaller one may lead to less accurate and less reliable results.
Besides the population size, there are two main terms you need to consider when calculating sample size:
- Margin of error: the margin of error is a percentage that shows how close the sample results will be concerning the actual value of the overall population. The smaller the margin of error, the closer you are to having the exact answer at a given confidence level, but it may require a larger sample.
- Confidence level: this value measures the degree of certainty of how well a sample represents the entire population within the margin of error chosen for the study. For example, if the confidence level is 95%, then you can be 95% certain that the results will accurately fall within the margin of error chosen.
Here’s a Google Spreadsheet template where you can calculate the sample size of any population using different margins of error and confidence intervals.
How to use the sample size calculator - an example
Let’s say you want to run an email marketing campaign on 1,000 prospects close to buying your product, but they might require a slight push to make them decide to buy. You have to pick the best email copy to give you a higher probability of conversion. There are two email copies to test: one will resonate more with their pains and fears, and the other with their dreams.
Using the template below, enter the population size in the first input row, then select your desired confidence level and margin of error on the rows below. It will give you the sample size of your email copy experiment on the last row, named “Sample Size”. If you experiment on a confidence level of 95% and a margin of error of 5%, you can be 95% sure that the winning copy will behave the same way on your whole population, within the margin of error chosen.
Calculate your optimal sample size
Simply input the total population size you're testing against and the confidence level and margin of error you'll be using for your test. Your optimal sample size will appear in the "Sample size" box.
Start running better, more reliable A/B tests
Use our template to launch into your next A/B test with more confidence in the results and efficiency in your testing. If you’re looking for more on testing and reporting on your marketing campaigns, check out these articles from the Kalungi blog:
