SAMPLE SIZE CALCULATORIf you are setting up a marketing test and wondering how many contacts you need to achieve a significant result, our new sample size calculator can provide answer for you.

Our calculator is designed provide sample sizes for tests when you are measuring differences in proportions, so you can use it for any result you look in terms of percentages—not just response rates, but also open rates, join rates, click rates or any other similar measure.

Here’s how it works:

  1. First, you need to choose your level of confidence. That confidence level is how certain you want to be that the difference you measure with your test is real and not due to chance. We’ve set it so you can either choose 90 percent confidence, 95 percent confidence or 99 percent confidence. You can think of it in this way. If you choose 90 percent confidence, you’ll get the right answer 9 out of 10 times. With 95 percent confidence, it will correct 95 out of a 100 times. And with 99 percent confidence, you’ll only get the wrong answer 1 out of 100 times.

    So, why wouldn’t you choose 99 percent confidence all the time? Unfortunately, the higher confidence you choose, the more volume you need to test. So, unless it’s critical to minimize uncertainty, it may be better to choose a lower confidence level.

    We’ve set the power of our calculator at 80 percent. That means, if you use the volume we suggest and you don’t achieve a significant result, 80 percent of the time you can be sure that there was no real difference between your test and control. While confidence level tells you the probability of avoiding false positive results, the power gives you the probability of avoiding false negative results.

  2. Second, you need to estimate what you think the response rate of your control panel will be. You should base this on the typical prior performance of your control.

  3. Last, you need to hypothesize what you think the response rate of your test will be. This is probably the hardest step, because you need to think about what you are testing and what would be a meaningful difference from your control. The bigger of a difference you expect between your control and your test, the lower the volume you will need to test.

Are you ready to give it a try? Our Sample Size Calculator is part of the suite of tools in the Creative Lab of our website.

Visit MWD’s Sample Size Calculator >>

Peter Schoewe is a Vice President and Director of Analytics at Mal Warwick Donordigital, the integrated fundraising, marketing, and advertising company. Peter has over 20 years experience in direct response fundraising and is one of the nation’s leading experts in direct mail and integrated fundraising.