What Should You Test?

by Stephen Hitchcock

The question we as fundraising consultants are asked most often is also the question we keep asking ourselves: What elements of a direct mail package are most important to test?

The question is important. What makes direct mail so successful as a fundraising medium is that we can track results - and learn from those results.

Everyone involved in direct mail makes decisions based on anecdotal information and on personal preferences. We also depend on experience: We compare one year's results to another year's results, or we line up returns from one package mailed in June against returns from another package in September.
But those judgment calls are at best a calculated risk. At worst, delusions.

Testing is a must

To make consistent improvement in your fundraising program, you must test. That means mailing random samples (with test and control panels of names from your own donor list or from prospect lists) at exactly the same time and then subjecting those results to the rules of statistical validity.

But the catch is that even if you're committed to testing, you must mail a large enough quantity to achieve statistical validity: To conduct reliable tests -with some important exceptions - your house list should be 10,000 names or larger, and an acquisition effort (prospect mailing) should include at least 50,000 pieces.

If that's the case, then here are three of my favorite elements to test:

1. In fundraising mail, the test that usually makes the greatest difference involves the amount you request. That's particularly the case in acquisition mailings: Will you persuade more individuals to become members or donors by asking for an initial gift of $15, $20, $25, or $35? For some organizations, asking for a lower initial gift results in a higher response rate without a decline in the average gift.

Our firm has also tested gift levels in special appeals and renewals. Since the purpose of these mailings is to generate net income, increasing response rate can make a big difference in income.

The great advantage here - besides an increased number of new members or higher net income - is that testing gift amounts is one of the least expensive tests you can perform. It doesn't involve expensive variations in copy or package design.

2. Postage is the other crucial element to test. In acquisition efforts, we test to learn whether we get more members by using Business Reply Envelopes (BREs) or, instead, reply envelopes where the new member places a stamp on the envelope. BRE postage is expensive and, in some cases, we've found through testing that results are actually higher when BRE envelopes aren't used.

For special appeals and renewal mailings sent to those who are already donors, we like to find out for which groups or segments of donors a "live stamp" reply envelope makes a difference. Adding a 32-cent postage stamp to each return envelope does dramatically increase costs, but in many cases it can double the response rate.

For some acquisition programs, it may also be worth testing whether sending your mail via first class rather than bulk (the new "standard mail") can significantly boost returns. If you struggle to acquire new members, investing in first class postage may be worth it.

3. Involvement techniques - surveys, petitions, post cards to elected officials - can often boost response rates. You might also try a test that involves asking prospective or current members to sign a "Statement of Principle" or a "commitment to act."

When you test these involvement devices in acquisition, be prepared to receive more responses without any contribution enclosed - or with low-dollar contributions. You may end up acquiring no more renewable members (typically, those whose initial gift is at least $15 or $20), but you'll have some additional income to offset the investment cost of your acquisition mailings.

There are some tests that I give lower priority - except for all but the largest organizations that regularly mail in quantities of 250,000 plus. The most notable example is testing the size of envelopes. Using bigger or odd-size envelopes hasn't had much of an impact for our clients. In smaller quantities, those special envelopes haven't increased the response enough to justify their increased cost.

Other articles on acquisition:
•How to get started with direct mail (for small organizations)
•Is it OK to take a loss on acquisition mailings?
•Do label packages work?
•Choosing the right lists
•What's the best postage for direct mail letters?
•How to handle donor complaints about duplicate appeals