June 2001
Minimizing bias Bruce Campbell
Information! Mal Warwick
Minimizing bias
by Bruce Campbell
WHAT GOOD is a survey if its very design skews the answers? The results can be worse than useless — they can be misleading. For this reason, it's critical that the designer of the questionnaire take pains to be certain there's no inherent bias in the questions. Let me outline some steps to take to avoid creating bias in your questionnaire.
1. Avoid unnecessary words that may stir emotions
The language of the questionnaire may cause reactions that distort the information you're looking for. For example, the word Democrat in the following question could stir the feelings of the respondent either for or against the legislation — regardless of its actual content. Do you feel that the Democratic-initiated House bill HB-119 is good?
2. Use objective descriptions
Sometimes background information is necessary. If so, ensure that it's objective. For example, this question could lead to biased answers: Building a performing arts complex will cost $5 million and bring great pride to our city. Do you think the necessary bond issue will be worthwhile?
On the one hand, mentioning that the project will cost $5 million is necessary for the respondent to determine whether the project is worthwhile. On the other hand, the statement that the project will "bring great pride to our city" is an opinion, and therefore it should be eliminated.
3. Avoid dual-issue questions
Sometimes a researcher will construct a question that simultaneously asks about two items, assuming that the answer to both will be the same. The respondent may not know how to answer the question. Moreover, the analyst will not be able to interpret the results of the answers the question does receive. Consider the following example: Are you comfortable with the frequency and tone of the museum's appeals?
People may be comfortable with the tone but not the frequency, or vice versa. On the back end, will the researcher be able to affirm with certainty how the respondents feel about each item? No. So the nonprofit will not know whether to change the tone, the frequency, or both.
4. Offer all reasonable possible responses
Respondents may quickly become frustrated if the answer that accurately reflects their opinion is not offered as a possibility. Consider this example: In what geographical regions do you feel it's most important for the ABC Nonprofit to concentrate its efforts?
1. Africa
2. Latin America
3. Eastern Europe
4. United States
5. Far East
Many regions — such as Australia, the Middle East, Canada, and the Indian subcontinent — are not offered among these possible answers. It would be better to form a more inclusive list or at least to offer "Other" as an option.
Here is another example. See if you can tell which response is missing: How successful do you feel the XYZ Nonprofit has been at fulfilling its goals?
1. Very successful
2. Somewhat successful
3. Not too successful
The missing response is "don't know." (In fact, almost 40% of the donors to major health charities give this answer.)
Including a "don't know" option in a mail survey is not usually necessary. If the respondent doesn't like the choices given, she's simply likely to skip the question.
Adapted from Listening to Your Donors, by Bruce Campbell (Jossey-Bass, 2000).
Bruce Campbell conducts marketing research for nonprofit organizations. He may be reached at Campbell Research, 218 West Carmen Lane #110, Santa Maria CA 93458, phone (805) 922-0880, fax (805) 922-3909, e-mail info@campbell-research.com.
--------------------------------------------------------------------------------
Information!
by Mal Warwick
ZOUNDS! I could hardly believe my eyes when the contents of my new-member welcome package from the National Rifle Association (Fairfax VA) tumbled out onto my desk. I, for one, have long advocated lavishing lots of information on new members or donors — because I strongly believe that course is most likely to lead to life-long donor relationships. But I had never before seen such a package crammed so full of information!
This fat, informative package includes the following items:
• A #10 white, double-window carrier printed front and back in black, blue, and red, with a pre-sort mailing indicia
• A two-page (one-sheet), 8-1/2 x 11" letter, with text printed in black, 12-point Courier and the logo, heading, and signature in dark blue
• An 8-1/2 x 11" "Confidential Member Survey" printed in red, black, and yellow on both sides, with a credit-card size plastic membership card printed in four colors
• A three-color folder (blue, yellow, and black) folded to 3-1/2 x 6" containing a postcard to be mailed to order a free NRA hat
• A 3-7/8 x 9" card printed in red and blue, separated by a line of perforations into a Business Reply Card and a tab bearing copy advertising the California Rifle and Pistol Association ("Join Now . . . Free!")
• A 24-page, full-color booklet headlined "NRA Membership Benefits & Service Guide," folded to 8-1/4 x 3-1/2"
• A white #9 envelope printed in black only on its front and containing a number of additional inserts: an application for NRA life insurance; a sheet containing two NRA decals; a card advertising real estate and home mortgage programs "Endorsed by NRA"; a Business Reply Card promoting discounted subscriptions to Gun Dog magazine; a small slip of paper advertising North American Van Lines moving services; and a six-panel brochure promoting the NRA's free "ArmsCare" insurance policy for firearms
• And a white #9 reply envelope addressed to the NRA's Membership Processing Center, printed in black only
Obviously, there's not enough space in this newsletter to illustrate the full contents of this elaborate package. We're showing only a few of the key components on these pages.
In any case, what we can all learn from this stellar example of direct mail is not necessarily how to write or design a better package (although the creative work is top-notch).
Rather, the lesson for all of us is how much care one of the most successful direct mail fundraisers in the country invests in cultivating, involving, and educating its new members or donors.
And by the way, just in case you're wondering why this inveterate gun-control advocate joined the NRA, you're looking at it! I'll take great direct mail, wherever it's from!
Congratulations and a Successful Direct Mail, Telephone & Online Fundraising Award Certificate to the National Rifle Association of America, 11250 Waples Mill Road, Fairfax VA 22030.
--------------------------------------------------------------------------------
Want more? Subscribe to Mal Warwick's Newsletter
