Design and pre-test questionnaires: Designing the questionnaire carefully and then pre-testing it before fielding it to your entire sample is crucial to getting data that are valid and reliable. For example, careful questionnaire design and pre-testing can help reduce the chance that respondents may interpret the meaning of questions differently. Future posts in this series will tackle these important steps in much greater detail.
One of the most common formats used in survey questions is the “agree-disagree” format. In this type of question, respondents are asked whether they agree or disagree with a particular statement. Research has shown that, compared with the better educated and better informed, less educated and less informed respondents have a greater tendency to agree with such statements. This is sometimes called an “acquiescence bias” (since some kinds of respondents are more likely to acquiesce to the assertion than are others). A better practice is to offer respondents a choice between alternative statements. A Pew Research Center experiment with one of its routinely asked values questions illustrates the difference that question format can make. Not only does the forced choice format yield a very different result overall from the agree-disagree format, but the pattern of answers among better- and lesser-educated respondents also tends to be very different.
“I participated in an in-home product trial study. A coffee machine company sent me one of the coffee makers along with about 200 coffee pods. I received $7 for completing 3 short surveys. Plus, after the study, the company said I could keep the coffee maker. So, I got like 4 mos of coffee for everyone in my household, a free coffee maker and $7!” - David W.