Everyone has completed a survey at some point in their lives. Because of everyone’s familiarity with surveys, they are a popular choice for evaluation methodology. There are tons of entire books devoted to how to design, administer, analyze, and report on surveys. The one I used for my PhD in Evaluation coursework – and was used for this post is Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method.
1. Surveys are powerful. To estimate preferences within 5% of 100 million voters, you’d only need a RANDOM sample of 400 voters.
2. Surveys can be done online, over the phone, or as a paper survey. Online surveys are great because they are inexpensive to administer and provide real-time responses and the ability to modify your survey quickly if you discover a mistake or see a need to adjust a question to increase the quality of your responses. There can be issues with online surveys being skewed towards younger demographics (though this isn’t as common anymore). Additionally, there can be issues with quality control as people can take a survey more than once and you cannot control who takes the survey via your link reducing the likelihood of a quality sample. Telephone surveys can increase your response rate and increase comprehension of questions (since the surveyor can explain what question means in real-time). They can be more costly due to labor involved – or high fees for automated calling systems. Also, there is some potential for interview bias (i.e. some people may be more comfortable disclosing certain information to a male versus a female) – and it is hard to get a quality sample since many people no longer have landlines. Mailed paper surveys will cost more than online surveys due to printing and postage – but they do allow respondents to answer at their own convenience similar to online surveys. They can gather large amounts of information and no interviewer bias is introduced as with telephone surveys. Although, their is often long time delays before surveys are received and they aren’t a good fit for questions that might require further clarification.
3. Your items (i.e. survey questions) will be well-written and tested. Surveys are not the place to add in extra questions that would be “nice to know.” For every extra question you’ll reduce your response rate. People are over-surveyed already, so keep it as concise as possible while still gathering the necessary information. You should use simply language and identify the need for every question. Put personal/demographic questions at the end of the survey! Try to avoid an abundance of open-ended questions, and combine categories when possible – but make sure they are appropriate.
4. To ensure the highest possible response rate, you would follow social exchange theory. Which means you would have a respondent-friendly survey, have four contacts by mail (pre-notice letter, survey, postcard thank you/reminder, follow up with replacement surveys for those that have lost them/special reminder), postage-paid return envelopes for respondents to return surveys in, personalized correspondence, and a token prepaid financial incentive.
5. Be transparent and try to get the best data possible. If you want your survey to be generalizable, you’ll need to have a sufficient response rate (surveys completed/surveys mailed = response rate) for your sample size. It will need to be a random sample. If you can’t get a random sample, then you should try to get a sample that replicates the population of participants as closely as possible so that you can make inferences about the survey results. As always, be transparent in your reports and explain how you got your data and what type of sample you have.
Next, 30 Days to Quality Evaluation: Focus Groups and the 5 most important things you need to know
Photo Credit: The Bees
