What is response bias?
Response bias is a statistical term that describes a simple issue: People can’t always answer surveys honestly and accurately. This can obviously skew your data and bring the wrong results, and it usually isn’t because anyone wants to mess with your results. Response bias is just the result of natural human traits.
Understanding response bias can help you to write better surveys that get the most accurate and honest data possible. There is a lot of academic discussion on response bias out there. As a science writer, I can attest that it isn’t always easy to digest all of the wordy academic jargon on this basic topic. This blog aims to break response bias down into practical terms, so that you know what you're dealing with when you create your next survey.
In this post, we’ll cover five common types of response bias. The fact is that understanding response bias on a practical level is not that complicated, and it can help you to get better results with your own survey and data collection efforts.
Why it happens
There are different reasons that people can’t give accurate answers to survey questions. These are all well established reasons for response bias that scientists have nailed down with years of research.
1. People are inclined to say “yes”
Acquiescence bias is when respondents tend to answer “yes” to whatever question they are asked. The will say that they prefer to be alone, and two minutes later they will say that they prefer the company of others. They aren’t lying, but when they read the questions they think of specific experiences that relate to it. When they think of being alone they recall the time when the family was out for the evening and they had a great time reading that book in peace in the bathtub. When they think of spending time with others they think of that great Christmas where all of the cousins came over and it was a wonderful moment for the whole family. Respondents are trying to answer honestly, but their mind comes up with scenarios which help them answer with a “yes tendency” to every question. Their answers are based off of the specific experiences their mind recalls at the time, as opposed to their general attitudes in life.
2. People act differently when they are taking part in research
Demand Characteristics are when people give skewed answers simply because they are taking part in a survey, so their disposition changes slightly. They aren’t lying or trying to mess with your results. Quite the opposite. Demand characteristics are born out of an altruistic social pressure to be a ‘good respondent.’ In some cases people will deem that the result of the survey is important (maybe it will lead to a raise or an improvement in company culture) and because they deem the research to be important, they will tell you what they think you want to hear in order to support the ‘right results’, as opposed to the ones which are true for them personally.
3. Some people tend to use extreme ends of the scale
Extreme Responding is when people don’t use the scale accurately and tend to only use the extreme positive or negative ends. On a scale of 1-5, they will always choose 1 or 5. Research has shown that this is often linked to certain cultural attitudes. Education and IQ can also effect the rate of extreme response. There are ways to normalize answers in order to take this type of bias into account. For example, Z scoring is a statistical method that normalizes raw data into a more comprehensive and accurate report. It can be particularly helpful in balancing out extreme response bias.
4. People want to appear like they are “good people”
Social Desirability Bias is where people feel pressure to answer in ways that make them seem like a “good person.” People will often under report on their negative tendencies while over reporting on positive ones. Personal habits like drinking or smoking levels may fall victim to this kind of response bias, as well as sensitive issues like racism or sexism. In both cases, people can feel obligated to present themselves in a better light than they actually stand.
5. Different people interpret scales differently
Sometimes people literally just have a different understanding of a scale. Some people might think that 7 is the highest reasonable answer that they can possibly give, while others might recognize that the number 10 is there for a reason. Not all participants will use the entire scale correctly due to a variety of reasons. Understand that relating your opinions and feelings to a number scale just isn’t that easy. One way to avoid this misinterpretation of scale is by using words as opposed to numbers. For example, consider using “very bad-very good” as opposed to "a scale of 1-5".
The interesting thing about all different styles of response bias is that people are not usually trying to lie or mislead you.
Response bias is a natural part of our human tendencies. Knowing why people answer ‘dishonestly’ is the first step towards writing and conducting surveys that keep response bias to a minimum. The good news is that there is a lot you can do to get accurate and honest data. Next week, you can read our follow-up blog post, 5 Tips for Avoiding Response Bias where you can learn more about what you can do to improve your data collection. You can also download our free survey guide below to learn best practices for writing great questions and analyzing your data in the smartest ways possible.