Suppose you send a survey to 1,000 employees and 300 respond. You analyse the data and report that 70% of employees are satisfied with their training opportunities. But what about the 700 people who did not respond? If they were less satisfied, too busy because of poor working conditions, or simply disengaged, your 70% figure may be far too optimistic.
This is the essence of non-response bias. It is one of the most important threats to the validity of survey research, and examiners and journal reviewers frequently ask how you addressed it. This post explains what non-response bias is, how it differs from a low response rate, the main methods for checking it, and how to report your checks honestly.
What is non-response bias?
Non-response bias occurs when the people who do not respond to a survey differ systematically from those who do, on the variables you are studying, so that your results do not accurately represent the target population.
There are two main types of non-response:
- Unit non-response: a selected person does not complete the survey at all.
- Item non-response: a person completes the survey but skips particular questions.
Both can create bias. This post focuses mainly on unit non-response, but item non-response is discussed briefly later.
Response rate and non-response bias are not the same thing
A common misunderstanding is that a low response rate automatically means biased results, and a high response rate means unbiased results. The relationship is more complex.
Bias depends on two things together:
- How many people did not respond.
- How different non-respondents are from respondents on the variables of interest.
If non-respondents are very similar to respondents on your key variables, a low response rate may cause little bias. If they are very different, even a fairly high response rate can leave meaningful bias. Methodological research on survey non-response has shown that response rates alone are often a weak predictor of bias. This is why checking for bias directly is more informative than simply reporting a response rate.
That said, a higher response rate limits how much bias is possible, so improving response is still valuable.
Why non-response bias happens
People decide not to respond for many reasons, and some of these reasons are related to the topic of the survey. For example:
- People with strong opinions, positive or negative, may be more motivated to respond.
- Busy or stressed people may be less likely to respond to a survey about workload or stress.
- People who are less satisfied with an organisation may not trust an organisational survey.
- People with lower literacy or limited internet access may be less likely to complete certain survey modes.
- Younger or older groups may respond at different rates depending on the channel used.
When the reason for non-response is linked to the variables you measure, bias is more likely.
Methods to check for non-response bias
No single method can prove that your data are free of non-response bias, because by definition you have limited information about non-respondents. However, several methods give useful evidence. Using more than one method strengthens your case.
1. Compare respondents with known population characteristics
If you know characteristics of the full population, such as age, gender, department, job level, region or school type, compare these with the characteristics of your respondents. Sources can include organisational records, official statistics, membership lists or census data.
For example, if 40% of employees in the organisation are in operations but only 15% of your respondents are, operations staff are under-represented. You can test differences with a chi-square goodness-of-fit test or simply present a comparison table.
This method is transparent and easy to report. Its limitation is that similarity on demographics does not guarantee similarity on attitudes or behaviours.
2. Compare early and late respondents (wave analysis)
A widely used approach, often linked to work by Armstrong and Overton (1977), assumes that people who respond late, for example after reminders, are more similar to non-respondents than early respondents are. You split respondents into early and late groups (for example, those who responded before the first reminder and those who responded after the final reminder) and compare them on key variables using t-tests or chi-square tests.
If there are no meaningful differences, this provides some reassurance. If late respondents differ, non-respondents may differ even more.
This method is easy to apply and is commonly reported, especially in management and marketing research. However, its key assumption, that late respondents resemble non-respondents, is not always true, so present it as partial evidence rather than proof.
3. Follow up a sample of non-respondents
You can contact a small random sample of non-respondents and ask them to answer a short version of the survey, perhaps just a few key questions, sometimes through a different mode such as phone. Comparing their answers with those of respondents gives direct evidence about possible bias.
This is one of the strongest methods but requires extra time, ethics approval and the ability to identify non-respondents, which is not possible in fully anonymous surveys.
4. Use information from the sampling frame
If your sampling frame includes information about everyone invited, such as organisation size, industry or location for a business survey, you can compare respondents and non-respondents directly on those variables. Logistic regression can be used to see which characteristics predict responding.
5. Compare with benchmark studies
Compare your key results with findings from large, high-quality surveys of similar populations. Large differences may signal bias, though they could also reflect real differences in context or time.
6. Examine response patterns by subgroup
If you can calculate response rates for different subgroups, such as departments or regions, look for groups with much lower response. Consider whether those groups might differ on your variables.
What to do if you find evidence of bias
Weighting
If certain groups are under-represented and you know their true population proportions, you can apply weights so that the sample better matches the population. For example, if women make up 50% of the population but 30% of respondents, their responses can be weighted up. Weighting can reduce bias on variables related to the weighting characteristics, but it cannot fix bias on unrelated variables, and heavy weighting increases variance. Seek statistical advice before weighting.
Targeted follow-up
During data collection, if you see that a group is under-represented, you can send targeted reminders or use additional channels to reach them, provided this is within your ethics approval.
Careful interpretation
If you cannot correct bias, be cautious in your conclusions. Avoid generalising beyond the groups that are well represented, and discuss the likely direction of bias. For instance: “Because staff with heavier workloads may have been less likely to respond, our estimates of workload stress may be conservative.”
Item non-response
Item non-response occurs when respondents skip particular questions. It is common for sensitive questions such as income, or for items at the end of long questionnaires.
To handle it:
- Report the amount of missing data for each key variable.
- Look for patterns: are certain groups more likely to skip certain questions?
- Consider whether data are likely missing completely at random, missing at random, or missing not at random.
- Choose an appropriate handling method. Listwise deletion is simple but can reduce sample size and introduce bias. Methods such as multiple imputation or full information maximum likelihood are often recommended when assumptions are met.
Prevention is best: keep surveys short, place sensitive questions carefully, and pre-test to identify confusing items.
How to report non-response bias checks
A good report in your methodology or results chapter includes:
- The number invited, number of responses, and how the response rate was calculated.
- The methods used to assess non-response bias.
- Results of those checks, with test statistics or comparison tables.
- Any corrective actions, such as weighting.
- An honest statement of remaining limitations.
For example: “Of 1,000 employees invited, 312 completed the survey (31.2%). Respondents were similar to the full workforce in gender and job level but slightly under-represented operations staff. Comparing early respondents (n = 198) and late respondents (n = 114) on all study variables showed no statistically significant differences. However, because late respondents may not fully represent non-respondents, some non-response bias cannot be ruled out.” (Numbers are illustrative.)
Common mistakes to avoid
- Assuming that a “good” response rate means there is no bias.
- Relying only on early-versus-late comparisons without acknowledging their limits.
- Comparing respondents with the population only on variables unrelated to your research questions.
- Ignoring item non-response and reporting results for different sample sizes without explanation.
- Not mentioning non-response bias at all in the limitations section.
A simple checklist
- I know how many people were invited and how many responded.
- I have compared respondents with the population on available characteristics.
- I have compared early and late respondents on key variables.
- I have considered follow-up of non-respondents where feasible.
- I have examined and reported item-level missing data.
- I have considered weighting if there are clear imbalances.
- I have discussed the possible direction and impact of bias honestly.
Final thoughts
Non-response bias is not something you can eliminate completely, but you can reduce it, assess it and discuss it responsibly. Focus on maximising response, then use several methods to check whether respondents differ from the population or from likely non-respondents. Report what you found, including the uncomfortable parts. Transparent reporting of non-response is a sign of rigorous research, and it gives your readers the information they need to judge how far your findings can be generalised.
How are you checking for non-response bias in your study? Share your approach or questions in the comments below.