Impact of demographic survey questions on response rate and measurement: a randomized experiment
- View All
Demographic survey questions are important to describe the population of survey responders, illuminate potential disparities, and ultimately advance equity. Little is known about their impact on survey response rate or measurement. Methods: A total of 4,448 individuals were randomly assigned to one of three conditions in a mailed paper questionnaire where demographic questions were (1) not asked, (2) integrated at the end of the survey, or (3) included as standalone questions on a separate piece of paper. Response rates to the main survey and demographic questions, as well as item nonresponse and correlation of responses to administrative records, are compared. Results: Overall, 33.4% of individuals who were mailed the survey responded. There were no substantive or statistical differences in survey response rate when demographic questions were not asked (34.2%), were integrated into the survey (33.1%), or were standalone (33.0%; p = 0.762). Sampled individuals responded to the demographic questions at a significantly higher rate when they were integrated into the main survey (32.7%) compared to when they were standalone (28.3%). Respondents, when asked about income, declined to answer at a significantly higher rate when demographics were integrated (16.5%) compared to standalone (10.5%). Discordance between administrative and self-reported race and ethnicity data ranged from 0.6% to 1.0% and were not statistically different across arms (p = 0.64 and p = 0.88, respectively). Discussion: While these findings are limited to the context of the experiment, our results suggest that embedding demographic questions in a survey (as opposed to on a separate page) may result in more usable demographic data. Future work could explore the differential impact of post-survey missing data adjustments on estimates of demographic characteristics and correlation with other survey content. Overall, there was little measurement error in reporting of race/ethnicity in both conditions. Conclusion: For collection of demographic data from the largest portion of individuals via a mailed survey without negative impact on response rate or measurement error, demographic questions are best integrated into surveys rather than included as standalone items on a separate piece of paper.
Link to Article