Professional respondents in nonprobability online panels

It is well-documented that there exists a pool of frequent survey takers who participate in many different online nonprobability panels in order to earn cash or other incentives--so-called 'professional' respondents. Despite widespread concern about the impact of these professional respondents on data quality, there is not a clear understanding of how they might differ from other respondents. This chapter reviews the previous research and expectations regarding professional respondents and then examines how frequent survey taking and multiple panel participation affects data quality in the 2010 Cooperative Congressional Election Study. In contrast to common assumptions, we do not find overwhelming and consistent evidence that frequent survey takers are more likely to satisfice. On the contrary, frequent survey takers spent more time completing the questionnaire, were less likely to attrite, were less likely to straightline, and reported putting more effort into answering the survey. While panel memberships and number of surveys completed were related to skipping questions, answering "don't know," or giving junk responses to open-ended questions, these relationships did not hold once we account for levels of political knowledge. However, our analysis finds that higher levels of participation in surveys and online panels are associated with lower levels of political knowledge, interest, engagement, and ideological extremism. These findings suggest there could be contrasting motivations for those volunteering to participate in nonprobability panel surveys, with professional respondents taking part for the incentives and nonprofessional respondents taking part based on interest in the survey topic. As such, eliminating professional respondents from survey estimates, as some have recommended, would actually result in a more biased estimate of political outcomes.

Duke Scholars

Author D. Sunshine Hillygus Political Science