Approximating the crowd

Journal Article (Journal Article)

The problem of "approximating the crowd" is that of estimating the crowd's majority opinion by querying only a subset of it. Algorithms that approximate the crowd can intelligently stretch a limited budget for a crowdsourcing task. We present an algorithm, "CrowdSense," that works in an online fashion where items come one at a time. CrowdSense dynamically samples subsets of the crowd based on an exploration/exploitation criterion. The algorithm produces a weighted combination of the subset's votes that approximates the crowd's opinion. We then introduce two variations of CrowdSense that make various distributional approximations to handle distinct crowd characteristics. In particular, the first algorithm makes a statistical independence approximation of the labelers for large crowds, whereas the second algorithm finds a lower bound on how often the current subcrowd agrees with the crowd's majority vote. Our experiments on CrowdSense and several baselines demonstrate that we can reliably approximate the entire crowd's vote by collecting opinions from a representative subset of the crowd. © 2014 The Author(s).

Full Text

Duke Authors

Cited Authors

  • Ertekin, S; Rudin, C; Hirsh, H

Published Date

  • January 1, 2014

Published In

Volume / Issue

  • 28 / 5-6

Start / End Page

  • 1189 - 1221

International Standard Serial Number (ISSN)

  • 1384-5810

Digital Object Identifier (DOI)

  • 10.1007/s10618-014-0354-1

Citation Source

  • Scopus