Cost-Effective Quality Assurance in Crowd Labeling

The emergence of online paid crowdsourcing platforms, such as Amazon Mechanical Turk (AMT), presents us huge opportunities to distribute tasks to human workers around the world, on-demand and at scale.  In such settings, online workers can come and complete tasks posted by a company, and work for as long or as little as they wish.  Given the absolute freedom of choice, crowdsourcing eliminates the overhead of the hiring (and dismissal) process.  However, this flexibility introduces a different set of inefficiencies: verifying the quality of every submitted piece of work is an expensive operation, which often requires the same level of effort as performing the task itself.  Many research challenges emerge in this paid-crowdsourcing setting.  How can we ensure that the submitted work is accurate?  How can we estimate the quality of the workers, and the quality of the submitted results?  How should we pay online workers that have imperfect quality?  We present a comprehensive scheme for managing quality of crowdsourcing processes:  First, we present an algorithm for estimating the quality of the participating workers and, by extension, of the generated data.  We show how we can separate systematic worker biases from unrecoverable errors and generate an unbiased “worker quality” measurement that can be used to objectively rank workers according to their performance.  Next, we describe a pricing scheme that identifies the fair payment level for a worker, adjusting the payment level according to the contributed information by each worker.  Our pricing policy, which pays workers based on their expected quality, reservation wage, and expected lifetime, estimates not only the payment level but also accommodates measurement uncertainties and allows the workers to receive a fair wage, even in the presence of temporary incorrect estimations of quality.  Our experimental results demonstrate that the proposed pricing strategy performs better than the commonly adopted uniform-pricing strategy.  We conclude the paper by describing strategies that build on our quality control and pricing framework, to build crowdsourced tasks of increasingly higher complexity, while still maintaining a tight quality control of the process, even if we allow participants of unknown quality to join the process.