Accurate integration of crowdsourced labels using workers' self-reported confidence scores

Satoshi Oyama, Yukino Baba, Yuko Sakurai, Hisashi Kashima

Research output: Chapter in Book/Report/Conference proceedingConference contribution

30 Citations (Scopus)

Abstract

We have developed a method for using confidence scores to integrate labels provided by crowdsourcing workers. Although confidence scores can be useful information for estimating the quality of the provided labels, a way to effectively incorporate them into the integration process has not been established. Moreover, some workers are overconfident about the quality of their labels while others are underconfident, and some workers are quite accurate in judging the quality of their labels. This differing reliability of the confidence scores among workers means that the probability distributions for the reported confidence scores differ among workers. To address this problem, we extended the Dawid-Skene model and created two probabilistic models in which the values of unobserved true labels are inferred from the observed provided labels and reported confidence scores by using the expectation-maximization algorithm. Results of experiments using actual crowdsourced data for image labeling and binary question answering tasks showed that incorporating workers' confidence scores can improve the accuracy of integrated crowdsourced labels.

Original languageEnglish
Title of host publicationIJCAI 2013 - Proceedings of the 23rd International Joint Conference on Artificial Intelligence
Pages2554-2560
Number of pages7
Publication statusPublished - 2013
Event23rd International Joint Conference on Artificial Intelligence, IJCAI 2013 - Beijing, China
Duration: Aug 3 2013Aug 9 2013

Other

Other23rd International Joint Conference on Artificial Intelligence, IJCAI 2013
CountryChina
CityBeijing
Period8/3/138/9/13

    Fingerprint

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence

Cite this

Oyama, S., Baba, Y., Sakurai, Y., & Kashima, H. (2013). Accurate integration of crowdsourced labels using workers' self-reported confidence scores. In IJCAI 2013 - Proceedings of the 23rd International Joint Conference on Artificial Intelligence (pp. 2554-2560)