EM-based inference of true labels using confidence judgments

Satoshi Oyama, Yuko Sakurai, Yukino Baba, Hisashi Kashima

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

We have developed a method for accurately inferring true labels from labels provided by crowdsourcing workers, with the aid of self-reported confidence judgments in their labels. Although confidence judgments can be useful information for estimating the quality of the provided labels, some workers are overconfident about the quality of their labels while others are underconfident. To address this problem, we extended the Dawid-Skene model and created a probabilistic model that considers the differences among workers in their accuracy of confidence judgments. Results of experiments using actual crowdsourced data showed that incorporating workers' confidence judgments can improve the accuracy of inferred labels.

Original languageEnglish
Title of host publicationHuman Computation and Crowdsourcing: Works in Progress and Demonstration Abstracts - An Adjunct to the Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing, Technical Report
PublisherAI Access Foundation
Pages58-59
Number of pages2
VolumeWS-13-18
ISBN (Print)9781577356318
Publication statusPublished - 2013
Event1st AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2013 - Palm Springs, CA, United States
Duration: Nov 6 2013Nov 9 2013

Other

Other1st AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2013
CountryUnited States
CityPalm Springs, CA
Period11/6/1311/9/13

    Fingerprint

All Science Journal Classification (ASJC) codes

  • Engineering(all)

Cite this

Oyama, S., Sakurai, Y., Baba, Y., & Kashima, H. (2013). EM-based inference of true labels using confidence judgments. In Human Computation and Crowdsourcing: Works in Progress and Demonstration Abstracts - An Adjunct to the Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing, Technical Report (Vol. WS-13-18, pp. 58-59). AI Access Foundation.