EM-based inference of true labels using confidence judgments

Satoshi Oyama, Yuko Sakurai, Yukino Baba, Hisashi Kashima

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

We have developed a method for accurately inferring true labels from labels provided by crowdsourcing workers, with the aid of self-reported confidence judgments in their labels. Although confidence judgments can be useful information for estimating the quality of the provided labels, some workers are overconfident about the quality of their labels while others are underconfident. To address this problem, we extended the Dawid-Skene model and created a probabilistic model that considers the differences among workers in their accuracy of confidence judgments. Results of experiments using actual crowdsourced data showed that incorporating workers' confidence judgments can improve the accuracy of inferred labels.

Original languageEnglish
Title of host publicationHuman Computation and Crowdsourcing: Works in Progress and Demonstration Abstracts - An Adjunct to the Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing, Technical Report
PublisherAI Access Foundation
Pages58-59
Number of pages2
VolumeWS-13-18
ISBN (Print)9781577356318
Publication statusPublished - 2013
Event1st AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2013 - Palm Springs, CA, United States
Duration: Nov 6 2013Nov 9 2013

Other

Other1st AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2013
CountryUnited States
CityPalm Springs, CA
Period11/6/1311/9/13

Fingerprint

Labels
Experiments

All Science Journal Classification (ASJC) codes

  • Engineering(all)

Cite this

Oyama, S., Sakurai, Y., Baba, Y., & Kashima, H. (2013). EM-based inference of true labels using confidence judgments. In Human Computation and Crowdsourcing: Works in Progress and Demonstration Abstracts - An Adjunct to the Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing, Technical Report (Vol. WS-13-18, pp. 58-59). AI Access Foundation.

EM-based inference of true labels using confidence judgments. / Oyama, Satoshi; Sakurai, Yuko; Baba, Yukino; Kashima, Hisashi.

Human Computation and Crowdsourcing: Works in Progress and Demonstration Abstracts - An Adjunct to the Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing, Technical Report. Vol. WS-13-18 AI Access Foundation, 2013. p. 58-59.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Oyama, S, Sakurai, Y, Baba, Y & Kashima, H 2013, EM-based inference of true labels using confidence judgments. in Human Computation and Crowdsourcing: Works in Progress and Demonstration Abstracts - An Adjunct to the Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing, Technical Report. vol. WS-13-18, AI Access Foundation, pp. 58-59, 1st AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2013, Palm Springs, CA, United States, 11/6/13.
Oyama S, Sakurai Y, Baba Y, Kashima H. EM-based inference of true labels using confidence judgments. In Human Computation and Crowdsourcing: Works in Progress and Demonstration Abstracts - An Adjunct to the Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing, Technical Report. Vol. WS-13-18. AI Access Foundation. 2013. p. 58-59
Oyama, Satoshi ; Sakurai, Yuko ; Baba, Yukino ; Kashima, Hisashi. / EM-based inference of true labels using confidence judgments. Human Computation and Crowdsourcing: Works in Progress and Demonstration Abstracts - An Adjunct to the Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing, Technical Report. Vol. WS-13-18 AI Access Foundation, 2013. pp. 58-59
@inproceedings{1cfff3f5e6484e09b59e929571a022f2,
title = "EM-based inference of true labels using confidence judgments",
abstract = "We have developed a method for accurately inferring true labels from labels provided by crowdsourcing workers, with the aid of self-reported confidence judgments in their labels. Although confidence judgments can be useful information for estimating the quality of the provided labels, some workers are overconfident about the quality of their labels while others are underconfident. To address this problem, we extended the Dawid-Skene model and created a probabilistic model that considers the differences among workers in their accuracy of confidence judgments. Results of experiments using actual crowdsourced data showed that incorporating workers' confidence judgments can improve the accuracy of inferred labels.",
author = "Satoshi Oyama and Yuko Sakurai and Yukino Baba and Hisashi Kashima",
year = "2013",
language = "English",
isbn = "9781577356318",
volume = "WS-13-18",
pages = "58--59",
booktitle = "Human Computation and Crowdsourcing: Works in Progress and Demonstration Abstracts - An Adjunct to the Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing, Technical Report",
publisher = "AI Access Foundation",
address = "United States",

}

TY - GEN

T1 - EM-based inference of true labels using confidence judgments

AU - Oyama, Satoshi

AU - Sakurai, Yuko

AU - Baba, Yukino

AU - Kashima, Hisashi

PY - 2013

Y1 - 2013

N2 - We have developed a method for accurately inferring true labels from labels provided by crowdsourcing workers, with the aid of self-reported confidence judgments in their labels. Although confidence judgments can be useful information for estimating the quality of the provided labels, some workers are overconfident about the quality of their labels while others are underconfident. To address this problem, we extended the Dawid-Skene model and created a probabilistic model that considers the differences among workers in their accuracy of confidence judgments. Results of experiments using actual crowdsourced data showed that incorporating workers' confidence judgments can improve the accuracy of inferred labels.

AB - We have developed a method for accurately inferring true labels from labels provided by crowdsourcing workers, with the aid of self-reported confidence judgments in their labels. Although confidence judgments can be useful information for estimating the quality of the provided labels, some workers are overconfident about the quality of their labels while others are underconfident. To address this problem, we extended the Dawid-Skene model and created a probabilistic model that considers the differences among workers in their accuracy of confidence judgments. Results of experiments using actual crowdsourced data showed that incorporating workers' confidence judgments can improve the accuracy of inferred labels.

UR - http://www.scopus.com/inward/record.url?scp=84899572862&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84899572862&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84899572862

SN - 9781577356318

VL - WS-13-18

SP - 58

EP - 59

BT - Human Computation and Crowdsourcing: Works in Progress and Demonstration Abstracts - An Adjunct to the Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing, Technical Report

PB - AI Access Foundation

ER -