Quality-control mechanism utilizing Worker's confidence for crowdsourced tasks

Yuko Sakurai, Tenda Okimoto, Masaaki Oka, Masato Shinoda, Makoto Yokoo

研究成果: 会議への寄与タイプ論文

3 引用 (Scopus)

抜粋

We propose a quality control mechanism that utilizes workers' self-reported confidences in crowdsourced labeling tasks. Generally, a worker has confidence in the correctness of her answers, and asking about it is useful for estimating the probability of correctness. However, we need to overcome two main obstacles in order to to use confidence for inferring correct answers. First, a worker is not always well-calibrated. Since she is sometimes over/underconfident, her level of confidence does not always accurately reflect the probability of correctness. In addition, she does not always truthfully report her actual confidence. Therefore, we design an indirect mechanism that enables a worker to declare her confidence by choosing a desirable reward plan from the set of plans that correspond to different confidence intervals. Our mechanism ensures that choosing the plan matching the worker's true confidence maximizes her expected utility.

元の言語英語
ページ1347-1348
ページ数2
出版物ステータス出版済み - 1 1 2013
イベント12th International Conference on Autonomous Agents and Multiagent Systems 2013, AAMAS 2013 - Saint Paul, MN, 米国
継続期間: 5 6 20135 10 2013

その他

その他12th International Conference on Autonomous Agents and Multiagent Systems 2013, AAMAS 2013
米国
Saint Paul, MN
期間5/6/135/10/13

    フィンガープリント

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence

これを引用

Sakurai, Y., Okimoto, T., Oka, M., Shinoda, M., & Yokoo, M. (2013). Quality-control mechanism utilizing Worker's confidence for crowdsourced tasks. 1347-1348. 論文発表場所 12th International Conference on Autonomous Agents and Multiagent Systems 2013, AAMAS 2013, Saint Paul, MN, 米国.