Flexible reward plans for crowdsourced tasks

Yuko Sakurai, Masato Shinoda, Satoshi Oyama, Makoto Yokoo

研究成果: 書籍/レポート タイプへの寄稿会議への寄与

1 被引用数 (Scopus)

抄録

We develop flexible reward plans to elicit truthful predictive probability distribution over a set of uncertain events from workers. In general, strictly proper scoring rules for categorical events only reward a worker for an event that actually occurred. However, different incorrect predictions vary in quality, and the principal would like to assign different rewards to them, according to her subjective similarity among events; e.g. a prediction of overcast is closer to sunny than rainy. We propose concrete methods so that the principal can assign rewards for incorrect predictions according to her similarity between events. We focus on two representative examples of strictly proper scoring rules: spherical and quadratic, where a worker’s expected utility is represented as the inner product of her truthful predictive probability and her declared probability. In this paper, we generalize the inner product by introducing a reward matrix that defines a reward for each prediction outcome pair. We first show that if the reward matrix is symmetric and positive definite, both the spherical and quadratic proper scoring rules guarantee the maximization of a worker’s expected utility when she truthfully declares her prediction. We next compare our rules with the original spherical/quadratic proper scoring rules in terms of the variance of rewards obtained by workers. Finally, we show our experimental results using Amazon Mechanical Turk.

本文言語英語
ホスト出版物のタイトルPRIMA 2015
ホスト出版物のサブタイトルPrinciples and Practice of Multi-Agent Systems - 18th International Conference, Proceedings
編集者Paolo Torroni, Andrea Omicini, Jane Hsu, Qingliang Chen, Paolo Torroni, Andrea Omicini, Jane Hsu, Qingliang Chen, Serena Villata, Serena Villata
出版社Springer Verlag
ページ400-415
ページ数16
ISBN(印刷版)9783319255231, 9783319255231
DOI
出版ステータス出版済み - 2015
イベント18th International Conference on Principles and Practice of Multi-Agent Systems, PRIMA 2015 - Bertinoro, イタリア
継続期間: 10月 26 201510月 30 2015

出版物シリーズ

名前Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
9387
ISSN(印刷版)0302-9743
ISSN(電子版)1611-3349

その他

その他18th International Conference on Principles and Practice of Multi-Agent Systems, PRIMA 2015
国/地域イタリア
CityBertinoro
Period10/26/1510/30/15

!!!All Science Journal Classification (ASJC) codes

  • 理論的コンピュータサイエンス
  • コンピュータ サイエンス(全般)

フィンガープリント

「Flexible reward plans for crowdsourced tasks」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル