ML p(r)ior | Multi-object Classification via Crowdsourcing with a Reject Option

Multi-object Classification via Crowdsourcing with a Reject Option

2016-02-01
Consider designing an effective crowdsourcing system for an $M$-ary classification task. Crowd workers complete simple binary microtasks whose results are aggregated to give the final result. We consider the novel scenario where workers have a reject option so they may skip microtasks when they are unable or choose not to respond. For example, in mismatched speech transcription, workers who do not know the language may not be able to respond to microtasks focused on phonological dimensions outside their categorical perception. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize the crowd's classification performance. We evaluate system performance in both exact and asymptotic forms. Further, we consider the setting where there may be a set of greedy workers that complete microtasks even when they are unable to perform it reliably. We consider an oblivious and an expurgation strategy to deal with greedy workers, developing an algorithm to adaptively switch between the two based on the estimated fraction of greedy workers in the anonymous crowd. Simulation results show improved performance compared with conventional majority voting.
PDF

Highlights - Most important sentences from the article

Login to like/save this paper, take notes and configure your recommendations

Related Articles

2019-02-25

Interpretability has always been a major concern for fuzzy rule-based classifiers. The usage of huma… show more
PDF

Highlights - Most important sentences from the article

2019-01-28

Classification may not be reliable for several reasons: noise in the data, insufficient input inform… show more
PDF

Highlights - Most important sentences from the article

2017-06-06

One possible approach to tackle the class imbalance in classification tasks is to resample a trainin… show more
PDF

Highlights - Most important sentences from the article

2019-05-02

Truth discovery is a general name for a broad range of statistical methods aimed to extract the corr… show more
PDF

Highlights - Most important sentences from the article

2019-01-07

The problem of designing bit-to-pattern mappings and power allocation schemes for orthogonal frequen… show more
PDF

Highlights - Most important sentences from the article

2018-10-13

Chromosome classification is critical for karyotyping in abnormality diagnosis. To expedite the diag… show more
PDF

Highlights - Most important sentences from the article

2019-01-27

Gradient descent algorithms are widely used in machine learning. In order to deal with huge volume o… show more
PDF

Highlights - Most important sentences from the article

2019-04-25

We consider worker skill estimation for the single-coin Dawid-Skene crowdsourcing model. In practice… show more
PDF

Highlights - Most important sentences from the article

2018-08-08
1808.02838 | cs.DC

We study the expected completion time of some recently proposed algorithms for distributed computing… show more
PDF

Highlights - Most important sentences from the article

2018-12-05
1812.02736 | cs.HC

Online crowdsourcing provides a scalable and inexpensive means to collect knowledge (e.g. labels) ab… show more
PDF

Highlights - Most important sentences from the article

2018-04-27

Large-scale machine learning and data mining applications require computer systems to perform massiv… show more
PDF

Highlights - Most important sentences from the article

2017-05-10

In mobile crowdsourcing (MCS), mobile users accomplish outsourced human intelligence tasks. MCS requ… show more
PDF

Highlights - Most important sentences from the article

2018-12-07

The k-nearest-neighbor method performs classification tasks for a query sample based on the informat… show more
PDF

Highlights - Most important sentences from the article

2019-02-24

Crowd-sourcing is a cheap and popular means of creating training and evaluation datasets for machine… show more
PDF

Highlights - Most important sentences from the article

2019-04-16

Coded distributed computing framework enables large-scale machine learning (ML) models to be trained… show more
PDF

Highlights - Most important sentences from the article

2017-12-08

The rising interest in pattern recognition and data analytics has spurred the development of innovat… show more
PDF

Highlights - Most important sentences from the article