Autonomous crowdsourcing through human-machine collaborative learning

Azad Abad, Moin Nabi, Alessandro Moschitti

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

In this paper, we introduce a general iterative human-machine collaborative method for training crowdsource workers: A classifier (i.e., the machine) selects the highest quality examples for training crowdsource workers (i.e., the humans). .Then, the la.er annotate the lower quality examples such that the classifier can be re-Trained with more accurate examples. This process can be iterated several times. We tested our approach on two di.erent tasks, Relation Extraction and Community .Thestion Answering, which are also in two di.erent languages, English and Arabic, respectively. Our experimental results show a significant improvement for creating Gold Standard data over distant supervision or just crowdsourcing without worker training. Additionally, our method can approach the performance of the state-of-The-Art methods that use expensive Gold Standard for training workers.

Original languageEnglish
Title of host publicationSIGIR 2017 - Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval
PublisherAssociation for Computing Machinery, Inc
Pages873-876
Number of pages4
ISBN (Electronic)9781450350228
DOIs
Publication statusPublished - 7 Aug 2017
Event40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2017 - Tokyo, Shinjuku, Japan
Duration: 7 Aug 201711 Aug 2017

Other

Other40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2017
CountryJapan
CityTokyo, Shinjuku
Period7/8/1711/8/17

Fingerprint

Classifiers

Keywords

  • Community
  • Crowdsourcing
  • Human in the Loop
  • Question Answering
  • Relation Extraction
  • Self-Training

ASJC Scopus subject areas

  • Information Systems
  • Software
  • Computer Graphics and Computer-Aided Design

Cite this

Abad, A., Nabi, M., & Moschitti, A. (2017). Autonomous crowdsourcing through human-machine collaborative learning. In SIGIR 2017 - Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 873-876). Association for Computing Machinery, Inc. https://doi.org/10.1145/3077136.3080666

Autonomous crowdsourcing through human-machine collaborative learning. / Abad, Azad; Nabi, Moin; Moschitti, Alessandro.

SIGIR 2017 - Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, Inc, 2017. p. 873-876.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abad, A, Nabi, M & Moschitti, A 2017, Autonomous crowdsourcing through human-machine collaborative learning. in SIGIR 2017 - Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, Inc, pp. 873-876, 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2017, Tokyo, Shinjuku, Japan, 7/8/17. https://doi.org/10.1145/3077136.3080666
Abad A, Nabi M, Moschitti A. Autonomous crowdsourcing through human-machine collaborative learning. In SIGIR 2017 - Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, Inc. 2017. p. 873-876 https://doi.org/10.1145/3077136.3080666
Abad, Azad ; Nabi, Moin ; Moschitti, Alessandro. / Autonomous crowdsourcing through human-machine collaborative learning. SIGIR 2017 - Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, Inc, 2017. pp. 873-876
@inproceedings{e94c85ca2d0b448f8f7969eb595c864a,
title = "Autonomous crowdsourcing through human-machine collaborative learning",
abstract = "In this paper, we introduce a general iterative human-machine collaborative method for training crowdsource workers: A classifier (i.e., the machine) selects the highest quality examples for training crowdsource workers (i.e., the humans). .Then, the la.er annotate the lower quality examples such that the classifier can be re-Trained with more accurate examples. This process can be iterated several times. We tested our approach on two di.erent tasks, Relation Extraction and Community .Thestion Answering, which are also in two di.erent languages, English and Arabic, respectively. Our experimental results show a significant improvement for creating Gold Standard data over distant supervision or just crowdsourcing without worker training. Additionally, our method can approach the performance of the state-of-The-Art methods that use expensive Gold Standard for training workers.",
keywords = "Community, Crowdsourcing, Human in the Loop, Question Answering, Relation Extraction, Self-Training",
author = "Azad Abad and Moin Nabi and Alessandro Moschitti",
year = "2017",
month = "8",
day = "7",
doi = "10.1145/3077136.3080666",
language = "English",
pages = "873--876",
booktitle = "SIGIR 2017 - Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval",
publisher = "Association for Computing Machinery, Inc",

}

TY - GEN

T1 - Autonomous crowdsourcing through human-machine collaborative learning

AU - Abad, Azad

AU - Nabi, Moin

AU - Moschitti, Alessandro

PY - 2017/8/7

Y1 - 2017/8/7

N2 - In this paper, we introduce a general iterative human-machine collaborative method for training crowdsource workers: A classifier (i.e., the machine) selects the highest quality examples for training crowdsource workers (i.e., the humans). .Then, the la.er annotate the lower quality examples such that the classifier can be re-Trained with more accurate examples. This process can be iterated several times. We tested our approach on two di.erent tasks, Relation Extraction and Community .Thestion Answering, which are also in two di.erent languages, English and Arabic, respectively. Our experimental results show a significant improvement for creating Gold Standard data over distant supervision or just crowdsourcing without worker training. Additionally, our method can approach the performance of the state-of-The-Art methods that use expensive Gold Standard for training workers.

AB - In this paper, we introduce a general iterative human-machine collaborative method for training crowdsource workers: A classifier (i.e., the machine) selects the highest quality examples for training crowdsource workers (i.e., the humans). .Then, the la.er annotate the lower quality examples such that the classifier can be re-Trained with more accurate examples. This process can be iterated several times. We tested our approach on two di.erent tasks, Relation Extraction and Community .Thestion Answering, which are also in two di.erent languages, English and Arabic, respectively. Our experimental results show a significant improvement for creating Gold Standard data over distant supervision or just crowdsourcing without worker training. Additionally, our method can approach the performance of the state-of-The-Art methods that use expensive Gold Standard for training workers.

KW - Community

KW - Crowdsourcing

KW - Human in the Loop

KW - Question Answering

KW - Relation Extraction

KW - Self-Training

UR - http://www.scopus.com/inward/record.url?scp=85029351052&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85029351052&partnerID=8YFLogxK

U2 - 10.1145/3077136.3080666

DO - 10.1145/3077136.3080666

M3 - Conference contribution

SP - 873

EP - 876

BT - SIGIR 2017 - Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

PB - Association for Computing Machinery, Inc

ER -