Taking the best from the Crowd

Learning Question Passage Classification from Noisy Data

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

In this paper, we propose methods to take into account the disagreement between crowd annotators as well as their skills for weighting instances in learning algorithms. The latter can thus better deal with noise in the annotation and produce higher accuracy. We created two passage reranking datasets: one with crowdsource platform, and the second with an expert who completely revised the crowd annotation. Our experiments show that our weighting approach reduces noise improving passage reranking up to 1.47% and 1.85% on MRR and P@1, respectively.

Original languageEnglish
Title of host publication*SEM 2016 - 5th Joint Conference on Lexical and Computational Semantics, Proceedings
PublisherAssociation for Computational Linguistics (ACL)
Pages136-141
Number of pages6
ISBN (Electronic)9781941643921
Publication statusPublished - 1 Jan 2016
Event5th Joint Conference on Lexical and Computational Semantics, *SEM 2016 - Berlin, Germany
Duration: 11 Aug 201612 Aug 2016

Other

Other5th Joint Conference on Lexical and Computational Semantics, *SEM 2016
CountryGermany
CityBerlin
Period11/8/1612/8/16

Fingerprint

Learning algorithms
Experiments

ASJC Scopus subject areas

  • Information Systems
  • Computer Networks and Communications
  • Computer Science Applications

Cite this

Abad, A., & Moschitti, A. (2016). Taking the best from the Crowd: Learning Question Passage Classification from Noisy Data. In *SEM 2016 - 5th Joint Conference on Lexical and Computational Semantics, Proceedings (pp. 136-141). Association for Computational Linguistics (ACL).

Taking the best from the Crowd : Learning Question Passage Classification from Noisy Data. / Abad, Azad; Moschitti, Alessandro.

*SEM 2016 - 5th Joint Conference on Lexical and Computational Semantics, Proceedings. Association for Computational Linguistics (ACL), 2016. p. 136-141.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abad, A & Moschitti, A 2016, Taking the best from the Crowd: Learning Question Passage Classification from Noisy Data. in *SEM 2016 - 5th Joint Conference on Lexical and Computational Semantics, Proceedings. Association for Computational Linguistics (ACL), pp. 136-141, 5th Joint Conference on Lexical and Computational Semantics, *SEM 2016, Berlin, Germany, 11/8/16.
Abad A, Moschitti A. Taking the best from the Crowd: Learning Question Passage Classification from Noisy Data. In *SEM 2016 - 5th Joint Conference on Lexical and Computational Semantics, Proceedings. Association for Computational Linguistics (ACL). 2016. p. 136-141
Abad, Azad ; Moschitti, Alessandro. / Taking the best from the Crowd : Learning Question Passage Classification from Noisy Data. *SEM 2016 - 5th Joint Conference on Lexical and Computational Semantics, Proceedings. Association for Computational Linguistics (ACL), 2016. pp. 136-141
@inproceedings{3b4ac0e8bc464abeb0c2620636bf3d7b,
title = "Taking the best from the Crowd: Learning Question Passage Classification from Noisy Data",
abstract = "In this paper, we propose methods to take into account the disagreement between crowd annotators as well as their skills for weighting instances in learning algorithms. The latter can thus better deal with noise in the annotation and produce higher accuracy. We created two passage reranking datasets: one with crowdsource platform, and the second with an expert who completely revised the crowd annotation. Our experiments show that our weighting approach reduces noise improving passage reranking up to 1.47{\%} and 1.85{\%} on MRR and P@1, respectively.",
author = "Azad Abad and Alessandro Moschitti",
year = "2016",
month = "1",
day = "1",
language = "English",
pages = "136--141",
booktitle = "*SEM 2016 - 5th Joint Conference on Lexical and Computational Semantics, Proceedings",
publisher = "Association for Computational Linguistics (ACL)",

}

TY - GEN

T1 - Taking the best from the Crowd

T2 - Learning Question Passage Classification from Noisy Data

AU - Abad, Azad

AU - Moschitti, Alessandro

PY - 2016/1/1

Y1 - 2016/1/1

N2 - In this paper, we propose methods to take into account the disagreement between crowd annotators as well as their skills for weighting instances in learning algorithms. The latter can thus better deal with noise in the annotation and produce higher accuracy. We created two passage reranking datasets: one with crowdsource platform, and the second with an expert who completely revised the crowd annotation. Our experiments show that our weighting approach reduces noise improving passage reranking up to 1.47% and 1.85% on MRR and P@1, respectively.

AB - In this paper, we propose methods to take into account the disagreement between crowd annotators as well as their skills for weighting instances in learning algorithms. The latter can thus better deal with noise in the annotation and produce higher accuracy. We created two passage reranking datasets: one with crowdsource platform, and the second with an expert who completely revised the crowd annotation. Our experiments show that our weighting approach reduces noise improving passage reranking up to 1.47% and 1.85% on MRR and P@1, respectively.

UR - http://www.scopus.com/inward/record.url?scp=85036479078&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85036479078&partnerID=8YFLogxK

M3 - Conference contribution

SP - 136

EP - 141

BT - *SEM 2016 - 5th Joint Conference on Lexical and Computational Semantics, Proceedings

PB - Association for Computational Linguistics (ACL)

ER -