Overview of the CLEF-2019 CheckThat! Lab: Automatic Identification and Verification of Claims

Tamer Elsayed, Preslav Nakov, Alberto Barron, Maram Hasanain, Reem Suwaileh, Giovanni Martino, Pepa Atanasova

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present an overview of the second edition of the CheckThat! Lab at CLEF 2019. The lab featured two tasks in two different languages: English and Arabic. Task 1 (English) challenged the participating systems to predict which claims in a political debate or speech should be prioritized for fact-checking. Task 2 (Arabic) asked to (A) rank a given set of Web pages with respect to a check-worthy claim based on their usefulness for fact-checking that claim, (B) classify these same Web pages according to their degree of usefulness for fact-checking the target claim, (C) identify useful passages from these pages, and (D) use the useful pages to predict the claim’s factuality. CheckThat! provided a full evaluation framework, consisting of data in English (derived from fact-checking sources) and Arabic (gathered and annotated from scratch) and evaluation based on mean average precision (MAP) and normalized discounted cumulative gain (nDCG) for ranking, and F $$:1$$ for classification. A total of 47 teams registered to participate in this lab, and fourteen of them actually submitted runs (compared to nine last year). The evaluation results show that the most successful approaches to Task 1 used various neural networks and logistic regression. As for Task 2, learning-to-rank was used by the highest scoring runs for subtask A, while different classifiers were used in the other subtasks. We release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research in the important tasks of check-worthiness estimation and automatic claim verification.

Original languageEnglish
Title of host publicationExperimental IR Meets Multilinguality, Multimodality, and Interaction - 10th International Conference of the CLEF Association, CLEF 2019, Proceedings
EditorsFabio Crestani, Martin Braschler, Jacques Savoy, Andreas Rauber, Henning Müller, David E. Losada, Gundula Heinatz Bürki, Linda Cappellato, Nicola Ferro
PublisherSpringer Verlag
Pages301-321
Number of pages21
ISBN (Print)9783030285760
DOIs
Publication statusPublished - 1 Jan 2019
Event10th International Conference of the CLEF Association, CLEF 2019 - Lugano, Switzerland
Duration: 9 Sep 201912 Sep 2019

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11696 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference10th International Conference of the CLEF Association, CLEF 2019
CountrySwitzerland
CityLugano
Period9/9/1912/9/19

Fingerprint

Websites
Logistics
Classifiers
Evaluation
Neural networks
Predict
Logistic Regression
Scoring
Ranking
Classify
Classifier
Neural Networks
Target

Keywords

  • Check-worthiness estimation
  • Computational journalism
  • Evidence-based verification
  • Fact-checking
  • Fake news detection
  • Veracity

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Elsayed, T., Nakov, P., Barron, A., Hasanain, M., Suwaileh, R., Martino, G., & Atanasova, P. (2019). Overview of the CLEF-2019 CheckThat! Lab: Automatic Identification and Verification of Claims. In F. Crestani, M. Braschler, J. Savoy, A. Rauber, H. Müller, D. E. Losada, G. Heinatz Bürki, L. Cappellato, ... N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction - 10th International Conference of the CLEF Association, CLEF 2019, Proceedings (pp. 301-321). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11696 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-28577-7_25

Overview of the CLEF-2019 CheckThat! Lab : Automatic Identification and Verification of Claims. / Elsayed, Tamer; Nakov, Preslav; Barron, Alberto; Hasanain, Maram; Suwaileh, Reem; Martino, Giovanni; Atanasova, Pepa.

Experimental IR Meets Multilinguality, Multimodality, and Interaction - 10th International Conference of the CLEF Association, CLEF 2019, Proceedings. ed. / Fabio Crestani; Martin Braschler; Jacques Savoy; Andreas Rauber; Henning Müller; David E. Losada; Gundula Heinatz Bürki; Linda Cappellato; Nicola Ferro. Springer Verlag, 2019. p. 301-321 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11696 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Elsayed, T, Nakov, P, Barron, A, Hasanain, M, Suwaileh, R, Martino, G & Atanasova, P 2019, Overview of the CLEF-2019 CheckThat! Lab: Automatic Identification and Verification of Claims. in F Crestani, M Braschler, J Savoy, A Rauber, H Müller, DE Losada, G Heinatz Bürki, L Cappellato & N Ferro (eds), Experimental IR Meets Multilinguality, Multimodality, and Interaction - 10th International Conference of the CLEF Association, CLEF 2019, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11696 LNCS, Springer Verlag, pp. 301-321, 10th International Conference of the CLEF Association, CLEF 2019, Lugano, Switzerland, 9/9/19. https://doi.org/10.1007/978-3-030-28577-7_25
Elsayed T, Nakov P, Barron A, Hasanain M, Suwaileh R, Martino G et al. Overview of the CLEF-2019 CheckThat! Lab: Automatic Identification and Verification of Claims. In Crestani F, Braschler M, Savoy J, Rauber A, Müller H, Losada DE, Heinatz Bürki G, Cappellato L, Ferro N, editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction - 10th International Conference of the CLEF Association, CLEF 2019, Proceedings. Springer Verlag. 2019. p. 301-321. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-28577-7_25
Elsayed, Tamer ; Nakov, Preslav ; Barron, Alberto ; Hasanain, Maram ; Suwaileh, Reem ; Martino, Giovanni ; Atanasova, Pepa. / Overview of the CLEF-2019 CheckThat! Lab : Automatic Identification and Verification of Claims. Experimental IR Meets Multilinguality, Multimodality, and Interaction - 10th International Conference of the CLEF Association, CLEF 2019, Proceedings. editor / Fabio Crestani ; Martin Braschler ; Jacques Savoy ; Andreas Rauber ; Henning Müller ; David E. Losada ; Gundula Heinatz Bürki ; Linda Cappellato ; Nicola Ferro. Springer Verlag, 2019. pp. 301-321 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{79731682bdfb4de987dfaf08af082d55,
title = "Overview of the CLEF-2019 CheckThat! Lab: Automatic Identification and Verification of Claims",
abstract = "We present an overview of the second edition of the CheckThat! Lab at CLEF 2019. The lab featured two tasks in two different languages: English and Arabic. Task 1 (English) challenged the participating systems to predict which claims in a political debate or speech should be prioritized for fact-checking. Task 2 (Arabic) asked to (A) rank a given set of Web pages with respect to a check-worthy claim based on their usefulness for fact-checking that claim, (B) classify these same Web pages according to their degree of usefulness for fact-checking the target claim, (C) identify useful passages from these pages, and (D) use the useful pages to predict the claim’s factuality. CheckThat! provided a full evaluation framework, consisting of data in English (derived from fact-checking sources) and Arabic (gathered and annotated from scratch) and evaluation based on mean average precision (MAP) and normalized discounted cumulative gain (nDCG) for ranking, and F $$:1$$ for classification. A total of 47 teams registered to participate in this lab, and fourteen of them actually submitted runs (compared to nine last year). The evaluation results show that the most successful approaches to Task 1 used various neural networks and logistic regression. As for Task 2, learning-to-rank was used by the highest scoring runs for subtask A, while different classifiers were used in the other subtasks. We release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research in the important tasks of check-worthiness estimation and automatic claim verification.",
keywords = "Check-worthiness estimation, Computational journalism, Evidence-based verification, Fact-checking, Fake news detection, Veracity",
author = "Tamer Elsayed and Preslav Nakov and Alberto Barron and Maram Hasanain and Reem Suwaileh and Giovanni Martino and Pepa Atanasova",
year = "2019",
month = "1",
day = "1",
doi = "10.1007/978-3-030-28577-7_25",
language = "English",
isbn = "9783030285760",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "301--321",
editor = "Fabio Crestani and Martin Braschler and Jacques Savoy and Andreas Rauber and Henning M{\"u}ller and Losada, {David E.} and {Heinatz B{\"u}rki}, Gundula and Linda Cappellato and Nicola Ferro",
booktitle = "Experimental IR Meets Multilinguality, Multimodality, and Interaction - 10th International Conference of the CLEF Association, CLEF 2019, Proceedings",

}

TY - GEN

T1 - Overview of the CLEF-2019 CheckThat! Lab

T2 - Automatic Identification and Verification of Claims

AU - Elsayed, Tamer

AU - Nakov, Preslav

AU - Barron, Alberto

AU - Hasanain, Maram

AU - Suwaileh, Reem

AU - Martino, Giovanni

AU - Atanasova, Pepa

PY - 2019/1/1

Y1 - 2019/1/1

N2 - We present an overview of the second edition of the CheckThat! Lab at CLEF 2019. The lab featured two tasks in two different languages: English and Arabic. Task 1 (English) challenged the participating systems to predict which claims in a political debate or speech should be prioritized for fact-checking. Task 2 (Arabic) asked to (A) rank a given set of Web pages with respect to a check-worthy claim based on their usefulness for fact-checking that claim, (B) classify these same Web pages according to their degree of usefulness for fact-checking the target claim, (C) identify useful passages from these pages, and (D) use the useful pages to predict the claim’s factuality. CheckThat! provided a full evaluation framework, consisting of data in English (derived from fact-checking sources) and Arabic (gathered and annotated from scratch) and evaluation based on mean average precision (MAP) and normalized discounted cumulative gain (nDCG) for ranking, and F $$:1$$ for classification. A total of 47 teams registered to participate in this lab, and fourteen of them actually submitted runs (compared to nine last year). The evaluation results show that the most successful approaches to Task 1 used various neural networks and logistic regression. As for Task 2, learning-to-rank was used by the highest scoring runs for subtask A, while different classifiers were used in the other subtasks. We release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research in the important tasks of check-worthiness estimation and automatic claim verification.

AB - We present an overview of the second edition of the CheckThat! Lab at CLEF 2019. The lab featured two tasks in two different languages: English and Arabic. Task 1 (English) challenged the participating systems to predict which claims in a political debate or speech should be prioritized for fact-checking. Task 2 (Arabic) asked to (A) rank a given set of Web pages with respect to a check-worthy claim based on their usefulness for fact-checking that claim, (B) classify these same Web pages according to their degree of usefulness for fact-checking the target claim, (C) identify useful passages from these pages, and (D) use the useful pages to predict the claim’s factuality. CheckThat! provided a full evaluation framework, consisting of data in English (derived from fact-checking sources) and Arabic (gathered and annotated from scratch) and evaluation based on mean average precision (MAP) and normalized discounted cumulative gain (nDCG) for ranking, and F $$:1$$ for classification. A total of 47 teams registered to participate in this lab, and fourteen of them actually submitted runs (compared to nine last year). The evaluation results show that the most successful approaches to Task 1 used various neural networks and logistic regression. As for Task 2, learning-to-rank was used by the highest scoring runs for subtask A, while different classifiers were used in the other subtasks. We release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research in the important tasks of check-worthiness estimation and automatic claim verification.

KW - Check-worthiness estimation

KW - Computational journalism

KW - Evidence-based verification

KW - Fact-checking

KW - Fake news detection

KW - Veracity

UR - http://www.scopus.com/inward/record.url?scp=85072834528&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85072834528&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-28577-7_25

DO - 10.1007/978-3-030-28577-7_25

M3 - Conference contribution

AN - SCOPUS:85072834528

SN - 9783030285760

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 301

EP - 321

BT - Experimental IR Meets Multilinguality, Multimodality, and Interaction - 10th International Conference of the CLEF Association, CLEF 2019, Proceedings

A2 - Crestani, Fabio

A2 - Braschler, Martin

A2 - Savoy, Jacques

A2 - Rauber, Andreas

A2 - Müller, Henning

A2 - Losada, David E.

A2 - Heinatz Bürki, Gundula

A2 - Cappellato, Linda

A2 - Ferro, Nicola

PB - Springer Verlag

ER -