Overview of the CLEF-2018 checkthat! lab on automatic identification and verification of political claims

Preslav Nakov, Alberto Barron, Tamer Elsayed, Reem Suwaileh, Lluis Marques, Wajdi Zaghouani, Pepa Atanasova, Spas Kyuchukov, Giovanni Martino

Research output: Chapter in Book/Report/Conference proceedingConference contribution

18 Citations (Scopus)

Abstract

We present an overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. In its starting year, the lab featured two tasks. Task 1 asked to predict which (potential) claims in a political debate should be prioritized for fact-checking; in particular, given a debate or a political speech, the goal was to produce a ranked list of its sentences based on their worthiness for fact-checking. Task 2 asked to assess whether a given check-worthy claim made by a politician in the context of a debate/speech is factually true, half-true, or false. We offered both tasks in English and in Arabic. In terms of data, for both tasks, we focused on debates from the 2016 US Presidential Campaign, as well as on some speeches during and after the campaign (we also provided translations in Arabic), and we relied on comments and factuality judgments from factcheck.org and snopes.com, which we further refined manually. A total of 30 teams registered to participate in the lab, and 9 of them actually submitted runs. The evaluation results show that the most successful approaches used various neural networks (esp. for Task 1) and evidence retrieval from the Web (esp. for Task 2). We release all datasets, the evaluation scripts, and the submissions by the participants, which should enable further research in both check-worthiness estimation and automatic claim verification.

Original languageEnglish
Title of host publicationExperimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings
EditorsEric SanJuan, Fionn Murtagh, Jian Yun Nie, Laure Soulier, Linda Cappellato, Patrice Bellot, Josiane Mothe, Chiraz Trabelsi, Nicola Ferro
PublisherSpringer Verlag
Pages372-387
Number of pages16
ISBN (Print)9783319989310
DOIs
Publication statusPublished - 1 Jan 2018
Event9th International Conference of the CLEF Association, CLEF 2018 - Avignon, France
Duration: 10 Sep 201814 Sep 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11018 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other9th International Conference of the CLEF Association, CLEF 2018
CountryFrance
CityAvignon
Period10/9/1814/9/18

    Fingerprint

Keywords

  • Check-worthiness estimation
  • Computational journalism
  • Fact-checking
  • Veracity

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Nakov, P., Barron, A., Elsayed, T., Suwaileh, R., Marques, L., Zaghouani, W., Atanasova, P., Kyuchukov, S., & Martino, G. (2018). Overview of the CLEF-2018 checkthat! lab on automatic identification and verification of political claims. In E. SanJuan, F. Murtagh, J. Y. Nie, L. Soulier, L. Cappellato, P. Bellot, J. Mothe, C. Trabelsi, & N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings (pp. 372-387). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11018 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-98932-7_32