Overview of the CLEF-2018 checkthat! lab on automatic identification and verification of political claims

Preslav Nakov, Alberto Barron, Tamer Elsayed, Reem Suwaileh, Lluis Marques, Wajdi Zaghouani, Pepa Atanasova, Spas Kyuchukov, Giovanni Martino

Research output: Chapter in Book/Report/Conference proceedingConference contribution

15 Citations (Scopus)

Abstract

We present an overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. In its starting year, the lab featured two tasks. Task 1 asked to predict which (potential) claims in a political debate should be prioritized for fact-checking; in particular, given a debate or a political speech, the goal was to produce a ranked list of its sentences based on their worthiness for fact-checking. Task 2 asked to assess whether a given check-worthy claim made by a politician in the context of a debate/speech is factually true, half-true, or false. We offered both tasks in English and in Arabic. In terms of data, for both tasks, we focused on debates from the 2016 US Presidential Campaign, as well as on some speeches during and after the campaign (we also provided translations in Arabic), and we relied on comments and factuality judgments from factcheck.org and snopes.com, which we further refined manually. A total of 30 teams registered to participate in the lab, and 9 of them actually submitted runs. The evaluation results show that the most successful approaches used various neural networks (esp. for Task 1) and evidence retrieval from the Web (esp. for Task 2). We release all datasets, the evaluation scripts, and the submissions by the participants, which should enable further research in both check-worthiness estimation and automatic claim verification.

Original languageEnglish
Title of host publicationExperimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings
EditorsEric SanJuan, Fionn Murtagh, Jian Yun Nie, Laure Soulier, Linda Cappellato, Patrice Bellot, Josiane Mothe, Chiraz Trabelsi, Nicola Ferro
PublisherSpringer Verlag
Pages372-387
Number of pages16
ISBN (Print)9783319989310
DOIs
Publication statusPublished - 1 Jan 2018
Event9th International Conference of the CLEF Association, CLEF 2018 - Avignon, France
Duration: 10 Sep 201814 Sep 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11018 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other9th International Conference of the CLEF Association, CLEF 2018
CountryFrance
CityAvignon
Period10/9/1814/9/18

Fingerprint

Evaluation
Neural networks
Retrieval
Neural Networks
Predict
Speech
Evidence
Judgment
Context
False

Keywords

  • Check-worthiness estimation
  • Computational journalism
  • Fact-checking
  • Veracity

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Nakov, P., Barron, A., Elsayed, T., Suwaileh, R., Marques, L., Zaghouani, W., ... Martino, G. (2018). Overview of the CLEF-2018 checkthat! lab on automatic identification and verification of political claims. In E. SanJuan, F. Murtagh, J. Y. Nie, L. Soulier, L. Cappellato, P. Bellot, J. Mothe, C. Trabelsi, ... N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings (pp. 372-387). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11018 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-98932-7_32

Overview of the CLEF-2018 checkthat! lab on automatic identification and verification of political claims. / Nakov, Preslav; Barron, Alberto; Elsayed, Tamer; Suwaileh, Reem; Marques, Lluis; Zaghouani, Wajdi; Atanasova, Pepa; Kyuchukov, Spas; Martino, Giovanni.

Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings. ed. / Eric SanJuan; Fionn Murtagh; Jian Yun Nie; Laure Soulier; Linda Cappellato; Patrice Bellot; Josiane Mothe; Chiraz Trabelsi; Nicola Ferro. Springer Verlag, 2018. p. 372-387 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11018 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Nakov, P, Barron, A, Elsayed, T, Suwaileh, R, Marques, L, Zaghouani, W, Atanasova, P, Kyuchukov, S & Martino, G 2018, Overview of the CLEF-2018 checkthat! lab on automatic identification and verification of political claims. in E SanJuan, F Murtagh, JY Nie, L Soulier, L Cappellato, P Bellot, J Mothe, C Trabelsi & N Ferro (eds), Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11018 LNCS, Springer Verlag, pp. 372-387, 9th International Conference of the CLEF Association, CLEF 2018, Avignon, France, 10/9/18. https://doi.org/10.1007/978-3-319-98932-7_32
Nakov P, Barron A, Elsayed T, Suwaileh R, Marques L, Zaghouani W et al. Overview of the CLEF-2018 checkthat! lab on automatic identification and verification of political claims. In SanJuan E, Murtagh F, Nie JY, Soulier L, Cappellato L, Bellot P, Mothe J, Trabelsi C, Ferro N, editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings. Springer Verlag. 2018. p. 372-387. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-98932-7_32
Nakov, Preslav ; Barron, Alberto ; Elsayed, Tamer ; Suwaileh, Reem ; Marques, Lluis ; Zaghouani, Wajdi ; Atanasova, Pepa ; Kyuchukov, Spas ; Martino, Giovanni. / Overview of the CLEF-2018 checkthat! lab on automatic identification and verification of political claims. Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings. editor / Eric SanJuan ; Fionn Murtagh ; Jian Yun Nie ; Laure Soulier ; Linda Cappellato ; Patrice Bellot ; Josiane Mothe ; Chiraz Trabelsi ; Nicola Ferro. Springer Verlag, 2018. pp. 372-387 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{30156d6659044e73b1d0f1b64b54d3d3,
title = "Overview of the CLEF-2018 checkthat! lab on automatic identification and verification of political claims",
abstract = "We present an overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. In its starting year, the lab featured two tasks. Task 1 asked to predict which (potential) claims in a political debate should be prioritized for fact-checking; in particular, given a debate or a political speech, the goal was to produce a ranked list of its sentences based on their worthiness for fact-checking. Task 2 asked to assess whether a given check-worthy claim made by a politician in the context of a debate/speech is factually true, half-true, or false. We offered both tasks in English and in Arabic. In terms of data, for both tasks, we focused on debates from the 2016 US Presidential Campaign, as well as on some speeches during and after the campaign (we also provided translations in Arabic), and we relied on comments and factuality judgments from factcheck.org and snopes.com, which we further refined manually. A total of 30 teams registered to participate in the lab, and 9 of them actually submitted runs. The evaluation results show that the most successful approaches used various neural networks (esp. for Task 1) and evidence retrieval from the Web (esp. for Task 2). We release all datasets, the evaluation scripts, and the submissions by the participants, which should enable further research in both check-worthiness estimation and automatic claim verification.",
keywords = "Check-worthiness estimation, Computational journalism, Fact-checking, Veracity",
author = "Preslav Nakov and Alberto Barron and Tamer Elsayed and Reem Suwaileh and Lluis Marques and Wajdi Zaghouani and Pepa Atanasova and Spas Kyuchukov and Giovanni Martino",
year = "2018",
month = "1",
day = "1",
doi = "10.1007/978-3-319-98932-7_32",
language = "English",
isbn = "9783319989310",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "372--387",
editor = "Eric SanJuan and Fionn Murtagh and Nie, {Jian Yun} and Laure Soulier and Linda Cappellato and Patrice Bellot and Josiane Mothe and Chiraz Trabelsi and Nicola Ferro",
booktitle = "Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings",

}

TY - GEN

T1 - Overview of the CLEF-2018 checkthat! lab on automatic identification and verification of political claims

AU - Nakov, Preslav

AU - Barron, Alberto

AU - Elsayed, Tamer

AU - Suwaileh, Reem

AU - Marques, Lluis

AU - Zaghouani, Wajdi

AU - Atanasova, Pepa

AU - Kyuchukov, Spas

AU - Martino, Giovanni

PY - 2018/1/1

Y1 - 2018/1/1

N2 - We present an overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. In its starting year, the lab featured two tasks. Task 1 asked to predict which (potential) claims in a political debate should be prioritized for fact-checking; in particular, given a debate or a political speech, the goal was to produce a ranked list of its sentences based on their worthiness for fact-checking. Task 2 asked to assess whether a given check-worthy claim made by a politician in the context of a debate/speech is factually true, half-true, or false. We offered both tasks in English and in Arabic. In terms of data, for both tasks, we focused on debates from the 2016 US Presidential Campaign, as well as on some speeches during and after the campaign (we also provided translations in Arabic), and we relied on comments and factuality judgments from factcheck.org and snopes.com, which we further refined manually. A total of 30 teams registered to participate in the lab, and 9 of them actually submitted runs. The evaluation results show that the most successful approaches used various neural networks (esp. for Task 1) and evidence retrieval from the Web (esp. for Task 2). We release all datasets, the evaluation scripts, and the submissions by the participants, which should enable further research in both check-worthiness estimation and automatic claim verification.

AB - We present an overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. In its starting year, the lab featured two tasks. Task 1 asked to predict which (potential) claims in a political debate should be prioritized for fact-checking; in particular, given a debate or a political speech, the goal was to produce a ranked list of its sentences based on their worthiness for fact-checking. Task 2 asked to assess whether a given check-worthy claim made by a politician in the context of a debate/speech is factually true, half-true, or false. We offered both tasks in English and in Arabic. In terms of data, for both tasks, we focused on debates from the 2016 US Presidential Campaign, as well as on some speeches during and after the campaign (we also provided translations in Arabic), and we relied on comments and factuality judgments from factcheck.org and snopes.com, which we further refined manually. A total of 30 teams registered to participate in the lab, and 9 of them actually submitted runs. The evaluation results show that the most successful approaches used various neural networks (esp. for Task 1) and evidence retrieval from the Web (esp. for Task 2). We release all datasets, the evaluation scripts, and the submissions by the participants, which should enable further research in both check-worthiness estimation and automatic claim verification.

KW - Check-worthiness estimation

KW - Computational journalism

KW - Fact-checking

KW - Veracity

UR - http://www.scopus.com/inward/record.url?scp=85051084265&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85051084265&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-98932-7_32

DO - 10.1007/978-3-319-98932-7_32

M3 - Conference contribution

AN - SCOPUS:85051084265

SN - 9783319989310

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 372

EP - 387

BT - Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Proceedings

A2 - SanJuan, Eric

A2 - Murtagh, Fionn

A2 - Nie, Jian Yun

A2 - Soulier, Laure

A2 - Cappellato, Linda

A2 - Bellot, Patrice

A2 - Mothe, Josiane

A2 - Trabelsi, Chiraz

A2 - Ferro, Nicola

PB - Springer Verlag

ER -