Coreference resolution: An empirical study based on SemEval-2010 shared Task 1

Lluis Marques, Marta Recasens, Emili Sapena

Research output: Contribution to journalArticle

7 Citations (Scopus)

Abstract

This paper presents an empirical evaluation of coreference resolution that covers several interrelated dimensions. The main goal is to complete the comparative analysis from the SemEval-2010 task on Coreference Resolution in Multiple Languages. To do so, the study restricts the number of languages and systems involved, but extends and deepens the analysis of the system outputs, including a more qualitative discussion. The paper compares three automatic coreference resolution systems for three languages (English, Catalan and Spanish) in four evaluation settings, and using four evaluation measures. Given that our main goal is not to provide a comparison between resolution algorithms, these are merely used as tools to shed light on the different conditions under which coreference resolution is evaluated. Although the dimensions are strongly interdependent, making it very difficult to extract general principles, the study reveals a series of interesting issues in relation to coreference resolution: the portability of systems across languages, the influence of the type and quality of input annotations, and the behavior of the scoring measures.

Original languageEnglish
Pages (from-to)661-694
Number of pages34
JournalLanguage Resources and Evaluation
Volume47
Issue number3
DOIs
Publication statusPublished - 1 Sep 2013
Externally publishedYes

Fingerprint

language
evaluation
Spanish language
English language
Empirical Study
Coreference
Evaluation
Language
Comparative Analysis
Scoring
Annotation

Keywords

  • Coreference resolution and evaluation
  • Discourse entities
  • Machine learning based NLP tools
  • NLP system analysis
  • SemEval-2010 (Task 1)

ASJC Scopus subject areas

  • Linguistics and Language
  • Library and Information Sciences
  • Language and Linguistics
  • Education

Cite this

Coreference resolution : An empirical study based on SemEval-2010 shared Task 1. / Marques, Lluis; Recasens, Marta; Sapena, Emili.

In: Language Resources and Evaluation, Vol. 47, No. 3, 01.09.2013, p. 661-694.

Research output: Contribution to journalArticle

@article{fc1654b9a7f74ba8b222977372881c62,
title = "Coreference resolution: An empirical study based on SemEval-2010 shared Task 1",
abstract = "This paper presents an empirical evaluation of coreference resolution that covers several interrelated dimensions. The main goal is to complete the comparative analysis from the SemEval-2010 task on Coreference Resolution in Multiple Languages. To do so, the study restricts the number of languages and systems involved, but extends and deepens the analysis of the system outputs, including a more qualitative discussion. The paper compares three automatic coreference resolution systems for three languages (English, Catalan and Spanish) in four evaluation settings, and using four evaluation measures. Given that our main goal is not to provide a comparison between resolution algorithms, these are merely used as tools to shed light on the different conditions under which coreference resolution is evaluated. Although the dimensions are strongly interdependent, making it very difficult to extract general principles, the study reveals a series of interesting issues in relation to coreference resolution: the portability of systems across languages, the influence of the type and quality of input annotations, and the behavior of the scoring measures.",
keywords = "Coreference resolution and evaluation, Discourse entities, Machine learning based NLP tools, NLP system analysis, SemEval-2010 (Task 1)",
author = "Lluis Marques and Marta Recasens and Emili Sapena",
year = "2013",
month = "9",
day = "1",
doi = "10.1007/s10579-012-9194-z",
language = "English",
volume = "47",
pages = "661--694",
journal = "Language Resources and Evaluation",
issn = "1574-020X",
publisher = "Springer Netherlands",
number = "3",

}

TY - JOUR

T1 - Coreference resolution

T2 - An empirical study based on SemEval-2010 shared Task 1

AU - Marques, Lluis

AU - Recasens, Marta

AU - Sapena, Emili

PY - 2013/9/1

Y1 - 2013/9/1

N2 - This paper presents an empirical evaluation of coreference resolution that covers several interrelated dimensions. The main goal is to complete the comparative analysis from the SemEval-2010 task on Coreference Resolution in Multiple Languages. To do so, the study restricts the number of languages and systems involved, but extends and deepens the analysis of the system outputs, including a more qualitative discussion. The paper compares three automatic coreference resolution systems for three languages (English, Catalan and Spanish) in four evaluation settings, and using four evaluation measures. Given that our main goal is not to provide a comparison between resolution algorithms, these are merely used as tools to shed light on the different conditions under which coreference resolution is evaluated. Although the dimensions are strongly interdependent, making it very difficult to extract general principles, the study reveals a series of interesting issues in relation to coreference resolution: the portability of systems across languages, the influence of the type and quality of input annotations, and the behavior of the scoring measures.

AB - This paper presents an empirical evaluation of coreference resolution that covers several interrelated dimensions. The main goal is to complete the comparative analysis from the SemEval-2010 task on Coreference Resolution in Multiple Languages. To do so, the study restricts the number of languages and systems involved, but extends and deepens the analysis of the system outputs, including a more qualitative discussion. The paper compares three automatic coreference resolution systems for three languages (English, Catalan and Spanish) in four evaluation settings, and using four evaluation measures. Given that our main goal is not to provide a comparison between resolution algorithms, these are merely used as tools to shed light on the different conditions under which coreference resolution is evaluated. Although the dimensions are strongly interdependent, making it very difficult to extract general principles, the study reveals a series of interesting issues in relation to coreference resolution: the portability of systems across languages, the influence of the type and quality of input annotations, and the behavior of the scoring measures.

KW - Coreference resolution and evaluation

KW - Discourse entities

KW - Machine learning based NLP tools

KW - NLP system analysis

KW - SemEval-2010 (Task 1)

UR - http://www.scopus.com/inward/record.url?scp=84883447787&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84883447787&partnerID=8YFLogxK

U2 - 10.1007/s10579-012-9194-z

DO - 10.1007/s10579-012-9194-z

M3 - Article

AN - SCOPUS:84883447787

VL - 47

SP - 661

EP - 694

JO - Language Resources and Evaluation

JF - Language Resources and Evaluation

SN - 1574-020X

IS - 3

ER -