Developing a successful SemEval task in sentiment analysis of Twitter and other social media texts

Preslav Nakov, Sara Rosenthal, Svetlana Kiritchenko, Saif M. Mohammad, Zornitsa Kozareva, Alan Ritter, Veselin Stoyanov, Xiaodan Zhu

Research output: Contribution to journalArticle

38 Citations (Scopus)

Abstract

We present the development and evaluation of a semantic analysis task that lies at the intersection of two very trendy lines of research in contemporary computational linguistics: (1) sentiment analysis, and (2) natural language processing of social media text. The task was part of SemEval, the International Workshop on Semantic Evaluation, a semantic evaluation forum previously known as SensEval. The task ran in 2013 and 2014, attracting the highest number of participating teams at SemEval in both years, and there is an ongoing edition in 2015. The task included the creation of a large contextual and message-level polarity corpus consisting of tweets, SMS messages, LiveJournal messages, and a special test set of sarcastic tweets. The evaluation attracted 44 teams in 2013 and 46 in 2014, who used a variety of approaches. The best teams were able to outperform several baselines by sizable margins with improvement across the 2 years the task has been run. We hope that the long-lasting role of this task and the accompanying datasets will be to serve as a test bed for comparing different approaches, thus facilitating research.

Original languageEnglish
Pages (from-to)35-65
Number of pages31
JournalLanguage Resources and Evaluation
Volume50
Issue number1
DOIs
Publication statusPublished - 1 Mar 2016

Fingerprint

twitter
social media
semantics
evaluation
computational linguistics
SMS
edition
Evaluation
Social Media
Sentiment
language

Keywords

  • SemEval
  • Sentiment analysis
  • Twitter

ASJC Scopus subject areas

  • Linguistics and Language
  • Library and Information Sciences
  • Language and Linguistics
  • Education

Cite this

Developing a successful SemEval task in sentiment analysis of Twitter and other social media texts. / Nakov, Preslav; Rosenthal, Sara; Kiritchenko, Svetlana; Mohammad, Saif M.; Kozareva, Zornitsa; Ritter, Alan; Stoyanov, Veselin; Zhu, Xiaodan.

In: Language Resources and Evaluation, Vol. 50, No. 1, 01.03.2016, p. 35-65.

Research output: Contribution to journalArticle

Nakov, P, Rosenthal, S, Kiritchenko, S, Mohammad, SM, Kozareva, Z, Ritter, A, Stoyanov, V & Zhu, X 2016, 'Developing a successful SemEval task in sentiment analysis of Twitter and other social media texts', Language Resources and Evaluation, vol. 50, no. 1, pp. 35-65. https://doi.org/10.1007/s10579-015-9328-1
Nakov, Preslav ; Rosenthal, Sara ; Kiritchenko, Svetlana ; Mohammad, Saif M. ; Kozareva, Zornitsa ; Ritter, Alan ; Stoyanov, Veselin ; Zhu, Xiaodan. / Developing a successful SemEval task in sentiment analysis of Twitter and other social media texts. In: Language Resources and Evaluation. 2016 ; Vol. 50, No. 1. pp. 35-65.
@article{8458ba0b96d24bd2841dbcc754443cf2,
title = "Developing a successful SemEval task in sentiment analysis of Twitter and other social media texts",
abstract = "We present the development and evaluation of a semantic analysis task that lies at the intersection of two very trendy lines of research in contemporary computational linguistics: (1) sentiment analysis, and (2) natural language processing of social media text. The task was part of SemEval, the International Workshop on Semantic Evaluation, a semantic evaluation forum previously known as SensEval. The task ran in 2013 and 2014, attracting the highest number of participating teams at SemEval in both years, and there is an ongoing edition in 2015. The task included the creation of a large contextual and message-level polarity corpus consisting of tweets, SMS messages, LiveJournal messages, and a special test set of sarcastic tweets. The evaluation attracted 44 teams in 2013 and 46 in 2014, who used a variety of approaches. The best teams were able to outperform several baselines by sizable margins with improvement across the 2 years the task has been run. We hope that the long-lasting role of this task and the accompanying datasets will be to serve as a test bed for comparing different approaches, thus facilitating research.",
keywords = "SemEval, Sentiment analysis, Twitter",
author = "Preslav Nakov and Sara Rosenthal and Svetlana Kiritchenko and Mohammad, {Saif M.} and Zornitsa Kozareva and Alan Ritter and Veselin Stoyanov and Xiaodan Zhu",
year = "2016",
month = "3",
day = "1",
doi = "10.1007/s10579-015-9328-1",
language = "English",
volume = "50",
pages = "35--65",
journal = "Language Resources and Evaluation",
issn = "1574-020X",
publisher = "Springer Netherlands",
number = "1",

}

TY - JOUR

T1 - Developing a successful SemEval task in sentiment analysis of Twitter and other social media texts

AU - Nakov, Preslav

AU - Rosenthal, Sara

AU - Kiritchenko, Svetlana

AU - Mohammad, Saif M.

AU - Kozareva, Zornitsa

AU - Ritter, Alan

AU - Stoyanov, Veselin

AU - Zhu, Xiaodan

PY - 2016/3/1

Y1 - 2016/3/1

N2 - We present the development and evaluation of a semantic analysis task that lies at the intersection of two very trendy lines of research in contemporary computational linguistics: (1) sentiment analysis, and (2) natural language processing of social media text. The task was part of SemEval, the International Workshop on Semantic Evaluation, a semantic evaluation forum previously known as SensEval. The task ran in 2013 and 2014, attracting the highest number of participating teams at SemEval in both years, and there is an ongoing edition in 2015. The task included the creation of a large contextual and message-level polarity corpus consisting of tweets, SMS messages, LiveJournal messages, and a special test set of sarcastic tweets. The evaluation attracted 44 teams in 2013 and 46 in 2014, who used a variety of approaches. The best teams were able to outperform several baselines by sizable margins with improvement across the 2 years the task has been run. We hope that the long-lasting role of this task and the accompanying datasets will be to serve as a test bed for comparing different approaches, thus facilitating research.

AB - We present the development and evaluation of a semantic analysis task that lies at the intersection of two very trendy lines of research in contemporary computational linguistics: (1) sentiment analysis, and (2) natural language processing of social media text. The task was part of SemEval, the International Workshop on Semantic Evaluation, a semantic evaluation forum previously known as SensEval. The task ran in 2013 and 2014, attracting the highest number of participating teams at SemEval in both years, and there is an ongoing edition in 2015. The task included the creation of a large contextual and message-level polarity corpus consisting of tweets, SMS messages, LiveJournal messages, and a special test set of sarcastic tweets. The evaluation attracted 44 teams in 2013 and 46 in 2014, who used a variety of approaches. The best teams were able to outperform several baselines by sizable margins with improvement across the 2 years the task has been run. We hope that the long-lasting role of this task and the accompanying datasets will be to serve as a test bed for comparing different approaches, thus facilitating research.

KW - SemEval

KW - Sentiment analysis

KW - Twitter

UR - http://www.scopus.com/inward/record.url?scp=84958753956&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84958753956&partnerID=8YFLogxK

U2 - 10.1007/s10579-015-9328-1

DO - 10.1007/s10579-015-9328-1

M3 - Article

AN - SCOPUS:84958753956

VL - 50

SP - 35

EP - 65

JO - Language Resources and Evaluation

JF - Language Resources and Evaluation

SN - 1574-020X

IS - 1

ER -