Multi-dialect Arabic POS tagging

A CRF approach

Kareem Darwish, Hamdy Mubarak, Mohamed Eldesouki, Ahmed Abdelali, Younes Samih, Randah Alharbi, Mohammed Attia, Walid Magdy, Laura Kallmeyer

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

This paper introduces a new dataset of POS-tagged Arabic tweets in four major dialects along with tagging guidelines. The data, which we are releasing publicly, includes tweets in Egyptian, Levantine, Gulf, and Maghrebi, with 350 tweets for each dialect with appropriate train/test/development splits for 5-fold cross validation. We use a Conditional Random Fields (CRF) sequence labeler to train POS taggers for each dialect and examine the effect of cross and joint dialect training, and give benchmark results for the datasets. Using clitic n-grams, clitic metatypes, and stem templates as features, we were able to train a joint model that can correctly tag four different dialects with an average accuracy of 89.3%.

Original languageEnglish
Title of host publicationLREC 2018 - 11th International Conference on Language Resources and Evaluation
EditorsHitoshi Isahara, Bente Maegaard, Stelios Piperidis, Christopher Cieri, Thierry Declerck, Koiti Hasida, Helene Mazo, Khalid Choukri, Sara Goggi, Joseph Mariani, Asuncion Moreno, Nicoletta Calzolari, Jan Odijk, Takenobu Tokunaga
PublisherEuropean Language Resources Association (ELRA)
Pages93-98
Number of pages6
ISBN (Electronic)9791095546009
Publication statusPublished - 1 Jan 2019
Event11th International Conference on Language Resources and Evaluation, LREC 2018 - Miyazaki, Japan
Duration: 7 May 201812 May 2018

Other

Other11th International Conference on Language Resources and Evaluation, LREC 2018
CountryJapan
CityMiyazaki
Period7/5/1812/5/18

Fingerprint

dialect
Tagging
Arabic Dialects
Train
Clitics

Keywords

  • Arabic dialects
  • CRF
  • POS tagging

ASJC Scopus subject areas

  • Linguistics and Language
  • Education
  • Library and Information Sciences
  • Language and Linguistics

Cite this

Darwish, K., Mubarak, H., Eldesouki, M., Abdelali, A., Samih, Y., Alharbi, R., ... Kallmeyer, L. (2019). Multi-dialect Arabic POS tagging: A CRF approach. In H. Isahara, B. Maegaard, S. Piperidis, C. Cieri, T. Declerck, K. Hasida, H. Mazo, K. Choukri, S. Goggi, J. Mariani, A. Moreno, N. Calzolari, J. Odijk, ... T. Tokunaga (Eds.), LREC 2018 - 11th International Conference on Language Resources and Evaluation (pp. 93-98). European Language Resources Association (ELRA).

Multi-dialect Arabic POS tagging : A CRF approach. / Darwish, Kareem; Mubarak, Hamdy; Eldesouki, Mohamed; Abdelali, Ahmed; Samih, Younes; Alharbi, Randah; Attia, Mohammed; Magdy, Walid; Kallmeyer, Laura.

LREC 2018 - 11th International Conference on Language Resources and Evaluation. ed. / Hitoshi Isahara; Bente Maegaard; Stelios Piperidis; Christopher Cieri; Thierry Declerck; Koiti Hasida; Helene Mazo; Khalid Choukri; Sara Goggi; Joseph Mariani; Asuncion Moreno; Nicoletta Calzolari; Jan Odijk; Takenobu Tokunaga. European Language Resources Association (ELRA), 2019. p. 93-98.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Darwish, K, Mubarak, H, Eldesouki, M, Abdelali, A, Samih, Y, Alharbi, R, Attia, M, Magdy, W & Kallmeyer, L 2019, Multi-dialect Arabic POS tagging: A CRF approach. in H Isahara, B Maegaard, S Piperidis, C Cieri, T Declerck, K Hasida, H Mazo, K Choukri, S Goggi, J Mariani, A Moreno, N Calzolari, J Odijk & T Tokunaga (eds), LREC 2018 - 11th International Conference on Language Resources and Evaluation. European Language Resources Association (ELRA), pp. 93-98, 11th International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, 7/5/18.
Darwish K, Mubarak H, Eldesouki M, Abdelali A, Samih Y, Alharbi R et al. Multi-dialect Arabic POS tagging: A CRF approach. In Isahara H, Maegaard B, Piperidis S, Cieri C, Declerck T, Hasida K, Mazo H, Choukri K, Goggi S, Mariani J, Moreno A, Calzolari N, Odijk J, Tokunaga T, editors, LREC 2018 - 11th International Conference on Language Resources and Evaluation. European Language Resources Association (ELRA). 2019. p. 93-98
Darwish, Kareem ; Mubarak, Hamdy ; Eldesouki, Mohamed ; Abdelali, Ahmed ; Samih, Younes ; Alharbi, Randah ; Attia, Mohammed ; Magdy, Walid ; Kallmeyer, Laura. / Multi-dialect Arabic POS tagging : A CRF approach. LREC 2018 - 11th International Conference on Language Resources and Evaluation. editor / Hitoshi Isahara ; Bente Maegaard ; Stelios Piperidis ; Christopher Cieri ; Thierry Declerck ; Koiti Hasida ; Helene Mazo ; Khalid Choukri ; Sara Goggi ; Joseph Mariani ; Asuncion Moreno ; Nicoletta Calzolari ; Jan Odijk ; Takenobu Tokunaga. European Language Resources Association (ELRA), 2019. pp. 93-98
@inproceedings{f3221e6c256c4872b86b830c6f0b44a6,
title = "Multi-dialect Arabic POS tagging: A CRF approach",
abstract = "This paper introduces a new dataset of POS-tagged Arabic tweets in four major dialects along with tagging guidelines. The data, which we are releasing publicly, includes tweets in Egyptian, Levantine, Gulf, and Maghrebi, with 350 tweets for each dialect with appropriate train/test/development splits for 5-fold cross validation. We use a Conditional Random Fields (CRF) sequence labeler to train POS taggers for each dialect and examine the effect of cross and joint dialect training, and give benchmark results for the datasets. Using clitic n-grams, clitic metatypes, and stem templates as features, we were able to train a joint model that can correctly tag four different dialects with an average accuracy of 89.3{\%}.",
keywords = "Arabic dialects, CRF, POS tagging",
author = "Kareem Darwish and Hamdy Mubarak and Mohamed Eldesouki and Ahmed Abdelali and Younes Samih and Randah Alharbi and Mohammed Attia and Walid Magdy and Laura Kallmeyer",
year = "2019",
month = "1",
day = "1",
language = "English",
pages = "93--98",
editor = "Hitoshi Isahara and Bente Maegaard and Stelios Piperidis and Christopher Cieri and Thierry Declerck and Koiti Hasida and Helene Mazo and Khalid Choukri and Sara Goggi and Joseph Mariani and Asuncion Moreno and Nicoletta Calzolari and Jan Odijk and Takenobu Tokunaga",
booktitle = "LREC 2018 - 11th International Conference on Language Resources and Evaluation",
publisher = "European Language Resources Association (ELRA)",

}

TY - GEN

T1 - Multi-dialect Arabic POS tagging

T2 - A CRF approach

AU - Darwish, Kareem

AU - Mubarak, Hamdy

AU - Eldesouki, Mohamed

AU - Abdelali, Ahmed

AU - Samih, Younes

AU - Alharbi, Randah

AU - Attia, Mohammed

AU - Magdy, Walid

AU - Kallmeyer, Laura

PY - 2019/1/1

Y1 - 2019/1/1

N2 - This paper introduces a new dataset of POS-tagged Arabic tweets in four major dialects along with tagging guidelines. The data, which we are releasing publicly, includes tweets in Egyptian, Levantine, Gulf, and Maghrebi, with 350 tweets for each dialect with appropriate train/test/development splits for 5-fold cross validation. We use a Conditional Random Fields (CRF) sequence labeler to train POS taggers for each dialect and examine the effect of cross and joint dialect training, and give benchmark results for the datasets. Using clitic n-grams, clitic metatypes, and stem templates as features, we were able to train a joint model that can correctly tag four different dialects with an average accuracy of 89.3%.

AB - This paper introduces a new dataset of POS-tagged Arabic tweets in four major dialects along with tagging guidelines. The data, which we are releasing publicly, includes tweets in Egyptian, Levantine, Gulf, and Maghrebi, with 350 tweets for each dialect with appropriate train/test/development splits for 5-fold cross validation. We use a Conditional Random Fields (CRF) sequence labeler to train POS taggers for each dialect and examine the effect of cross and joint dialect training, and give benchmark results for the datasets. Using clitic n-grams, clitic metatypes, and stem templates as features, we were able to train a joint model that can correctly tag four different dialects with an average accuracy of 89.3%.

KW - Arabic dialects

KW - CRF

KW - POS tagging

UR - http://www.scopus.com/inward/record.url?scp=85058117794&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85058117794&partnerID=8YFLogxK

M3 - Conference contribution

SP - 93

EP - 98

BT - LREC 2018 - 11th International Conference on Language Resources and Evaluation

A2 - Isahara, Hitoshi

A2 - Maegaard, Bente

A2 - Piperidis, Stelios

A2 - Cieri, Christopher

A2 - Declerck, Thierry

A2 - Hasida, Koiti

A2 - Mazo, Helene

A2 - Choukri, Khalid

A2 - Goggi, Sara

A2 - Mariani, Joseph

A2 - Moreno, Asuncion

A2 - Calzolari, Nicoletta

A2 - Odijk, Jan

A2 - Tokunaga, Takenobu

PB - European Language Resources Association (ELRA)

ER -