Eyes don't lie

Predicting machine translation quality using eye movement

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Poorly translated text is often disfluent and difficult to read. In contrast, well-formed translations require less time to process. In this paper, we model the differences in reading patterns of Machine Translation (MT) evaluators using novel features extracted from their gaze data, and we learn to predict the quality scores given by those evaluators. We test our predictions in a pairwise ranking scenario, measuring Kendall's tau correlation with the judgments. We show that our features provide information beyond fluency, and can be combined with BLEU for better predictions. Furthermore, our results show that reading patterns can be used to build semi-automatic metrics that anticipate the scores given by the evaluators.

Original languageEnglish
Title of host publication2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference
PublisherAssociation for Computational Linguistics (ACL)
Pages1082-1088
Number of pages7
ISBN (Electronic)9781941643914
Publication statusPublished - 2016
Event15th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - San Diego, United States
Duration: 12 Jun 201617 Jun 2016

Other

Other15th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016
CountryUnited States
CitySan Diego
Period12/6/1617/6/16

Fingerprint

Eye movements
ranking
scenario
Eye Movements
Prediction
Machine Translation
time
Fluency
Ranking
Scenarios

ASJC Scopus subject areas

  • Computer Science Applications
  • Linguistics and Language
  • Language and Linguistics

Cite this

Sajjad, H., Guzmán, F., Durrani, N., Abdelali, A., Bouamor, H., Temnikova, I., & Vogel, S. (2016). Eyes don't lie: Predicting machine translation quality using eye movement. In 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference (pp. 1082-1088). Association for Computational Linguistics (ACL).

Eyes don't lie : Predicting machine translation quality using eye movement. / Sajjad, Hassan; Guzmán, Francisco; Durrani, Nadir; Abdelali, Ahmed; Bouamor, Houda; Temnikova, Irina; Vogel, Stephan.

2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference. Association for Computational Linguistics (ACL), 2016. p. 1082-1088.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Sajjad, H, Guzmán, F, Durrani, N, Abdelali, A, Bouamor, H, Temnikova, I & Vogel, S 2016, Eyes don't lie: Predicting machine translation quality using eye movement. in 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference. Association for Computational Linguistics (ACL), pp. 1082-1088, 15th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016, San Diego, United States, 12/6/16.
Sajjad H, Guzmán F, Durrani N, Abdelali A, Bouamor H, Temnikova I et al. Eyes don't lie: Predicting machine translation quality using eye movement. In 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference. Association for Computational Linguistics (ACL). 2016. p. 1082-1088
Sajjad, Hassan ; Guzmán, Francisco ; Durrani, Nadir ; Abdelali, Ahmed ; Bouamor, Houda ; Temnikova, Irina ; Vogel, Stephan. / Eyes don't lie : Predicting machine translation quality using eye movement. 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference. Association for Computational Linguistics (ACL), 2016. pp. 1082-1088
@inproceedings{8daef544f8b54e9e9eb23f202316b6f0,
title = "Eyes don't lie: Predicting machine translation quality using eye movement",
abstract = "Poorly translated text is often disfluent and difficult to read. In contrast, well-formed translations require less time to process. In this paper, we model the differences in reading patterns of Machine Translation (MT) evaluators using novel features extracted from their gaze data, and we learn to predict the quality scores given by those evaluators. We test our predictions in a pairwise ranking scenario, measuring Kendall's tau correlation with the judgments. We show that our features provide information beyond fluency, and can be combined with BLEU for better predictions. Furthermore, our results show that reading patterns can be used to build semi-automatic metrics that anticipate the scores given by the evaluators.",
author = "Hassan Sajjad and Francisco Guzm{\'a}n and Nadir Durrani and Ahmed Abdelali and Houda Bouamor and Irina Temnikova and Stephan Vogel",
year = "2016",
language = "English",
pages = "1082--1088",
booktitle = "2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference",
publisher = "Association for Computational Linguistics (ACL)",

}

TY - GEN

T1 - Eyes don't lie

T2 - Predicting machine translation quality using eye movement

AU - Sajjad, Hassan

AU - Guzmán, Francisco

AU - Durrani, Nadir

AU - Abdelali, Ahmed

AU - Bouamor, Houda

AU - Temnikova, Irina

AU - Vogel, Stephan

PY - 2016

Y1 - 2016

N2 - Poorly translated text is often disfluent and difficult to read. In contrast, well-formed translations require less time to process. In this paper, we model the differences in reading patterns of Machine Translation (MT) evaluators using novel features extracted from their gaze data, and we learn to predict the quality scores given by those evaluators. We test our predictions in a pairwise ranking scenario, measuring Kendall's tau correlation with the judgments. We show that our features provide information beyond fluency, and can be combined with BLEU for better predictions. Furthermore, our results show that reading patterns can be used to build semi-automatic metrics that anticipate the scores given by the evaluators.

AB - Poorly translated text is often disfluent and difficult to read. In contrast, well-formed translations require less time to process. In this paper, we model the differences in reading patterns of Machine Translation (MT) evaluators using novel features extracted from their gaze data, and we learn to predict the quality scores given by those evaluators. We test our predictions in a pairwise ranking scenario, measuring Kendall's tau correlation with the judgments. We show that our features provide information beyond fluency, and can be combined with BLEU for better predictions. Furthermore, our results show that reading patterns can be used to build semi-automatic metrics that anticipate the scores given by the evaluators.

UR - http://www.scopus.com/inward/record.url?scp=84994130654&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84994130654&partnerID=8YFLogxK

M3 - Conference contribution

SP - 1082

EP - 1088

BT - 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference

PB - Association for Computational Linguistics (ACL)

ER -