Analyzing optimization for statistical machine translation: MERT learns verbosity, PRO learns length

Francisco Guzmán, Preslav Nakov, Stephan Vogel

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Citations (Scopus)

Abstract

We study the impact of source length and verbosity of the tuning dataset on the performance of parameter optimizers such as MERT and PRO for statistical machine translation. In particular, we test whether the verbosity of the resulting translations can be modified by varying the length or the verbosity of the tuning sentences. We find that MERT learns the tuning set verbosity very well, while PRO is sensitive to both the verbosity and the length of the source sentences in the tuning set; yet, overall PRO learns best from high-verbosity tuning datasets. Given these dependencies, and potentially some other such as amount of reordering, number of unknown words, syntactic complexity, and evaluation measure, to mention just a few, we argue for the need of controlled evaluation scenarios, so that the selection of tuning set and optimization strategy does not overshadow scientific advances in modeling or decoding. In the mean time, until we develop such controlled scenarios, we recommend using PRO with a large verbosity tuning set, which, in our experiments, yields highest BLEU across datasets and language pairs.

Original languageEnglish
Title of host publicationCoNLL 2015 - 19th Conference on Computational Natural Language Learning, Proceedings
PublisherAssociation for Computational Linguistics (ACL)
Pages62-72
Number of pages11
ISBN (Electronic)9781941643778
Publication statusPublished - 1 Jan 2015
Event19th Conference on Computational Natural Language Learning, CoNLL 2015 - Beijing, China
Duration: 30 Jul 201531 Jul 2015

Publication series

NameCoNLL 2015 - 19th Conference on Computational Natural Language Learning, Proceedings

Conference

Conference19th Conference on Computational Natural Language Learning, CoNLL 2015
CountryChina
CityBeijing
Period30/7/1531/7/15

Fingerprint

Tuning
scenario
evaluation
experiment
language
performance
Syntactics
Decoding
time
Experiments

ASJC Scopus subject areas

  • Artificial Intelligence
  • Human-Computer Interaction
  • Linguistics and Language

Cite this

Guzmán, F., Nakov, P., & Vogel, S. (2015). Analyzing optimization for statistical machine translation: MERT learns verbosity, PRO learns length. In CoNLL 2015 - 19th Conference on Computational Natural Language Learning, Proceedings (pp. 62-72). (CoNLL 2015 - 19th Conference on Computational Natural Language Learning, Proceedings). Association for Computational Linguistics (ACL).

Analyzing optimization for statistical machine translation : MERT learns verbosity, PRO learns length. / Guzmán, Francisco; Nakov, Preslav; Vogel, Stephan.

CoNLL 2015 - 19th Conference on Computational Natural Language Learning, Proceedings. Association for Computational Linguistics (ACL), 2015. p. 62-72 (CoNLL 2015 - 19th Conference on Computational Natural Language Learning, Proceedings).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Guzmán, F, Nakov, P & Vogel, S 2015, Analyzing optimization for statistical machine translation: MERT learns verbosity, PRO learns length. in CoNLL 2015 - 19th Conference on Computational Natural Language Learning, Proceedings. CoNLL 2015 - 19th Conference on Computational Natural Language Learning, Proceedings, Association for Computational Linguistics (ACL), pp. 62-72, 19th Conference on Computational Natural Language Learning, CoNLL 2015, Beijing, China, 30/7/15.
Guzmán F, Nakov P, Vogel S. Analyzing optimization for statistical machine translation: MERT learns verbosity, PRO learns length. In CoNLL 2015 - 19th Conference on Computational Natural Language Learning, Proceedings. Association for Computational Linguistics (ACL). 2015. p. 62-72. (CoNLL 2015 - 19th Conference on Computational Natural Language Learning, Proceedings).
Guzmán, Francisco ; Nakov, Preslav ; Vogel, Stephan. / Analyzing optimization for statistical machine translation : MERT learns verbosity, PRO learns length. CoNLL 2015 - 19th Conference on Computational Natural Language Learning, Proceedings. Association for Computational Linguistics (ACL), 2015. pp. 62-72 (CoNLL 2015 - 19th Conference on Computational Natural Language Learning, Proceedings).
@inproceedings{481d161369214dddb08a029c4fe5f18c,
title = "Analyzing optimization for statistical machine translation: MERT learns verbosity, PRO learns length",
abstract = "We study the impact of source length and verbosity of the tuning dataset on the performance of parameter optimizers such as MERT and PRO for statistical machine translation. In particular, we test whether the verbosity of the resulting translations can be modified by varying the length or the verbosity of the tuning sentences. We find that MERT learns the tuning set verbosity very well, while PRO is sensitive to both the verbosity and the length of the source sentences in the tuning set; yet, overall PRO learns best from high-verbosity tuning datasets. Given these dependencies, and potentially some other such as amount of reordering, number of unknown words, syntactic complexity, and evaluation measure, to mention just a few, we argue for the need of controlled evaluation scenarios, so that the selection of tuning set and optimization strategy does not overshadow scientific advances in modeling or decoding. In the mean time, until we develop such controlled scenarios, we recommend using PRO with a large verbosity tuning set, which, in our experiments, yields highest BLEU across datasets and language pairs.",
author = "Francisco Guzm{\'a}n and Preslav Nakov and Stephan Vogel",
year = "2015",
month = "1",
day = "1",
language = "English",
series = "CoNLL 2015 - 19th Conference on Computational Natural Language Learning, Proceedings",
publisher = "Association for Computational Linguistics (ACL)",
pages = "62--72",
booktitle = "CoNLL 2015 - 19th Conference on Computational Natural Language Learning, Proceedings",

}

TY - GEN

T1 - Analyzing optimization for statistical machine translation

T2 - MERT learns verbosity, PRO learns length

AU - Guzmán, Francisco

AU - Nakov, Preslav

AU - Vogel, Stephan

PY - 2015/1/1

Y1 - 2015/1/1

N2 - We study the impact of source length and verbosity of the tuning dataset on the performance of parameter optimizers such as MERT and PRO for statistical machine translation. In particular, we test whether the verbosity of the resulting translations can be modified by varying the length or the verbosity of the tuning sentences. We find that MERT learns the tuning set verbosity very well, while PRO is sensitive to both the verbosity and the length of the source sentences in the tuning set; yet, overall PRO learns best from high-verbosity tuning datasets. Given these dependencies, and potentially some other such as amount of reordering, number of unknown words, syntactic complexity, and evaluation measure, to mention just a few, we argue for the need of controlled evaluation scenarios, so that the selection of tuning set and optimization strategy does not overshadow scientific advances in modeling or decoding. In the mean time, until we develop such controlled scenarios, we recommend using PRO with a large verbosity tuning set, which, in our experiments, yields highest BLEU across datasets and language pairs.

AB - We study the impact of source length and verbosity of the tuning dataset on the performance of parameter optimizers such as MERT and PRO for statistical machine translation. In particular, we test whether the verbosity of the resulting translations can be modified by varying the length or the verbosity of the tuning sentences. We find that MERT learns the tuning set verbosity very well, while PRO is sensitive to both the verbosity and the length of the source sentences in the tuning set; yet, overall PRO learns best from high-verbosity tuning datasets. Given these dependencies, and potentially some other such as amount of reordering, number of unknown words, syntactic complexity, and evaluation measure, to mention just a few, we argue for the need of controlled evaluation scenarios, so that the selection of tuning set and optimization strategy does not overshadow scientific advances in modeling or decoding. In the mean time, until we develop such controlled scenarios, we recommend using PRO with a large verbosity tuning set, which, in our experiments, yields highest BLEU across datasets and language pairs.

UR - http://www.scopus.com/inward/record.url?scp=85044434069&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85044434069&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85044434069

T3 - CoNLL 2015 - 19th Conference on Computational Natural Language Learning, Proceedings

SP - 62

EP - 72

BT - CoNLL 2015 - 19th Conference on Computational Natural Language Learning, Proceedings

PB - Association for Computational Linguistics (ACL)

ER -