Optimizing for sentence-level BLEU+1 yields short translations

Preslav Nakov, Francisco Guzmán, Stephan Vogel

Research output: Contribution to conferencePaper

30 Citations (Scopus)

Abstract

We study a problem with pairwise ranking optimization (PRO): That it tends to yield too short translations. We find that this is partially due to the inadequate smoothing in PRO's BLEU+1, which boosts the precision component of BLEU but leaves the brevity penalty unchanged, thus destroying the balance between the two, compared to BLEU. It is also partially due to PRO optimizing for a sentence-level score without a global view on the overall length, which introducing a bias towards short translations; we show that letting PRO optimize a corpus-level BLEU yields a perfect length. Finally, we find some residual bias due to the interaction of PRO with BLEU+1: such a bias does not exist for a version of MIRA with sentence-level BLEU+1. We propose several ways to fix the length problem of PRO, including smoothing the brevity penalty, scaling the effective reference length, grounding the precision component, and unclipping the brevity penalty, which yield sizable improvements in test BLEU on two Arabic-English datasets: IWSLT (+0.65) and NIST (+0.37).

Original languageEnglish
Pages1979-1994
Number of pages16
Publication statusPublished - 1 Dec 2012
Event24th International Conference on Computational Linguistics, COLING 2012 - Mumbai, India
Duration: 8 Dec 201215 Dec 2012

Other

Other24th International Conference on Computational Linguistics, COLING 2012
CountryIndia
CityMumbai
Period8/12/1215/12/12

Keywords

  • MERT
  • MIRA
  • Parameter optimization
  • PRO
  • Statistical machine translation

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Language and Linguistics
  • Linguistics and Language

Fingerprint Dive into the research topics of 'Optimizing for sentence-level BLEU+1 yields short translations'. Together they form a unique fingerprint.

  • Cite this

    Nakov, P., Guzmán, F., & Vogel, S. (2012). Optimizing for sentence-level BLEU+1 yields short translations. 1979-1994. Paper presented at 24th International Conference on Computational Linguistics, COLING 2012, Mumbai, India.