Domain adaptation using neural network joint model

Research output: Contribution to journalArticle

Abstract

We explore neural joint models for the task of domain adaptation in machine translation in two ways: (i) we apply state-of-the-art domain adaptation techniques, such as mixture modelling and data selection using the recently proposed Neural Network Joint Model (NNJM) (Devlin et al., 2014); (ii) we propose two novel approaches to perform adaptation through instance weighting and weight readjustment in the NNJM framework. In our first approach, we propose a pair of models called Neural Domain Adaptation Models (NDAM) that minimizes the cross entropy by regularizing the loss function with respect to in-domain (and optionally to out-domain) model. In the second approach, we present a set of Neural Fusion Models (NFM) that combines the in- and the out-domain models by readjusting their parameters based on the in-domain data.We evaluated our models on the standard task of translating English-to-German and Arabic-to-English TED talks. The NDAM models achieved better perplexities and modest BLEU improvements compared to the baseline NNJM, trained either on in-domain or on a concatenation of in- and out-domain data. On the other hand, the NFM models obtained significant improvements of up to +0.9 and +0.7 BLEU points, respectively. We also demonstrate improvements over existing adaptation methods such as instance weighting, phrasetable fill-up, linear and log-linear interpolations.

Original languageEnglish
JournalComputer Speech and Language
DOIs
Publication statusAccepted/In press - 6 May 2016

Fingerprint

Joint Model
Network Model
Neural Networks
Neural networks
Domain Model
Model
Weighting
Fusion
Cross-entropy
Mixture Modeling
Machine Translation
Linear Interpolation
Concatenation
Loss Function
Fusion reactions
Baseline
Minimise

Keywords

  • Distributed representation of texts
  • Domain adaptation
  • Machine translation
  • Neural network joint model
  • Noise contrastive estimation

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Software
  • Human-Computer Interaction

Cite this

Domain adaptation using neural network joint model. / Joty, Shafiq; Durrani, Nadir; Sajjad, Hassan; Abdelali, Ahmed.

In: Computer Speech and Language, 06.05.2016.

Research output: Contribution to journalArticle

@article{3cd2e27a49824bec8aa70e8bc12f80ad,
title = "Domain adaptation using neural network joint model",
abstract = "We explore neural joint models for the task of domain adaptation in machine translation in two ways: (i) we apply state-of-the-art domain adaptation techniques, such as mixture modelling and data selection using the recently proposed Neural Network Joint Model (NNJM) (Devlin et al., 2014); (ii) we propose two novel approaches to perform adaptation through instance weighting and weight readjustment in the NNJM framework. In our first approach, we propose a pair of models called Neural Domain Adaptation Models (NDAM) that minimizes the cross entropy by regularizing the loss function with respect to in-domain (and optionally to out-domain) model. In the second approach, we present a set of Neural Fusion Models (NFM) that combines the in- and the out-domain models by readjusting their parameters based on the in-domain data.We evaluated our models on the standard task of translating English-to-German and Arabic-to-English TED talks. The NDAM models achieved better perplexities and modest BLEU improvements compared to the baseline NNJM, trained either on in-domain or on a concatenation of in- and out-domain data. On the other hand, the NFM models obtained significant improvements of up to +0.9 and +0.7 BLEU points, respectively. We also demonstrate improvements over existing adaptation methods such as instance weighting, phrasetable fill-up, linear and log-linear interpolations.",
keywords = "Distributed representation of texts, Domain adaptation, Machine translation, Neural network joint model, Noise contrastive estimation",
author = "Shafiq Joty and Nadir Durrani and Hassan Sajjad and Ahmed Abdelali",
year = "2016",
month = "5",
day = "6",
doi = "10.1016/j.csl.2016.12.006",
language = "English",
journal = "Computer Speech and Language",
issn = "0885-2308",
publisher = "Academic Press Inc.",

}

TY - JOUR

T1 - Domain adaptation using neural network joint model

AU - Joty, Shafiq

AU - Durrani, Nadir

AU - Sajjad, Hassan

AU - Abdelali, Ahmed

PY - 2016/5/6

Y1 - 2016/5/6

N2 - We explore neural joint models for the task of domain adaptation in machine translation in two ways: (i) we apply state-of-the-art domain adaptation techniques, such as mixture modelling and data selection using the recently proposed Neural Network Joint Model (NNJM) (Devlin et al., 2014); (ii) we propose two novel approaches to perform adaptation through instance weighting and weight readjustment in the NNJM framework. In our first approach, we propose a pair of models called Neural Domain Adaptation Models (NDAM) that minimizes the cross entropy by regularizing the loss function with respect to in-domain (and optionally to out-domain) model. In the second approach, we present a set of Neural Fusion Models (NFM) that combines the in- and the out-domain models by readjusting their parameters based on the in-domain data.We evaluated our models on the standard task of translating English-to-German and Arabic-to-English TED talks. The NDAM models achieved better perplexities and modest BLEU improvements compared to the baseline NNJM, trained either on in-domain or on a concatenation of in- and out-domain data. On the other hand, the NFM models obtained significant improvements of up to +0.9 and +0.7 BLEU points, respectively. We also demonstrate improvements over existing adaptation methods such as instance weighting, phrasetable fill-up, linear and log-linear interpolations.

AB - We explore neural joint models for the task of domain adaptation in machine translation in two ways: (i) we apply state-of-the-art domain adaptation techniques, such as mixture modelling and data selection using the recently proposed Neural Network Joint Model (NNJM) (Devlin et al., 2014); (ii) we propose two novel approaches to perform adaptation through instance weighting and weight readjustment in the NNJM framework. In our first approach, we propose a pair of models called Neural Domain Adaptation Models (NDAM) that minimizes the cross entropy by regularizing the loss function with respect to in-domain (and optionally to out-domain) model. In the second approach, we present a set of Neural Fusion Models (NFM) that combines the in- and the out-domain models by readjusting their parameters based on the in-domain data.We evaluated our models on the standard task of translating English-to-German and Arabic-to-English TED talks. The NDAM models achieved better perplexities and modest BLEU improvements compared to the baseline NNJM, trained either on in-domain or on a concatenation of in- and out-domain data. On the other hand, the NFM models obtained significant improvements of up to +0.9 and +0.7 BLEU points, respectively. We also demonstrate improvements over existing adaptation methods such as instance weighting, phrasetable fill-up, linear and log-linear interpolations.

KW - Distributed representation of texts

KW - Domain adaptation

KW - Machine translation

KW - Neural network joint model

KW - Noise contrastive estimation

UR - http://www.scopus.com/inward/record.url?scp=85009988978&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85009988978&partnerID=8YFLogxK

U2 - 10.1016/j.csl.2016.12.006

DO - 10.1016/j.csl.2016.12.006

M3 - Article

JO - Computer Speech and Language

JF - Computer Speech and Language

SN - 0885-2308

ER -