A statistical approach to crosslingual natural language tasks

David Pinto, Jorge Civera, Alberto Barron, Alfons Juan, Paolo Rosso

Research output: Contribution to journalArticle

35 Citations (Scopus)

Abstract

The existence of huge volumes of documents written in multiple languages on Internet leads to investigate novel algorithmic approaches to deal with information of this kind. However, most crosslingual natural language processing (NLP) tasks consider a decoupled approach in which monolingual NLP techniques are applied along with an independent translation process. This two-step approach is too sensitive to translation errors, and in general to the accumulative effect of errors. To solve this problem, we propose to use a direct probabilistic crosslingual NLP system which integrates both steps, translation and the specific NLP task, into a single one. In order to perform this integrated approach to crosslingual tasks, we propose to use the statistical IBM 1 word alignment model (M1). The M1 model may show a non-monotonic behaviour when aligning words from a sentence in a source language to words from another sentence in a different, target language. This is the case of languages with different word order. In English, for instance, adjectives appear before nouns, whereas in Spanish it is exactly the opposite. The successful experimental results reported in three different tasks - text classification, information retrieval and plagiarism analysis - highlight the benefits of the statistical integrated approach proposed in this work.

Original languageEnglish
Pages (from-to)51-60
Number of pages10
JournalJournal of Algorithms
Volume64
Issue number1
DOIs
Publication statusPublished - 2009
Externally publishedYes

    Fingerprint

Keywords

  • Crosslingual data
  • IBM translation models
  • Information retrieval
  • Natural language processing
  • Plagiarism analysis
  • Text classification

ASJC Scopus subject areas

  • Computational Mathematics
  • Control and Optimization
  • Computational Theory and Mathematics

Cite this