Machine reading comprehension for answer re-ranking in customer support chatbots

Momchil Hardalov, Ivan Koychev, Preslav Nakov

Research output: Contribution to journalArticle

Abstract

Recent advances in deep neural networks, language modeling and language generation have introduced new ideas to the field of conversational agents. As a result, deep neural models such as sequence-to-sequence, memory networks, and the Transformer have become key ingredients of state-of-the-art dialog systems. While those models are able to generate meaningful responses even in unseen situations, they need a lot of training data to build a reliable model. Thus, most real-world systems have used traditional approaches based on information retrieval (IR) and even hand-crafted rules, due to their robustness and effectiveness, especially for narrow-focused conversations. Here, we present a method that adapts a deep neural architecture from the domain of machine reading comprehension to re-rank the suggested answers from different models using the question as a context. We train our model using negative sampling based on question-answer pairs from the Twitter Customer Support Dataset. The experimental results show that our re-ranking framework can improve the performance in terms of word overlap and semantics both for individual models as well as for model combinations.

Original languageEnglish
Article number82
JournalInformation (Switzerland)
Volume10
Issue number3
DOIs
Publication statusPublished - 1 Jan 2019

Fingerprint

Information retrieval
Semantics
Sampling
Data storage equipment
Deep neural networks

Keywords

  • Answer re-ranking
  • Chatbots
  • Conversational agents
  • Information retrieval
  • Machine reading comprehension
  • Question answering

ASJC Scopus subject areas

  • Information Systems

Cite this

Machine reading comprehension for answer re-ranking in customer support chatbots. / Hardalov, Momchil; Koychev, Ivan; Nakov, Preslav.

In: Information (Switzerland), Vol. 10, No. 3, 82, 01.01.2019.

Research output: Contribution to journalArticle

@article{a755f876cfa84fb2a7b02e774b97726d,
title = "Machine reading comprehension for answer re-ranking in customer support chatbots",
abstract = "Recent advances in deep neural networks, language modeling and language generation have introduced new ideas to the field of conversational agents. As a result, deep neural models such as sequence-to-sequence, memory networks, and the Transformer have become key ingredients of state-of-the-art dialog systems. While those models are able to generate meaningful responses even in unseen situations, they need a lot of training data to build a reliable model. Thus, most real-world systems have used traditional approaches based on information retrieval (IR) and even hand-crafted rules, due to their robustness and effectiveness, especially for narrow-focused conversations. Here, we present a method that adapts a deep neural architecture from the domain of machine reading comprehension to re-rank the suggested answers from different models using the question as a context. We train our model using negative sampling based on question-answer pairs from the Twitter Customer Support Dataset. The experimental results show that our re-ranking framework can improve the performance in terms of word overlap and semantics both for individual models as well as for model combinations.",
keywords = "Answer re-ranking, Chatbots, Conversational agents, Information retrieval, Machine reading comprehension, Question answering",
author = "Momchil Hardalov and Ivan Koychev and Preslav Nakov",
year = "2019",
month = "1",
day = "1",
doi = "10.3390/info10030082",
language = "English",
volume = "10",
journal = "Information (Switzerland)",
issn = "2078-2489",
publisher = "Multidisciplinary Digital Publishing Institute (MDPI)",
number = "3",

}

TY - JOUR

T1 - Machine reading comprehension for answer re-ranking in customer support chatbots

AU - Hardalov, Momchil

AU - Koychev, Ivan

AU - Nakov, Preslav

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Recent advances in deep neural networks, language modeling and language generation have introduced new ideas to the field of conversational agents. As a result, deep neural models such as sequence-to-sequence, memory networks, and the Transformer have become key ingredients of state-of-the-art dialog systems. While those models are able to generate meaningful responses even in unseen situations, they need a lot of training data to build a reliable model. Thus, most real-world systems have used traditional approaches based on information retrieval (IR) and even hand-crafted rules, due to their robustness and effectiveness, especially for narrow-focused conversations. Here, we present a method that adapts a deep neural architecture from the domain of machine reading comprehension to re-rank the suggested answers from different models using the question as a context. We train our model using negative sampling based on question-answer pairs from the Twitter Customer Support Dataset. The experimental results show that our re-ranking framework can improve the performance in terms of word overlap and semantics both for individual models as well as for model combinations.

AB - Recent advances in deep neural networks, language modeling and language generation have introduced new ideas to the field of conversational agents. As a result, deep neural models such as sequence-to-sequence, memory networks, and the Transformer have become key ingredients of state-of-the-art dialog systems. While those models are able to generate meaningful responses even in unseen situations, they need a lot of training data to build a reliable model. Thus, most real-world systems have used traditional approaches based on information retrieval (IR) and even hand-crafted rules, due to their robustness and effectiveness, especially for narrow-focused conversations. Here, we present a method that adapts a deep neural architecture from the domain of machine reading comprehension to re-rank the suggested answers from different models using the question as a context. We train our model using negative sampling based on question-answer pairs from the Twitter Customer Support Dataset. The experimental results show that our re-ranking framework can improve the performance in terms of word overlap and semantics both for individual models as well as for model combinations.

KW - Answer re-ranking

KW - Chatbots

KW - Conversational agents

KW - Information retrieval

KW - Machine reading comprehension

KW - Question answering

UR - http://www.scopus.com/inward/record.url?scp=85063884453&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85063884453&partnerID=8YFLogxK

U2 - 10.3390/info10030082

DO - 10.3390/info10030082

M3 - Article

AN - SCOPUS:85063884453

VL - 10

JO - Information (Switzerland)

JF - Information (Switzerland)

SN - 2078-2489

IS - 3

M1 - 82

ER -