Detecting semantic uncertainty by learning hedge cues in sentences using an HMM

Xiujun Li, Wei Gao, Jude W. Shavlik

Research output: Contribution to journalConference article

2 Citations (Scopus)

Abstract

Detecting speculative assertions is essential to distinguish semantically uncertain information from the factual ones in text. This is critical to the trustworthiness of many intelligent systems that are based on information retrieval and natural language processing techniques, such as question answering or information extraction. We empirically explore three fundamental issues of uncertainty de- tection: (1) the predictive ability of different learning methods on this task; (2) whether using unlabeled data can lead to a more ac- curate model; and (3) whether closed-domain training or cross- domain training is better. For these purposes, we adopt two statistical learning approaches to this problem: the commonly used bag-of-words model based on Naive Bayes, and the sequence labeling approach using a Hidden Markov Model (HMM). We empirically compare between our two approaches as well as externally compare with prior results on the CoNLL-2010 Shared Task 1. Overall, our results are promising: (1) on Wikipedia and biomedical datasets, the HMM model improves over Naive Bayes up to 17.4% and 29.0%, respectively, in terms of absolute F score; (2) compared to CoNLL-2010 systems, our best HMM model achieves 62.9% F score with MLE parameter estimation and 64.0% with EM parameter estimation on Wikipedia dataset, both outperforming the best result (60.2%) of the CoNLL-2010 systems, but our results on the biomedical dataset are less impressive; (3) when the expression ability of a model (e.g., Naive Bayes) is not strong enough, cross-domain training is helpful, and when a model is powerful (e.g., HMM), cross-domain training may produce biased parameters; and (4) under Maximum Likelihood Estimation, combining the unlabeled examples with the labeled helps.

Original languageEnglish
Pages (from-to)30-37
Number of pages8
JournalCEUR Workshop Proceedings
Volume1204
Publication statusPublished - 1 Jan 2014
EventWorkshop on Semantic Matching in Information Retrieval, SMIR 2014 - Gold Coast, Australia
Duration: 11 Jul 2014 → …

    Fingerprint

Keywords

  • Cross-domain training
  • Hedge cues
  • HMM
  • Naive Bayes
  • Uncertainty detection

ASJC Scopus subject areas

  • Computer Science(all)

Cite this