Finding a haystack in haystacks - Simultaneous identification of concepts in large bio-medical corpora

Ying Liu, Lucian V. Lita, R. Stefan Niculescu, Prasenjit Mitra, C. Lee Giles

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Since nearly all information is now created digitally, large text databases have become more prevalent than ever. Automatically mining information from these databases proves to be a challenge due to slow pattern/string matching techniques. In this paper we introduce a new, fast multi-string pattern matching method called the Block Suffix Shifting (BSS) algorithm, which is based on the well known Aho-Chorasick algorithm. The advantages of our algorithm include: the ability to exploit the natural structure of text, perform significant character shifting, avoid useless backtracking jumps, efficient matching time and avoid the typical "sub-string" false positive errors. Our algorithm is applicable to many fields with free text, such as the health care domain and the scientific document field. In this paper, we apply the BSS algorithm to health care data and mine hundreds of thousands of medical concepts from a large Electronic Medical Record (EMR) corpora simultaneously and efficiently. Experimental results show the superiority of our algorithm when compared with the top of the line multi-string matching algorithms (the Aho-Corasick and the Wu-Manber algorithm).

Original languageEnglish
Title of host publicationSociety for Industrial and Applied Mathematics - 8th SIAM International Conference on Data Mining 2008, Proceedings in Applied Mathematics 130
Pages668-679
Number of pages12
Volume2
Publication statusPublished - 2008
Externally publishedYes
Event8th SIAM International Conference on Data Mining 2008, Applied Mathematics 130 - Atlanta, GA
Duration: 24 Apr 200826 Apr 2008

Other

Other8th SIAM International Conference on Data Mining 2008, Applied Mathematics 130
CityAtlanta, GA
Period24/4/0826/4/08

    Fingerprint

ASJC Scopus subject areas

  • Information Systems
  • Software
  • Signal Processing
  • Theoretical Computer Science

Cite this

Liu, Y., Lita, L. V., Niculescu, R. S., Mitra, P., & Giles, C. L. (2008). Finding a haystack in haystacks - Simultaneous identification of concepts in large bio-medical corpora. In Society for Industrial and Applied Mathematics - 8th SIAM International Conference on Data Mining 2008, Proceedings in Applied Mathematics 130 (Vol. 2, pp. 668-679)