Automatic extraction of informative blocks from webpages

Sandip Debnath, Prasenjit Mitra, C. Lee Giles

Research output: Contribution to conferencePaper

48 Citations (Scopus)

Abstract

Search engines crawl and index webpages depending upon their informative content. However, webpages - especially dynamically generated ones - contain items that cannot be classified as the "primary content", e.g., navigation side-bars, advertisements, copyright notices, etc. Most end-users search for the primary content, and largely do not seek the non-informative content. A tool that assists an end-user or application to search and process information from web-pages automatically, must separate the "primary content blocks" from the other blocks. In this paper, two new algorithms, ContentExtractor, and FeatureExtractor are proposed. The algorithms identify primary content blocks by i) looking for blocks that do not occur a large number of times across webpages and ii) looking for blocks with desired features respectively. They identify the primary content blocks with high precision and recall, reduce the storage requirement for search engines, result in smaller indexes and thereby faster search times, and better user satisfaction. While operating on several thousand webpages obtained from 11 news websites, our algorithms significantly outperform the Entropy-based algorithm proposed by Lin and Ho [7] in both accuracy and run-time.

Original languageEnglish
Pages1722-1726
Number of pages5
DOIs
Publication statusPublished - 1 Dec 2005
Event20th Annual ACM Symposium on Applied Computing - Santa Fe, NM, United States
Duration: 13 Mar 200517 Mar 2005

Conference

Conference20th Annual ACM Symposium on Applied Computing
CountryUnited States
CitySanta Fe, NM
Period13/3/0517/3/05

    Fingerprint

Keywords

  • Data Mining
  • Electronic Publishing
  • Information Systems Applications

ASJC Scopus subject areas

  • Software

Cite this

Debnath, S., Mitra, P., & Lee Giles, C. (2005). Automatic extraction of informative blocks from webpages. 1722-1726. Paper presented at 20th Annual ACM Symposium on Applied Computing, Santa Fe, NM, United States. https://doi.org/10.1145/1066677.1067065