PRES: A score metric for evaluating recall-oriented information retrieval applications

Walid Magdy, Gareth J.F. Jones

Research output: Chapter in Book/Report/Conference proceedingConference contribution

69 Citations (Scopus)

Abstract

Information retrieval (IR) evaluation scores are generally designed to measure the effectiveness with which relevant documents are identified and retrieved. Many scores have been proposed for this purpose over the years. These have primarily focused on aspects of precision and recall, and while these are often discussed with equal importance, in practice most attention has been given to precision focused metrics. Even for recall-oriented IR tasks of growing importance, such as patent retrieval, these precision based scores remain the primary evaluation measures. Our study examines different evaluation measures for a recall-oriented patent retrieval task and demonstrates the limitations of the current scores in comparing different IR systems for this task. We introduce PRES, a novel evaluation metric for this type of application taking account of recall and the user's search effort. The behaviour of PRES is demonstrated on 48 runs from the CLEF-IP 2009 patent retrieval track. A full analysis of the performance of PRES shows its suitability for measuring the retrieval effectiveness of systems from a recall focused perspective taking into account the user's expected search effort.

Original languageEnglish
Title of host publicationSIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
Pages611-618
Number of pages8
DOIs
Publication statusPublished - 1 Sep 2010
Event33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2010 - Geneva, Switzerland
Duration: 19 Jul 201023 Jul 2010

Publication series

NameSIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Other

Other33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2010
CountrySwitzerland
CityGeneva
Period19/7/1023/7/10

    Fingerprint

Keywords

  • Evaluation metric
  • PRES
  • Patent retrieval
  • Recall-oriented information retrieval

ASJC Scopus subject areas

  • Information Systems

Cite this

Magdy, W., & Jones, G. J. F. (2010). PRES: A score metric for evaluating recall-oriented information retrieval applications. In SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 611-618). (SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval). https://doi.org/10.1145/1835449.1835551