PRES: A score metric for evaluating recall-oriented information retrieval applications

Walid Magdy, Gareth J F Jones

Research output: Chapter in Book/Report/Conference proceedingConference contribution

67 Citations (Scopus)

Abstract

Information retrieval (IR) evaluation scores are generally designed to measure the effectiveness with which relevant documents are identified and retrieved. Many scores have been proposed for this purpose over the years. These have primarily focused on aspects of precision and recall, and while these are often discussed with equal importance, in practice most attention has been given to precision focused metrics. Even for recall-oriented IR tasks of growing importance, such as patent retrieval, these precision based scores remain the primary evaluation measures. Our study examines different evaluation measures for a recall-oriented patent retrieval task and demonstrates the limitations of the current scores in comparing different IR systems for this task. We introduce PRES, a novel evaluation metric for this type of application taking account of recall and the user's search effort. The behaviour of PRES is demonstrated on 48 runs from the CLEF-IP 2009 patent retrieval track. A full analysis of the performance of PRES shows its suitability for measuring the retrieval effectiveness of systems from a recall focused perspective taking into account the user's expected search effort.

Original languageEnglish
Title of host publicationSIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
Pages611-618
Number of pages8
DOIs
Publication statusPublished - 1 Sep 2010
Externally publishedYes
Event33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2010 - Geneva, Switzerland
Duration: 19 Jul 201023 Jul 2010

Other

Other33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2010
CountrySwitzerland
CityGeneva
Period19/7/1023/7/10

Fingerprint

Information retrieval
Information retrieval systems

Keywords

  • Evaluation metric
  • Patent retrieval
  • PRES
  • Recall-oriented information retrieval

ASJC Scopus subject areas

  • Information Systems

Cite this

Magdy, W., & Jones, G. J. F. (2010). PRES: A score metric for evaluating recall-oriented information retrieval applications. In SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 611-618) https://doi.org/10.1145/1835449.1835551

PRES : A score metric for evaluating recall-oriented information retrieval applications. / Magdy, Walid; Jones, Gareth J F.

SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 2010. p. 611-618.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Magdy, W & Jones, GJF 2010, PRES: A score metric for evaluating recall-oriented information retrieval applications. in SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 611-618, 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2010, Geneva, Switzerland, 19/7/10. https://doi.org/10.1145/1835449.1835551
Magdy W, Jones GJF. PRES: A score metric for evaluating recall-oriented information retrieval applications. In SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 2010. p. 611-618 https://doi.org/10.1145/1835449.1835551
Magdy, Walid ; Jones, Gareth J F. / PRES : A score metric for evaluating recall-oriented information retrieval applications. SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 2010. pp. 611-618
@inproceedings{4174c8be25f0425e893387123b9df672,
title = "PRES: A score metric for evaluating recall-oriented information retrieval applications",
abstract = "Information retrieval (IR) evaluation scores are generally designed to measure the effectiveness with which relevant documents are identified and retrieved. Many scores have been proposed for this purpose over the years. These have primarily focused on aspects of precision and recall, and while these are often discussed with equal importance, in practice most attention has been given to precision focused metrics. Even for recall-oriented IR tasks of growing importance, such as patent retrieval, these precision based scores remain the primary evaluation measures. Our study examines different evaluation measures for a recall-oriented patent retrieval task and demonstrates the limitations of the current scores in comparing different IR systems for this task. We introduce PRES, a novel evaluation metric for this type of application taking account of recall and the user's search effort. The behaviour of PRES is demonstrated on 48 runs from the CLEF-IP 2009 patent retrieval track. A full analysis of the performance of PRES shows its suitability for measuring the retrieval effectiveness of systems from a recall focused perspective taking into account the user's expected search effort.",
keywords = "Evaluation metric, Patent retrieval, PRES, Recall-oriented information retrieval",
author = "Walid Magdy and Jones, {Gareth J F}",
year = "2010",
month = "9",
day = "1",
doi = "10.1145/1835449.1835551",
language = "English",
isbn = "9781605588964",
pages = "611--618",
booktitle = "SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval",

}

TY - GEN

T1 - PRES

T2 - A score metric for evaluating recall-oriented information retrieval applications

AU - Magdy, Walid

AU - Jones, Gareth J F

PY - 2010/9/1

Y1 - 2010/9/1

N2 - Information retrieval (IR) evaluation scores are generally designed to measure the effectiveness with which relevant documents are identified and retrieved. Many scores have been proposed for this purpose over the years. These have primarily focused on aspects of precision and recall, and while these are often discussed with equal importance, in practice most attention has been given to precision focused metrics. Even for recall-oriented IR tasks of growing importance, such as patent retrieval, these precision based scores remain the primary evaluation measures. Our study examines different evaluation measures for a recall-oriented patent retrieval task and demonstrates the limitations of the current scores in comparing different IR systems for this task. We introduce PRES, a novel evaluation metric for this type of application taking account of recall and the user's search effort. The behaviour of PRES is demonstrated on 48 runs from the CLEF-IP 2009 patent retrieval track. A full analysis of the performance of PRES shows its suitability for measuring the retrieval effectiveness of systems from a recall focused perspective taking into account the user's expected search effort.

AB - Information retrieval (IR) evaluation scores are generally designed to measure the effectiveness with which relevant documents are identified and retrieved. Many scores have been proposed for this purpose over the years. These have primarily focused on aspects of precision and recall, and while these are often discussed with equal importance, in practice most attention has been given to precision focused metrics. Even for recall-oriented IR tasks of growing importance, such as patent retrieval, these precision based scores remain the primary evaluation measures. Our study examines different evaluation measures for a recall-oriented patent retrieval task and demonstrates the limitations of the current scores in comparing different IR systems for this task. We introduce PRES, a novel evaluation metric for this type of application taking account of recall and the user's search effort. The behaviour of PRES is demonstrated on 48 runs from the CLEF-IP 2009 patent retrieval track. A full analysis of the performance of PRES shows its suitability for measuring the retrieval effectiveness of systems from a recall focused perspective taking into account the user's expected search effort.

KW - Evaluation metric

KW - Patent retrieval

KW - PRES

KW - Recall-oriented information retrieval

UR - http://www.scopus.com/inward/record.url?scp=77956042082&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77956042082&partnerID=8YFLogxK

U2 - 10.1145/1835449.1835551

DO - 10.1145/1835449.1835551

M3 - Conference contribution

AN - SCOPUS:77956042082

SN - 9781605588964

SP - 611

EP - 618

BT - SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

ER -