New metrics for meaningful evaluation of informally structured speech retrieval

Maria Eskevich, Walid Magdy, Gareth J F Jones

Research output: Chapter in Book/Report/Conference proceedingConference contribution

15 Citations (Scopus)

Abstract

Search effectiveness for tasks where the retrieval units are clearly defined documents is generally evaluated using standard measures such as mean average precision (MAP). However, many practical speech search tasks focus on content within large spoken files lacking defined structure. These data must be segmented into smaller units for search which may only partially overlap with relevant material. We introduce two new metrics for the evaluation of search effectiveness for informally structured speech data: mean average segment precision (MASP) which measures retrieval performance in terms of both content segmentation and ranking with respect to relevance; and mean average segment distance-weighted precision (MASDWP) which takes into account the distance between the start of the relevant segment and the retrieved segment. We demonstrate the effectiveness of these new metrics on a retrieval test collection based on the AMI meeting corpus.

Original languageEnglish
Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Pages170-181
Number of pages12
Volume7224 LNCS
DOIs
Publication statusPublished - 27 Apr 2012
Externally publishedYes
Event34th European Conference on Information Retrieval, ECIR 2012 - Barcelona, Spain
Duration: 1 Apr 20125 Apr 2012

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume7224 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other34th European Conference on Information Retrieval, ECIR 2012
CountrySpain
CityBarcelona
Period1/4/125/4/12

Fingerprint

Retrieval
Metric
Evaluation
Unit
Overlap
Ranking
Segmentation
Speech
Demonstrate

Keywords

  • evaluation metrics
  • informally structured speech
  • Speech retrieval

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Cite this

Eskevich, M., Magdy, W., & Jones, G. J. F. (2012). New metrics for meaningful evaluation of informally structured speech retrieval. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 7224 LNCS, pp. 170-181). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 7224 LNCS). https://doi.org/10.1007/978-3-642-28997-2_15

New metrics for meaningful evaluation of informally structured speech retrieval. / Eskevich, Maria; Magdy, Walid; Jones, Gareth J F.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 7224 LNCS 2012. p. 170-181 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 7224 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Eskevich, M, Magdy, W & Jones, GJF 2012, New metrics for meaningful evaluation of informally structured speech retrieval. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 7224 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 7224 LNCS, pp. 170-181, 34th European Conference on Information Retrieval, ECIR 2012, Barcelona, Spain, 1/4/12. https://doi.org/10.1007/978-3-642-28997-2_15
Eskevich M, Magdy W, Jones GJF. New metrics for meaningful evaluation of informally structured speech retrieval. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 7224 LNCS. 2012. p. 170-181. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-642-28997-2_15
Eskevich, Maria ; Magdy, Walid ; Jones, Gareth J F. / New metrics for meaningful evaluation of informally structured speech retrieval. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 7224 LNCS 2012. pp. 170-181 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{1f26538792e947a9a37689a60377c2bd,
title = "New metrics for meaningful evaluation of informally structured speech retrieval",
abstract = "Search effectiveness for tasks where the retrieval units are clearly defined documents is generally evaluated using standard measures such as mean average precision (MAP). However, many practical speech search tasks focus on content within large spoken files lacking defined structure. These data must be segmented into smaller units for search which may only partially overlap with relevant material. We introduce two new metrics for the evaluation of search effectiveness for informally structured speech data: mean average segment precision (MASP) which measures retrieval performance in terms of both content segmentation and ranking with respect to relevance; and mean average segment distance-weighted precision (MASDWP) which takes into account the distance between the start of the relevant segment and the retrieved segment. We demonstrate the effectiveness of these new metrics on a retrieval test collection based on the AMI meeting corpus.",
keywords = "evaluation metrics, informally structured speech, Speech retrieval",
author = "Maria Eskevich and Walid Magdy and Jones, {Gareth J F}",
year = "2012",
month = "4",
day = "27",
doi = "10.1007/978-3-642-28997-2_15",
language = "English",
isbn = "9783642289965",
volume = "7224 LNCS",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
pages = "170--181",
booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

}

TY - GEN

T1 - New metrics for meaningful evaluation of informally structured speech retrieval

AU - Eskevich, Maria

AU - Magdy, Walid

AU - Jones, Gareth J F

PY - 2012/4/27

Y1 - 2012/4/27

N2 - Search effectiveness for tasks where the retrieval units are clearly defined documents is generally evaluated using standard measures such as mean average precision (MAP). However, many practical speech search tasks focus on content within large spoken files lacking defined structure. These data must be segmented into smaller units for search which may only partially overlap with relevant material. We introduce two new metrics for the evaluation of search effectiveness for informally structured speech data: mean average segment precision (MASP) which measures retrieval performance in terms of both content segmentation and ranking with respect to relevance; and mean average segment distance-weighted precision (MASDWP) which takes into account the distance between the start of the relevant segment and the retrieved segment. We demonstrate the effectiveness of these new metrics on a retrieval test collection based on the AMI meeting corpus.

AB - Search effectiveness for tasks where the retrieval units are clearly defined documents is generally evaluated using standard measures such as mean average precision (MAP). However, many practical speech search tasks focus on content within large spoken files lacking defined structure. These data must be segmented into smaller units for search which may only partially overlap with relevant material. We introduce two new metrics for the evaluation of search effectiveness for informally structured speech data: mean average segment precision (MASP) which measures retrieval performance in terms of both content segmentation and ranking with respect to relevance; and mean average segment distance-weighted precision (MASDWP) which takes into account the distance between the start of the relevant segment and the retrieved segment. We demonstrate the effectiveness of these new metrics on a retrieval test collection based on the AMI meeting corpus.

KW - evaluation metrics

KW - informally structured speech

KW - Speech retrieval

UR - http://www.scopus.com/inward/record.url?scp=84860186165&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84860186165&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-28997-2_15

DO - 10.1007/978-3-642-28997-2_15

M3 - Conference contribution

AN - SCOPUS:84860186165

SN - 9783642289965

VL - 7224 LNCS

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 170

EP - 181

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

ER -