Evaluating the effectiveness of and patterns of interactions with automated searching assistance

Bernard Jansen, Michael D. McNeese

Research output: Contribution to journalArticle

57 Citations (Scopus)

Abstract

We report quantitative and qualitative results of an empirical evaluation to determine whether automated assistance improves searching performance and when searchers desire system intervention in the search process. Forty participants interacted with two fully functional information retrieval systems in a counterbalanced, within-participant study. The systems were identical in all respects except that one offered automated assistance and the other did not. The study used a client-side automated assistance application, an approximately 500,000-document Text REtrieval Conference content collection, and six topics. Results indicate that automated assistance can improve searching performance. However, the improvement is less dramatic than one might expect, with an approximately 20% performance increase, as measured by the number of user-selected relevant documents. Concerning patterns of interaction, we identified 1,879 occurrences of searcher-system interactions and classified them into 9 major categories and 27 subcategories or states. Results indicate that there are predictable patterns of times when searchers desire and implement searching assistance. The most common three-state pattern is Execute Query-View Results: With Scrolling-View Assistance. Searchers appear receptive to automated assistance; there is a 71% implementation rate. There does not seem to be a correlation between the use of assistance and previous searching performance. We discuss the implications for the design of information retrieval systems and future research directions.

Original languageEnglish
Pages (from-to)1480-1503
Number of pages24
JournalJournal of the American Society for Information Science and Technology
Volume56
Issue number14
DOIs
Publication statusPublished - Dec 2005
Externally publishedYes

Fingerprint

Information retrieval systems
assistance
interaction
information retrieval
performance
system of action
Interaction
evaluation

ASJC Scopus subject areas

  • Information Systems
  • Library and Information Sciences

Cite this

Evaluating the effectiveness of and patterns of interactions with automated searching assistance. / Jansen, Bernard; McNeese, Michael D.

In: Journal of the American Society for Information Science and Technology, Vol. 56, No. 14, 12.2005, p. 1480-1503.

Research output: Contribution to journalArticle

@article{d26a4d9e66b94380ac3b86cc662763bf,
title = "Evaluating the effectiveness of and patterns of interactions with automated searching assistance",
abstract = "We report quantitative and qualitative results of an empirical evaluation to determine whether automated assistance improves searching performance and when searchers desire system intervention in the search process. Forty participants interacted with two fully functional information retrieval systems in a counterbalanced, within-participant study. The systems were identical in all respects except that one offered automated assistance and the other did not. The study used a client-side automated assistance application, an approximately 500,000-document Text REtrieval Conference content collection, and six topics. Results indicate that automated assistance can improve searching performance. However, the improvement is less dramatic than one might expect, with an approximately 20{\%} performance increase, as measured by the number of user-selected relevant documents. Concerning patterns of interaction, we identified 1,879 occurrences of searcher-system interactions and classified them into 9 major categories and 27 subcategories or states. Results indicate that there are predictable patterns of times when searchers desire and implement searching assistance. The most common three-state pattern is Execute Query-View Results: With Scrolling-View Assistance. Searchers appear receptive to automated assistance; there is a 71{\%} implementation rate. There does not seem to be a correlation between the use of assistance and previous searching performance. We discuss the implications for the design of information retrieval systems and future research directions.",
author = "Bernard Jansen and McNeese, {Michael D.}",
year = "2005",
month = "12",
doi = "10.1002/asi.20242",
language = "English",
volume = "56",
pages = "1480--1503",
journal = "Journal of the Association for Information Science and Technology",
issn = "2330-1635",
publisher = "John Wiley and Sons Ltd",
number = "14",

}

TY - JOUR

T1 - Evaluating the effectiveness of and patterns of interactions with automated searching assistance

AU - Jansen, Bernard

AU - McNeese, Michael D.

PY - 2005/12

Y1 - 2005/12

N2 - We report quantitative and qualitative results of an empirical evaluation to determine whether automated assistance improves searching performance and when searchers desire system intervention in the search process. Forty participants interacted with two fully functional information retrieval systems in a counterbalanced, within-participant study. The systems were identical in all respects except that one offered automated assistance and the other did not. The study used a client-side automated assistance application, an approximately 500,000-document Text REtrieval Conference content collection, and six topics. Results indicate that automated assistance can improve searching performance. However, the improvement is less dramatic than one might expect, with an approximately 20% performance increase, as measured by the number of user-selected relevant documents. Concerning patterns of interaction, we identified 1,879 occurrences of searcher-system interactions and classified them into 9 major categories and 27 subcategories or states. Results indicate that there are predictable patterns of times when searchers desire and implement searching assistance. The most common three-state pattern is Execute Query-View Results: With Scrolling-View Assistance. Searchers appear receptive to automated assistance; there is a 71% implementation rate. There does not seem to be a correlation between the use of assistance and previous searching performance. We discuss the implications for the design of information retrieval systems and future research directions.

AB - We report quantitative and qualitative results of an empirical evaluation to determine whether automated assistance improves searching performance and when searchers desire system intervention in the search process. Forty participants interacted with two fully functional information retrieval systems in a counterbalanced, within-participant study. The systems were identical in all respects except that one offered automated assistance and the other did not. The study used a client-side automated assistance application, an approximately 500,000-document Text REtrieval Conference content collection, and six topics. Results indicate that automated assistance can improve searching performance. However, the improvement is less dramatic than one might expect, with an approximately 20% performance increase, as measured by the number of user-selected relevant documents. Concerning patterns of interaction, we identified 1,879 occurrences of searcher-system interactions and classified them into 9 major categories and 27 subcategories or states. Results indicate that there are predictable patterns of times when searchers desire and implement searching assistance. The most common three-state pattern is Execute Query-View Results: With Scrolling-View Assistance. Searchers appear receptive to automated assistance; there is a 71% implementation rate. There does not seem to be a correlation between the use of assistance and previous searching performance. We discuss the implications for the design of information retrieval systems and future research directions.

UR - http://www.scopus.com/inward/record.url?scp=30544439959&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=30544439959&partnerID=8YFLogxK

U2 - 10.1002/asi.20242

DO - 10.1002/asi.20242

M3 - Article

VL - 56

SP - 1480

EP - 1503

JO - Journal of the Association for Information Science and Technology

JF - Journal of the Association for Information Science and Technology

SN - 2330-1635

IS - 14

ER -