Relevant knowledge helps in choosing right teacher: Active query selection for ranking adaptation

Peng Cai, Wei Gao, Aoying Zhou, Kam Fai Wong

Research output: Chapter in Book/Report/Conference proceedingConference contribution

15 Citations (Scopus)

Abstract

Learning to adapt in a new setting is a common challenge to our knowledge and capability. New life would be easier if we actively pursued supervision from the right mentor chosen with our relevant but limited prior knowledge. This variant principle of active learning seems intuitively useful to many domain adaptation problems. In this paper, we substantiate its power for advancing automatic ranking adaptation, which is important in web search since it's prohibitive to gather enough labeled data for every search domain for fully training domain-specific rankers. For the cost-effectiveness, it is expected that only those most informative instances in target domain are collected to annotate while we can still utilize the abundant ranking knowledge in source domain. We propose a unified ranking framework to mutually reinforce the active selection of informative target-domain queries and the appropriate weighting of source training data as related prior knowledge. We select to annotate those target queries whose documents' order most disagrees among the members of a committee built on the mixture of source training data and the already selected target data. Then the replenished labeled set is used to adjust the importance of source queries for enhancing their rank transfer. This procedure iterates until labeling budget exhausts. Based on LETOR3.0 and Yahoo! Learning to Rank Challenge data sets, our approach significantly outperforms the random query annotation commonly used in ranking adaptation and the active rank learner on target-domain data only.

Original languageEnglish
Title of host publicationSIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval
Pages115-124
Number of pages10
DOIs
Publication statusPublished - 1 Sep 2011
Externally publishedYes
Event34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'11 - Beijing, China
Duration: 24 Jul 201128 Jul 2011

Other

Other34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'11
CountryChina
CityBeijing
Period24/7/1128/7/11

Fingerprint

Cost effectiveness
Labeling
Problem-Based Learning

Keywords

  • Active learning
  • Query by committee
  • Ranking adaptation

ASJC Scopus subject areas

  • Information Systems

Cite this

Cai, P., Gao, W., Zhou, A., & Wong, K. F. (2011). Relevant knowledge helps in choosing right teacher: Active query selection for ranking adaptation. In SIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 115-124) https://doi.org/10.1145/2009916.2009935

Relevant knowledge helps in choosing right teacher : Active query selection for ranking adaptation. / Cai, Peng; Gao, Wei; Zhou, Aoying; Wong, Kam Fai.

SIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2011. p. 115-124.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Cai, P, Gao, W, Zhou, A & Wong, KF 2011, Relevant knowledge helps in choosing right teacher: Active query selection for ranking adaptation. in SIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 115-124, 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'11, Beijing, China, 24/7/11. https://doi.org/10.1145/2009916.2009935
Cai P, Gao W, Zhou A, Wong KF. Relevant knowledge helps in choosing right teacher: Active query selection for ranking adaptation. In SIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2011. p. 115-124 https://doi.org/10.1145/2009916.2009935
Cai, Peng ; Gao, Wei ; Zhou, Aoying ; Wong, Kam Fai. / Relevant knowledge helps in choosing right teacher : Active query selection for ranking adaptation. SIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2011. pp. 115-124
@inproceedings{0a1b57ee1eb5422d9bdbedbc4efa2402,
title = "Relevant knowledge helps in choosing right teacher: Active query selection for ranking adaptation",
abstract = "Learning to adapt in a new setting is a common challenge to our knowledge and capability. New life would be easier if we actively pursued supervision from the right mentor chosen with our relevant but limited prior knowledge. This variant principle of active learning seems intuitively useful to many domain adaptation problems. In this paper, we substantiate its power for advancing automatic ranking adaptation, which is important in web search since it's prohibitive to gather enough labeled data for every search domain for fully training domain-specific rankers. For the cost-effectiveness, it is expected that only those most informative instances in target domain are collected to annotate while we can still utilize the abundant ranking knowledge in source domain. We propose a unified ranking framework to mutually reinforce the active selection of informative target-domain queries and the appropriate weighting of source training data as related prior knowledge. We select to annotate those target queries whose documents' order most disagrees among the members of a committee built on the mixture of source training data and the already selected target data. Then the replenished labeled set is used to adjust the importance of source queries for enhancing their rank transfer. This procedure iterates until labeling budget exhausts. Based on LETOR3.0 and Yahoo! Learning to Rank Challenge data sets, our approach significantly outperforms the random query annotation commonly used in ranking adaptation and the active rank learner on target-domain data only.",
keywords = "Active learning, Query by committee, Ranking adaptation",
author = "Peng Cai and Wei Gao and Aoying Zhou and Wong, {Kam Fai}",
year = "2011",
month = "9",
day = "1",
doi = "10.1145/2009916.2009935",
language = "English",
isbn = "9781450309349",
pages = "115--124",
booktitle = "SIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval",

}

TY - GEN

T1 - Relevant knowledge helps in choosing right teacher

T2 - Active query selection for ranking adaptation

AU - Cai, Peng

AU - Gao, Wei

AU - Zhou, Aoying

AU - Wong, Kam Fai

PY - 2011/9/1

Y1 - 2011/9/1

N2 - Learning to adapt in a new setting is a common challenge to our knowledge and capability. New life would be easier if we actively pursued supervision from the right mentor chosen with our relevant but limited prior knowledge. This variant principle of active learning seems intuitively useful to many domain adaptation problems. In this paper, we substantiate its power for advancing automatic ranking adaptation, which is important in web search since it's prohibitive to gather enough labeled data for every search domain for fully training domain-specific rankers. For the cost-effectiveness, it is expected that only those most informative instances in target domain are collected to annotate while we can still utilize the abundant ranking knowledge in source domain. We propose a unified ranking framework to mutually reinforce the active selection of informative target-domain queries and the appropriate weighting of source training data as related prior knowledge. We select to annotate those target queries whose documents' order most disagrees among the members of a committee built on the mixture of source training data and the already selected target data. Then the replenished labeled set is used to adjust the importance of source queries for enhancing their rank transfer. This procedure iterates until labeling budget exhausts. Based on LETOR3.0 and Yahoo! Learning to Rank Challenge data sets, our approach significantly outperforms the random query annotation commonly used in ranking adaptation and the active rank learner on target-domain data only.

AB - Learning to adapt in a new setting is a common challenge to our knowledge and capability. New life would be easier if we actively pursued supervision from the right mentor chosen with our relevant but limited prior knowledge. This variant principle of active learning seems intuitively useful to many domain adaptation problems. In this paper, we substantiate its power for advancing automatic ranking adaptation, which is important in web search since it's prohibitive to gather enough labeled data for every search domain for fully training domain-specific rankers. For the cost-effectiveness, it is expected that only those most informative instances in target domain are collected to annotate while we can still utilize the abundant ranking knowledge in source domain. We propose a unified ranking framework to mutually reinforce the active selection of informative target-domain queries and the appropriate weighting of source training data as related prior knowledge. We select to annotate those target queries whose documents' order most disagrees among the members of a committee built on the mixture of source training data and the already selected target data. Then the replenished labeled set is used to adjust the importance of source queries for enhancing their rank transfer. This procedure iterates until labeling budget exhausts. Based on LETOR3.0 and Yahoo! Learning to Rank Challenge data sets, our approach significantly outperforms the random query annotation commonly used in ranking adaptation and the active rank learner on target-domain data only.

KW - Active learning

KW - Query by committee

KW - Ranking adaptation

UR - http://www.scopus.com/inward/record.url?scp=80052127267&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=80052127267&partnerID=8YFLogxK

U2 - 10.1145/2009916.2009935

DO - 10.1145/2009916.2009935

M3 - Conference contribution

AN - SCOPUS:80052127267

SN - 9781450309349

SP - 115

EP - 124

BT - SIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval

ER -