Abstract
Learning to adapt in a new setting is a common challenge to our knowledge and capability. New life would be easier if we actively pursued supervision from the right mentor chosen with our relevant but limited prior knowledge. This variant principle of active learning seems intuitively useful to many domain adaptation problems. In this paper, we substantiate its power for advancing automatic ranking adaptation, which is important in web search since it's prohibitive to gather enough labeled data for every search domain for fully training domain-specific rankers. For the cost-effectiveness, it is expected that only those most informative instances in target domain are collected to annotate while we can still utilize the abundant ranking knowledge in source domain. We propose a unified ranking framework to mutually reinforce the active selection of informative target-domain queries and the appropriate weighting of source training data as related prior knowledge. We select to annotate those target queries whose documents' order most disagrees among the members of a committee built on the mixture of source training data and the already selected target data. Then the replenished labeled set is used to adjust the importance of source queries for enhancing their rank transfer. This procedure iterates until labeling budget exhausts. Based on LETOR3.0 and Yahoo! Learning to Rank Challenge data sets, our approach significantly outperforms the random query annotation commonly used in ranking adaptation and the active rank learner on target-domain data only.
| Original language | English |
|---|---|
| Title of host publication | SIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval |
| Pages | 115-124 |
| Number of pages | 10 |
| DOIs | |
| Publication status | Published - 1 Sep 2011 |
| Externally published | Yes |
| Event | 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'11 - Beijing, China Duration: 24 Jul 2011 → 28 Jul 2011 |
Other
| Other | 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'11 |
|---|---|
| Country | China |
| City | Beijing |
| Period | 24/7/11 → 28/7/11 |
Fingerprint
Keywords
- Active learning
- Query by committee
- Ranking adaptation
ASJC Scopus subject areas
- Information Systems
Cite this
Relevant knowledge helps in choosing right teacher : Active query selection for ranking adaptation. / Cai, Peng; Gao, Wei; Zhou, Aoying; Wong, Kam Fai.
SIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2011. p. 115-124.Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
}
TY - GEN
T1 - Relevant knowledge helps in choosing right teacher
T2 - Active query selection for ranking adaptation
AU - Cai, Peng
AU - Gao, Wei
AU - Zhou, Aoying
AU - Wong, Kam Fai
PY - 2011/9/1
Y1 - 2011/9/1
N2 - Learning to adapt in a new setting is a common challenge to our knowledge and capability. New life would be easier if we actively pursued supervision from the right mentor chosen with our relevant but limited prior knowledge. This variant principle of active learning seems intuitively useful to many domain adaptation problems. In this paper, we substantiate its power for advancing automatic ranking adaptation, which is important in web search since it's prohibitive to gather enough labeled data for every search domain for fully training domain-specific rankers. For the cost-effectiveness, it is expected that only those most informative instances in target domain are collected to annotate while we can still utilize the abundant ranking knowledge in source domain. We propose a unified ranking framework to mutually reinforce the active selection of informative target-domain queries and the appropriate weighting of source training data as related prior knowledge. We select to annotate those target queries whose documents' order most disagrees among the members of a committee built on the mixture of source training data and the already selected target data. Then the replenished labeled set is used to adjust the importance of source queries for enhancing their rank transfer. This procedure iterates until labeling budget exhausts. Based on LETOR3.0 and Yahoo! Learning to Rank Challenge data sets, our approach significantly outperforms the random query annotation commonly used in ranking adaptation and the active rank learner on target-domain data only.
AB - Learning to adapt in a new setting is a common challenge to our knowledge and capability. New life would be easier if we actively pursued supervision from the right mentor chosen with our relevant but limited prior knowledge. This variant principle of active learning seems intuitively useful to many domain adaptation problems. In this paper, we substantiate its power for advancing automatic ranking adaptation, which is important in web search since it's prohibitive to gather enough labeled data for every search domain for fully training domain-specific rankers. For the cost-effectiveness, it is expected that only those most informative instances in target domain are collected to annotate while we can still utilize the abundant ranking knowledge in source domain. We propose a unified ranking framework to mutually reinforce the active selection of informative target-domain queries and the appropriate weighting of source training data as related prior knowledge. We select to annotate those target queries whose documents' order most disagrees among the members of a committee built on the mixture of source training data and the already selected target data. Then the replenished labeled set is used to adjust the importance of source queries for enhancing their rank transfer. This procedure iterates until labeling budget exhausts. Based on LETOR3.0 and Yahoo! Learning to Rank Challenge data sets, our approach significantly outperforms the random query annotation commonly used in ranking adaptation and the active rank learner on target-domain data only.
KW - Active learning
KW - Query by committee
KW - Ranking adaptation
UR - http://www.scopus.com/inward/record.url?scp=80052127267&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=80052127267&partnerID=8YFLogxK
U2 - 10.1145/2009916.2009935
DO - 10.1145/2009916.2009935
M3 - Conference contribution
SN - 9781450309349
SP - 115
EP - 124
BT - SIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval
ER -