Kingrani, Suneel Kumar and Levene, Mark and Zhang, Dell (2018) A meta-evaluation of evaluation methods for diversified search. In: Pasi, G. and Piwowarski, B. and Azzopardi, L. and Hanbury, A. (eds.) Advances in Information Retrieval. ECIR 2018. Lecture Notes in Computer Science 10772. Springer, pp. 550-555. ISBN 9783319769400.
|
Text
20719.pdf - Author's Accepted Manuscript Download (224kB) | Preview |
Abstract
For the evaluation of diversified search results, a number of different methods have been proposed in the literature. Prior to making use of such evaluation methods, it is important to have a good understanding of how diversity and relevance contribute to the performance metric of each method. In this paper, we use the statistical technique ANOVA to analyse and compare three representative evaluation methods for diversified search, namely alpha-nDCG, MAP-IA, and ERR-IA, on the TREC-2009 Web track dataset. It is shown that the performance scores provided by those evaluation methods can indeed reflect two crucial aspects of diversity --- richness and evenness --- as well as relevance, though to different degrees.
Metadata
| Item Type: | Book Section |
|---|---|
| Additional Information: | The final publication is available at Springer via the link above. |
| Keyword(s) / Subject(s): | web search, diversity, evaluation |
| School: | Birkbeck Faculties and Schools > Faculty of Science > School of Computing and Mathematical Sciences |
| Research Centres and Institutes: | Birkbeck Knowledge Lab, Data Analytics, Birkbeck Institute for |
| Depositing User: | Dell Zhang |
| Date Deposited: | 22 Aug 2018 16:55 |
| Last Modified: | 09 Aug 2023 12:42 |
| URI: | https://eprints.bbk.ac.uk/id/eprint/20719 |
Statistics
Additional statistics are available via IRStats2.
Archive Staff Only (login required)
