BIROn - Birkbeck Institutional Research Online

    A meta-evaluation of evaluation methods for diversified search

    Kingrani, Suneel Kumar and Levene, Mark and Zhang, Dell (2018) A meta-evaluation of evaluation methods for diversified search. In: Pasi, G. and Piwowarski, B. and Azzopardi, L. and Hanbury, A. (eds.) Advances in Information Retrieval. ECIR 2018. Lecture Notes in Computer Science 10772. Springer, pp. 550-555. ISBN 9783319769400.

    20719.pdf - Author's Accepted Manuscript

    Download (224kB) | Preview


    For the evaluation of diversified search results, a number of different methods have been proposed in the literature. Prior to making use of such evaluation methods, it is important to have a good understanding of how diversity and relevance contribute to the performance metric of each method. In this paper, we use the statistical technique ANOVA to analyse and compare three representative evaluation methods for diversified search, namely alpha-nDCG, MAP-IA, and ERR-IA, on the TREC-2009 Web track dataset. It is shown that the performance scores provided by those evaluation methods can indeed reflect two crucial aspects of diversity --- richness and evenness --- as well as relevance, though to different degrees.


    Item Type: Book Section
    Additional Information: The final publication is available at Springer via the link above.
    Keyword(s) / Subject(s): web search, diversity, evaluation
    School: School of Business, Economics & Informatics > Computer Science and Information Systems
    Research Centres and Institutes: Birkbeck Knowledge Lab, Data Analytics, Birkbeck Institute for
    Depositing User: Dell Zhang
    Date Deposited: 22 Aug 2018 16:55
    Last Modified: 25 Apr 2022 17:54


    Activity Overview
    6 month trend
    6 month trend

    Additional statistics are available via IRStats2.

    Archive Staff Only (login required)

    Edit/View Item Edit/View Item