BIROn - Birkbeck Institutional Research Online

    A meta-evaluation of evaluation methods for diversified search

    Kingrani, Suneel Kumar and Levene, Mark and Zhang, Dell (2018) A meta-evaluation of evaluation methods for diversified search. In: Pasi, G. and Piwowarski, B. and Azzopardi, L. and Hanbury, A. (eds.) Advances in Information Retrieval. ECIR 2018. Lecture Notes in Computer Science 10772. Springer, pp. 550-555. ISBN 9783319769400. (In Press)

    [img]
    Preview
    Text
    20719.pdf - Author's Accepted Manuscript

    Download (224kB) | Preview

    Abstract

    For the evaluation of diversified search results, a number of different methods have been proposed in the literature. Prior to making use of such evaluation methods, it is important to have a good understanding of how diversity and relevance contribute to the performance metric of each method. In this paper, we use the statistical technique ANOVA to analyse and compare three representative evaluation methods for diversified search, namely alpha-nDCG, MAP-IA, and ERR-IA, on the TREC-2009 Web track dataset. It is shown that the performance scores provided by those evaluation methods can indeed reflect two crucial aspects of diversity --- richness and evenness --- as well as relevance, though to different degrees.

    Metadata

    Item Type: Book Section
    Additional Information: The final publication is available at Springer via the link above.
    Keyword(s) / Subject(s): web search, diversity, evaluation
    School: Birkbeck Schools and Departments > School of Business, Economics & Informatics > Computer Science and Information Systems
    Research Centre: Birkbeck Knowledge Lab
    Depositing User: Dr Dell Zhang
    Date Deposited: 22 Aug 2018 16:55
    Last Modified: 29 Jul 2019 04:14
    URI: http://eprints.bbk.ac.uk/id/eprint/20719

    Statistics

    Downloads
    Activity Overview
    81Downloads
    56Hits

    Additional statistics are available via IRStats2.

    Archive Staff Only (login required)

    Edit/View Item Edit/View Item