BIROn - Birkbeck Institutional Research Online

    A meta-evaluation of evaluation methods for diversified search

    Kingrani, Suneel Kumar and Levene, Mark and Zhang, Dell (2018) A meta-evaluation of evaluation methods for diversified search. In: Pasi, G. and Piwowarski, B. and Azzopardi, L. and Hanbury, A. (eds.) Advances in Information Retrieval. ECIR 2018. Lecture Notes in Computer Science 10772. Springer, pp. 550-555. ISBN 9783319769400.

    [img]
    Preview
    Text
    20719.pdf - Author's Accepted Manuscript

    Download (224kB) | Preview

    Abstract

    For the evaluation of diversified search results, a number of different methods have been proposed in the literature. Prior to making use of such evaluation methods, it is important to have a good understanding of how diversity and relevance contribute to the performance metric of each method. In this paper, we use the statistical technique ANOVA to analyse and compare three representative evaluation methods for diversified search, namely alpha-nDCG, MAP-IA, and ERR-IA, on the TREC-2009 Web track dataset. It is shown that the performance scores provided by those evaluation methods can indeed reflect two crucial aspects of diversity --- richness and evenness --- as well as relevance, though to different degrees.

    Metadata

    Item Type: Book Section
    Additional Information: The final publication is available at Springer via the link above.
    Keyword(s) / Subject(s): web search, diversity, evaluation
    School: Birkbeck Faculties and Schools > Faculty of Science > School of Computing and Mathematical Sciences
    Research Centres and Institutes: Birkbeck Knowledge Lab, Data Analytics, Birkbeck Institute for
    Depositing User: Dell Zhang
    Date Deposited: 22 Aug 2018 16:55
    Last Modified: 09 Aug 2023 12:42
    URI: https://eprints.bbk.ac.uk/id/eprint/20719

    Statistics

    Activity Overview
    6 month trend
    359Downloads
    6 month trend
    211Hits

    Additional statistics are available via IRStats2.

    Archive Staff Only (login required)

    Edit/View Item Edit/View Item