BIROn - Birkbeck Institutional Research Online

    Analysis of change in users' assessment of search results over time

    Zhitomirsky-Geffet, M. and Bar-Ilan, J. and Levene, Mark (2017) Analysis of change in users' assessment of search results over time. Journal of the Association for Information Science and Technology 68 (5), pp. 1137-1148. ISSN 2330-1643.

    JASIST - preprint - Analysis of change in users assessment of search results over time.pdf - Author's Accepted Manuscript

    Download (340kB) | Preview


    We present the first systematic study of the influence of time on user judgements for rankings and relevance grades of web search engine results. The goal of this study is to evaluate the change in user assessment of search results and explore how users' judgements change. To this end, we conducted a large-scale user study with 86 participants who evaluated two different queries and four diverse result sets twice with an interval of two months. To analyse the results we investigate whether two types of patterns of user behaviour from the theory of categorical thinking hold for the case of evaluation of search results: (1) coarseness and (2) locality. To quantify these patterns we devised two new measures of change in user judgements and distinguish between local (when users swap between close ranks and relevance values) and non-local changes. Two types of judgements were considered in this study: 1) relevance on a 4-point scale, and 2) ranking on a 10-point scale without ties. We found that users tend to change their judgements of the results over time in about 50% of cases for relevance and in 85% of cases for ranking. However, the majority of these changes were local.


    Item Type: Article
    Additional Information: This is the peer reviewed version of the article, which has been published in final form at the link above. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Self-Archiving.
    School: School of Business, Economics & Informatics > Computer Science and Information Systems
    Research Centres and Institutes: Birkbeck Knowledge Lab, Data Analytics, Birkbeck Institute for
    Depositing User: Mark Levene
    Date Deposited: 27 Jan 2017 09:50
    Last Modified: 03 Jul 2020 18:53


    Activity Overview

    Additional statistics are available via IRStats2.

    Archive Staff Only (login required)

    Edit/View Item Edit/View Item