BIROn - Birkbeck Institutional Research Online

    A Markov chain model for changes in users’ assessment of search results

    Zhitomirsky-Geffet, M. and Bar-Ilan, J. and Levene, Mark (2016) A Markov chain model for changes in users’ assessment of search results. PLoS One , ISSN 1932-6203.

    [img] Text
    15074.pdf - Author's Accepted Manuscript
    Restricted to Repository staff only

    Download (806kB)
    15074A.pdf - Published Version of Record
    Available under License Creative Commons Attribution.

    Download (188kB) | Preview


    Previous research shows that users tend to change their assessment of search results over time. This is a first study that investigates the factors and reasons for these changes, and describes a stochastic model of user behaviour that may explain these changes. In particular, we hypothesise that most of the changes are local, i.e. between results with similar or close relevance to the query, and thus belong to the same ”coarse” relevance category. According to the theory of coarse beliefs and categorical thinking, humans tend to divide the range of values under consideration into coarse categories, and are thus able to distinguish only between cross-category values but not within them. To test this hypothesis we conducted five experiments with about 120 subjects divided into 3 groups. Each student in every group was asked to rank and assign relevance scores to the same set of search results over two or three rounds, with a period of three to nine weeks between each round. The subjects of the last three-round experiment were then exposed to the differences in their judgements and were asked to explain them. We make use of a Markov chain model to measure change in users’ judgments between the different rounds. The Markov chain demonstrates that the changes converge, and that a majority of the changes are local to a neighbouring relevance category. We found that most of the subjects were satisfied with their changes, and did not perceive them as mistakes but rather as a legitimate phenomenon, since they believe that time has influenced their relevance assessment. Both our quantitative analysis and user comments support the hypothesis of the existence of coarse relevance categories resulting from categorical thinking in the context of user evaluation of search results.


    Item Type: Article
    Keyword(s) / Subject(s): Relevance judgements, change over time, locality, coarse categories, Markov chain, search engines
    School: Birkbeck Schools and Departments > School of Business, Economics & Informatics > Computer Science and Information Systems
    Research Centre: Birkbeck Knowledge Lab
    Depositing User: Administrator
    Date Deposited: 13 May 2016 10:30
    Last Modified: 05 Apr 2020 10:36


    Activity Overview

    Additional statistics are available via IRStats2.

    Archive Staff Only (login required)

    Edit/View Item Edit/View Item