BIROn - Birkbeck Institutional Research Online

    A Markov chain model for changes in users’ assessment of search results

    Zhitomirsky-Geffet, M. and Bar-Ilan, J. and Levene, Mark (2016) A Markov chain model for changes in users’ assessment of search results. PLoS One , ISSN 1932-6203.

    [img] Text
    15074.pdf - Author's Accepted Manuscript
    Restricted to Repository staff only

    Download (806kB)
    [img]
    Preview
    Text
    15074A.pdf - Published Version of Record
    Available under License Creative Commons Attribution.

    Download (188kB) | Preview

    Abstract

    Previous research shows that users tend to change their assessment of search results over time. This is a first study that investigates the factors and reasons for these changes, and describes a stochastic model of user behaviour that may explain these changes. In particular, we hypothesise that most of the changes are local, i.e. between results with similar or close relevance to the query, and thus belong to the same ”coarse” relevance category. According to the theory of coarse beliefs and categorical thinking, humans tend to divide the range of values under consideration into coarse categories, and are thus able to distinguish only between cross-category values but not within them. To test this hypothesis we conducted five experiments with about 120 subjects divided into 3 groups. Each student in every group was asked to rank and assign relevance scores to the same set of search results over two or three rounds, with a period of three to nine weeks between each round. The subjects of the last three-round experiment were then exposed to the differences in their judgements and were asked to explain them. We make use of a Markov chain model to measure change in users’ judgments between the different rounds. The Markov chain demonstrates that the changes converge, and that a majority of the changes are local to a neighbouring relevance category. We found that most of the subjects were satisfied with their changes, and did not perceive them as mistakes but rather as a legitimate phenomenon, since they believe that time has influenced their relevance assessment. Both our quantitative analysis and user comments support the hypothesis of the existence of coarse relevance categories resulting from categorical thinking in the context of user evaluation of search results.

    Metadata

    Item Type: Article
    Keyword(s) / Subject(s): Relevance judgements, change over time, locality, coarse categories, Markov chain, search engines
    School: Birkbeck Faculties and Schools > Faculty of Science > School of Computing and Mathematical Sciences
    Research Centres and Institutes: Birkbeck Knowledge Lab, Data Analytics, Birkbeck Institute for
    Depositing User: Administrator
    Date Deposited: 13 May 2016 10:30
    Last Modified: 09 Aug 2023 12:38
    URI: https://eprints.bbk.ac.uk/id/eprint/15074

    Statistics

    Activity Overview
    6 month trend
    319Downloads
    6 month trend
    253Hits

    Additional statistics are available via IRStats2.

    Archive Staff Only (login required)

    Edit/View Item Edit/View Item