A Markov chain model for changes in users’ assessment of search results
Zhitomirsky-Geffet, M. and Bar-Ilan, J. and Levene, Mark (2016) A Markov chain model for changes in users’ assessment of search results. PLoS One , ISSN 1932-6203.
Text
15074.pdf - Author's Accepted Manuscript Restricted to Repository staff only Download (806kB) |
||
|
Text
15074A.pdf - Published Version of Record Available under License Creative Commons Attribution. Download (188kB) | Preview |
Abstract
Previous research shows that users tend to change their assessment of search results over time. This is a first study that investigates the factors and reasons for these changes, and describes a stochastic model of user behaviour that may explain these changes. In particular, we hypothesise that most of the changes are local, i.e. between results with similar or close relevance to the query, and thus belong to the same ”coarse” relevance category. According to the theory of coarse beliefs and categorical thinking, humans tend to divide the range of values under consideration into coarse categories, and are thus able to distinguish only between cross-category values but not within them. To test this hypothesis we conducted five experiments with about 120 subjects divided into 3 groups. Each student in every group was asked to rank and assign relevance scores to the same set of search results over two or three rounds, with a period of three to nine weeks between each round. The subjects of the last three-round experiment were then exposed to the differences in their judgements and were asked to explain them. We make use of a Markov chain model to measure change in users’ judgments between the different rounds. The Markov chain demonstrates that the changes converge, and that a majority of the changes are local to a neighbouring relevance category. We found that most of the subjects were satisfied with their changes, and did not perceive them as mistakes but rather as a legitimate phenomenon, since they believe that time has influenced their relevance assessment. Both our quantitative analysis and user comments support the hypothesis of the existence of coarse relevance categories resulting from categorical thinking in the context of user evaluation of search results.
Metadata
Item Type: | Article |
---|---|
Keyword(s) / Subject(s): | Relevance judgements, change over time, locality, coarse categories, Markov chain, search engines |
School: | Birkbeck Faculties and Schools > Faculty of Science > School of Computing and Mathematical Sciences |
Research Centres and Institutes: | Birkbeck Knowledge Lab, Data Analytics, Birkbeck Institute for |
Depositing User: | Administrator |
Date Deposited: | 13 May 2016 10:30 |
Last Modified: | 09 Aug 2023 12:38 |
URI: | https://eprints.bbk.ac.uk/id/eprint/15074 |
Statistics
Additional statistics are available via IRStats2.