Zhitomirsky-Geffet, M. and Bar-Ilan, J. and Levene, Mark (2016) How and why do users change their assessment of search results over time? Proceedings of the Association for Information Science and Technology 52 (1), pp. 1-4. ISSN 2373-9231.
Abstract
In this study we investigate whether and why users change their preferences when assessing search engine results over time. We conducted a study with 35 subjects who were asked to rank and assign relevance scores to the same set of search results for three times, with a few weeks period between each round. The subjects were then exposed to the differences in their judgements and were asked to explain them. A new coefficient to measure change was introduced to assess the results of the experiment. We found that all the subjects judge the vast majority of the results differently in every round. However, there was less change in relevance judgements than in rankings. Most of the subjects were satisfied with their changes, and did not perceive them as mistakes but rather as a legitimate phenomenon, since they believe that time influences the relevance assessment. Our analysis reveals that the main factors that caused these changes were due to categorical thinking, influence of the learnt information, and environmental and emotional changes.
Metadata
| Item Type: | Article |
|---|---|
| Keyword(s) / Subject(s): | Ranking, relevance judgements, search engines, change over time, locality, categorical thinking |
| School: | Birkbeck Faculties and Schools > Faculty of Science > School of Computing and Mathematical Sciences |
| Research Centres and Institutes: | Data Analytics, Birkbeck Institute for |
| Depositing User: | Administrator |
| Date Deposited: | 15 Dec 2016 10:22 |
| Last Modified: | 09 Aug 2023 12:41 |
| URI: | https://eprints.bbk.ac.uk/id/eprint/17641 |
Statistics
Additional statistics are available via IRStats2.
Archive Staff Only (login required)
