Zhitomirsky-Geffet, M. and Bar-Ilan, J. and Levene, Mark (2017) Analysis of change in users' assessment of search results over time. Journal of the Association for Information Science and Technology 68 (5), pp. 1137-1148. ISSN 2330-1643.
|
Text
JASIST - preprint - Analysis of change in users assessment of search results over time.pdf - Author's Accepted Manuscript Download (340kB) | Preview |
Abstract
We present the first systematic study of the influence of time on user judgements for rankings and relevance grades of web search engine results. The goal of this study is to evaluate the change in user assessment of search results and explore how users' judgements change. To this end, we conducted a large-scale user study with 86 participants who evaluated two different queries and four diverse result sets twice with an interval of two months. To analyse the results we investigate whether two types of patterns of user behaviour from the theory of categorical thinking hold for the case of evaluation of search results: (1) coarseness and (2) locality. To quantify these patterns we devised two new measures of change in user judgements and distinguish between local (when users swap between close ranks and relevance values) and non-local changes. Two types of judgements were considered in this study: 1) relevance on a 4-point scale, and 2) ranking on a 10-point scale without ties. We found that users tend to change their judgements of the results over time in about 50% of cases for relevance and in 85% of cases for ranking. However, the majority of these changes were local.
Metadata
| Item Type: | Article |
|---|---|
| Additional Information: | This is the peer reviewed version of the article, which has been published in final form at the link above. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Self-Archiving. |
| School: | Birkbeck Faculties and Schools > Faculty of Science > School of Computing and Mathematical Sciences |
| Research Centres and Institutes: | Birkbeck Knowledge Lab, Data Analytics, Birkbeck Institute for |
| Depositing User: | Mark Levene |
| Date Deposited: | 27 Jan 2017 09:50 |
| Last Modified: | 09 Aug 2023 12:37 |
| URI: | https://eprints.bbk.ac.uk/id/eprint/14647 |
Statistics
Additional statistics are available via IRStats2.
Archive Staff Only (login required)
