Levene, Mark and Bar-Ilan, J. and Zhitomirsky-Geffet, M. (2018) Categorical relevance judgment. Journal of the Association for Information Science and Technology 69 (9), pp. 1084-1094. ISSN 2330-1643.
|
Text
21342.pdf - Author's Accepted Manuscript Download (259kB) | Preview |
Abstract
In this study we aim to explore users' behaviour when assessing search results relevance based on the hypothesis of categorical thinking. In order to investigate how users categorise search engine results, we perform several experiments where users are asked to group a list of 20 search results into a number of categories, while attaching a relevance judgment to each formed category. Moreover, to determine how users change their minds over time, each experiment was repeated three times under the same conditions, with a gap of one month between rounds. The results show that on average users form 4-5 categories. Within each round the size of a category decreases with the relevance of a category. To measure the agreement between the search engine’s ranking and the users’ relevance judgments, we defined two novel similarity measures, the average concordance and the MinMax swap ratio. Similarity is shown to be the highest for the third round as the users' opinion stabilises. Qualitative analysis uncovered some interesting points, in particular, that users tended to categorise results by type and reliability of their source, and particularly, found commercial sites less trustworthy, and attached high relevance to Wikipedia when their prior domain knowledge was limited.
Metadata
| Item Type: | Article |
|---|---|
| Additional Information: | This is the peer reviewed version of the article, which has been published in final form at the link above. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Self-Archiving. |
| School: | Birkbeck Faculties and Schools > Faculty of Science > School of Computing and Mathematical Sciences |
| Research Centres and Institutes: | Data Analytics, Birkbeck Institute for |
| Depositing User: | Administrator |
| Date Deposited: | 21 Feb 2018 11:17 |
| Last Modified: | 09 Aug 2023 12:43 |
| URI: | https://eprints.bbk.ac.uk/id/eprint/21342 |
Statistics
Additional statistics are available via IRStats2.
Archive Staff Only (login required)
