BIROn - Birkbeck Institutional Research Online

    Categorical relevance judgment

    Levene, Mark and Bar-Ilan, J. and Zhitomirsky-Geffet, M. (2018) Categorical relevance judgment. Journal of the Association for Information Science and Technology 69 (9), pp. 1084-1094. ISSN 2330-1643.

    21342.pdf - Author's Accepted Manuscript

    Download (259kB) | Preview


    In this study we aim to explore users' behaviour when assessing search results relevance based on the hypothesis of categorical thinking. In order to investigate how users categorise search engine results, we perform several experiments where users are asked to group a list of 20 search results into a number of categories, while attaching a relevance judgment to each formed category. Moreover, to determine how users change their minds over time, each experiment was repeated three times under the same conditions, with a gap of one month between rounds. The results show that on average users form 4-5 categories. Within each round the size of a category decreases with the relevance of a category. To measure the agreement between the search engine’s ranking and the users’ relevance judgments, we defined two novel similarity measures, the average concordance and the MinMax swap ratio. Similarity is shown to be the highest for the third round as the users' opinion stabilises. Qualitative analysis uncovered some interesting points, in particular, that users tended to categorise results by type and reliability of their source, and particularly, found commercial sites less trustworthy, and attached high relevance to Wikipedia when their prior domain knowledge was limited.


    Item Type: Article
    Additional Information: This is the peer reviewed version of the article, which has been published in final form at the link above. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Self-Archiving.
    School: School of Business, Economics & Informatics > Computer Science and Information Systems
    Research Centres and Institutes: Data Analytics, Birkbeck Institute for
    Depositing User: Administrator
    Date Deposited: 21 Feb 2018 11:17
    Last Modified: 12 Apr 2022 18:09


    Activity Overview
    6 month trend
    6 month trend

    Additional statistics are available via IRStats2.

    Archive Staff Only (login required)

    Edit/View Item Edit/View Item