BIROn - Birkbeck Institutional Research Online

    Testing the stability of “wisdom of crowds” judgments of search results over time and their similarity with the search engine rankings

    Zhitomirsky-Geffet, M. and Bari-Ilan, J. and Levene, Mark (2016) Testing the stability of “wisdom of crowds” judgments of search results over time and their similarity with the search engine rankings. ASLIB Journal of Information Management 68 (4), pp. 407-427. ISSN 2050-3806.

    14848.pdf - Author's Accepted Manuscript

    Download (2MB) | Preview


    PURPOSE: One of the under-explored aspects in the process of user information seeking behaviour is influence of time on relevance evaluation. It has been shown in previous studies that individual users might change their assessment of search results over time. It is also known that aggregated judgments of multiple individual users can lead to correct and reliable decisions; this phenomenon is known as the “wisdom of crowds”. The aim of this study is to examine whether aggregated judgments will be more stable and thus more reliable over time than individual user judgments. DESIGN/METHODS: In this study two simple measures are proposed to calculate the aggregated judgments of search results and compare their reliability and stability to individual user judgments. In addition, the aggregated “wisdom of crowds” judgments were used as a means to compare the differences between human assessments of search results and search engine’s rankings. A large-scale user study was conducted with 87 participants who evaluated two different queries and four diverse result sets twice, with an interval of two months. Two types of judgments were considered in this study: 1) relevance on a 4-point scale, and 2) ranking on a 10-point scale without ties. FINDINGS: It was found that aggregated judgments are much more stable than individual user judgments, yet they are quite different from search engine rankings. Practical implications: The proposed “wisdom of crowds” based approach provides a reliable reference point for the evaluation of search engines. This is also important for exploring the need of personalization and adapting search engine’s ranking over time to changes in users preferences. ORIGINALITY/VALUE: This is a first study that applies the notion of “wisdom of crowds” to examine the under-explored phenomenon in the literature of “change in time” in user evaluation of relevance.


    Item Type: Article
    Keyword(s) / Subject(s): ranking, relevance judgment, wisdom of crowds, change in time, user evaluation of search results, change coefficient
    School: Birkbeck Faculties and Schools > Faculty of Science > School of Computing and Mathematical Sciences
    Research Centres and Institutes: Birkbeck Knowledge Lab, Data Analytics, Birkbeck Institute for
    Depositing User: Administrator
    Date Deposited: 05 Apr 2016 15:27
    Last Modified: 09 Aug 2023 12:37


    Activity Overview
    6 month trend
    6 month trend

    Additional statistics are available via IRStats2.

    Archive Staff Only (login required)

    Edit/View Item Edit/View Item