BIROn - Birkbeck Institutional Research Online

    Ranked-listed or categorized results in IR: 2 is better than 1

    Zhu, Zheng and Cox, I. and Levene, Mark (2008) Ranked-listed or categorized results in IR: 2 is better than 1. In: Kapetanios, E. and Sugumaran, V. and Spiliopoulou, M. (eds.) Natural Language and Information Systems. Berlin, Germany: Springer Verlag, pp. 111-123. ISBN 9783540698586.

    Full text not available from this repository.


    In this paper we examine the performance of both ranked-listed and categorized results in the context of known-item search (target testing). Performance of known-item search is easy to quantify based on the number of examined documents and class descriptions. Results are reported on a subset of the Open Directory classification hierarchy, which enable us to control the error rate and investigate how performance degrades with error. Three types of simulated user model are identified together with the two operating scenarios of correct and incorrect classification. Extensive empirical testing reveals that in the ideal scenario, i.e. perfect classification by both human and machine, a category-based system significantly outperforms a ranked list for all but the best queries, i.e. queries for which the target document was initially retrieved in the top-5. When either human or machine error occurs, and the user performs a search strategy that is exclusively category based, then performance is much worse than for a ranked list. However, most interestingly, if the user follows a hybrid strategy of first looking in the expected category and then reverting to a ranked list if the target is absent, then performance can remain significantly better than for a ranked list, even with misclassification rates as high as 30%. We also observe that this hybrid strategy results in performance degradations that degrade gracefully with error rate.


    Item Type: Book Section
    School: Birkbeck Faculties and Schools > Faculty of Science > School of Computing and Mathematical Sciences
    Research Centres and Institutes: Birkbeck Knowledge Lab
    Depositing User: Sarah Hall
    Date Deposited: 31 May 2013 09:50
    Last Modified: 09 Aug 2023 12:33


    Activity Overview
    6 month trend
    6 month trend

    Additional statistics are available via IRStats2.

    Archive Staff Only (login required)

    Edit/View Item Edit/View Item