Zhu, Zheng and Cox, I. and Levene, Mark (2008) Ranked-listed or categorized results in IR: 2 is better than 1. In: Kapetanios, E. and Sugumaran, V. and Spiliopoulou, M. (eds.) Natural Language and Information Systems. Berlin, Germany: Springer Verlag, pp. 111-123. ISBN 9783540698586.
Abstract
In this paper we examine the performance of both ranked-listed and categorized results in the context of known-item search (target testing). Performance of known-item search is easy to quantify based on the number of examined documents and class descriptions. Results are reported on a subset of the Open Directory classification hierarchy, which enable us to control the error rate and investigate how performance degrades with error. Three types of simulated user model are identified together with the two operating scenarios of correct and incorrect classification. Extensive empirical testing reveals that in the ideal scenario, i.e. perfect classification by both human and machine, a category-based system significantly outperforms a ranked list for all but the best queries, i.e. queries for which the target document was initially retrieved in the top-5. When either human or machine error occurs, and the user performs a search strategy that is exclusively category based, then performance is much worse than for a ranked list. However, most interestingly, if the user follows a hybrid strategy of first looking in the expected category and then reverting to a ranked list if the target is absent, then performance can remain significantly better than for a ranked list, even with misclassification rates as high as 30%. We also observe that this hybrid strategy results in performance degradations that degrade gracefully with error rate.
Metadata
| Item Type: | Book Section |
|---|---|
| School: | Birkbeck Faculties and Schools > Faculty of Science > School of Computing and Mathematical Sciences |
| Research Centres and Institutes: | Birkbeck Knowledge Lab |
| Depositing User: | Sarah Hall |
| Date Deposited: | 31 May 2013 09:50 |
| Last Modified: | 09 Aug 2023 12:33 |
| URI: | https://eprints.bbk.ac.uk/id/eprint/7164 |
Statistics
Additional statistics are available via IRStats2.
Archive Staff Only (login required)
