An Axiomatic Analysis of Diversity Evaluation Metrics: Introducing the Rank-Biased Utility Metric

  • Enrique Amigó
  • Damiano Spina
  • Jorge Carrillo-de-Albornoz
Proceedings of SIGIR'18, 2018

Many evaluation metrics have been defined to evaluate the effectiveness ad-hoc retrieval and search result diversification systems. However, it is often unclear which evaluation metric should be used to analyze the performance of retrieval systems given a specific task. Axiomatic analysis is an informative mechanism to understand the fundamentals of metrics and their suitability for particular scenarios. In this paper, we define a constraint-based axiomatic framework to study the suitability of existing metrics in search result diversification scenarios. The analysis informed the definition of Rank-Biased Utility (RBU) -- an adaptation of the well-known Rank-Biased Precision metric -- that takes into account redundancy and the user effort associated to the inspection of documents in the ranking. Our experiments over standard diversity evaluation campaigns show that the proposed metric captures quality criteria reflected by different metrics, being suitable in the absence of knowledge about particular features of the scenario under study.

@InProceedings{amigo2018axiomatic,
 author = {Amig{\'o}, Enrique and Spina, Damiano and Carrillo-de-Albornoz, Jorge},
 title = {An Axiomatic Analysis of Diversity Evaluation Metrics: 
               Introducing the Rank-Biased Utility Metric}, 
 booktitle = {The 41st International ACM SIGIR Conference on Research 
              \&\#38; Development in Information Retrieval},
 series = {SIGIR '18},
 year = {2018},
 isbn = {978-1-4503-5657-2},
 location = {Ann Arbor, MI, USA},
 pages = {625--634},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/3209978.3210024},
 doi = {10.1145/3209978.3210024},
 acmid = {3210024},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {axiomatic analysis, evaluation, search result diversification}
} 

RBU is implemented as one of the diversity metrics in EvALL: http://evall.uned.es

Damiano Spina
Assign a menu in the Right Menu options.