A Formal Account of Effectiveness Evaluation and Ranking Fusion

  • Enrique Amigó
  • Fernando Giner
  • Stefano Mizzaro
  • Damiano Spina
Proceedings of ICTIR'18, 2018

This paper proposes a theoretical framework which models the information provided by retrieval systems in terms of Information Theory. The proposed framework allows to formalize: (i) system effectiveness as an information theoretic similarity between system outputs and human assessments, and (ii) ranking fusion as an information quantity measure. As a result, the proposed effectiveness metric improves popular metrics in terms of formal constraints. In addition, our empirical experiments suggest that it captures quality aspects from traditional metrics, while the reverse is not true. Our work also advances the understanding of theoretical foundations of the empirically known phenomenon of effectiveness increase when combining retrieval system outputs in an unsupervised manner.

 title={A Formal Account of Effectiveness Evaluation and Ranking Fusion},
 author={Amig{\'o}, Enrique and Giner, Fernando and Mizzaro, Stefano and Spina, Damiano},
 booktitle={Proceedings of {ICTIR'18}},
 pages = {123--130},
 doi = {10.1145/3234944.3234958},
Damiano Spina