Towards a Basic Principle for Ranking Effectiveness Prediction without Human Assessments: A Preliminary Study

Abstract

We present in this paper a preliminary study about the Observational Information Linearity (OIL) assumption as a basic principle that explains the accuracy of pseudo relevance assessments in the IR literature. The proposed model predicts the effectiveness drop curve along positions in a single ranking, the relative performance of two rankings, and converges into the traditional pseudo assessment method when considering multiple rankings and statistical independence between them.

Publication
GLARE'18 CIKM Workshop on 1st International Workshop on GeneraLization in InformAtion REtrieval