Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor Bias
News content can sometimes be misleading and influence users’ decision making processes (e.g., voting decisions). Quantitatively assessing the truthfulness of content becomes key, but it is often challenging and thus done by experts. In this work we look at how experts and non-expert assess truthfulness of content by focusing on the effect of the adopted judgment scale and of assessors’ own bias on the judgments they perform. Our results indicate a clear effect of the assessors’ political background on their judgments where they tend to trust content which is aligned to their own belief, even if experts have marked it as false. Crowd assessors also seem to have a preference towards coarse-grained scales, as they tend to use a few extreme values rather than the full breadth of fine-grained scales.
@inproceedings{labarbera2020crowdsourcing, title={{Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor Bias}}, booktitle={Proceedings of ECIR'20}, author={{La Barbera}, David and Roitero, Kevin and Demartini, Gianluca and Mizzaro, Stefano and Spina, Damiano}, year={2020} }
Best Short Paper Award
In the news: in Italian, and not fake 🙂
- Da Udine uno studio per individuare le fake news in tempo di Covid 19. Rai News TGR Friuli-Venezia Giulia. April 21, 2020.
- Fake news, premiato un articolo al convegno europeo annuale Ecir, Il Friuli. April 21, 2020.
- Smascherare le fake news che circolano in rete: premiato articolo firmato da giovani ricercatori. Qui.Uniud. April 21, 2020.
- Smascherare le fake news durante l’emergenza del Covid 19. Udine Today. April 21, 2020.