The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?

  • Kevin Roitero
  • Michael Soprano
  • Beatrice Portelli
  • Damiano Spina
  • Vincenzo Della Mea
  • Giuseppe Serra
  • Stefano Mizzaro
  • Gianluca Demartini
Proceedings of CIKM'2020, 2020

Misinformation is an ever increasing problem that is difficult to solve for the research community and has a negative impact on the society at large. Very recently, in the last few months, the problem has been addressed with an approach based on crowdsourcing to scale up labeling efforts: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd of (non expert) judges is exploited. We follow the same approach and study the issue of whether crowdsourcing is an effective and reliable method to assess truthfulness of statements during a pandemic. We specifically target statements related to the COVID-19 health emergency, that is still ongoing at the time of the study and has arguably caused an increase of the amount of misinformation that is spreading online (a phenomenon for which the term "infodemic" has been used). By doing so, we are able to address (mis)information that is both related to a sensitive and personal issue like health and very recent as compared to when the judgment is done: two issues that have not been analyzed in related work. In our experiment, crowd workers are asked to assess the truthfulness of statements, as well as to provide evidence for the assessments in the form of a URL (obtained by using our customized search engine) and a text justification. By deploying a set of quality controls, we can ensure that the hundreds of assessments collected on 60 COVID-19 statements are of adequate quality. Besides showing that the crowd is able to accurately judge the truthfulness of the statements, we also report results on many different aspects, including: agreement among workers, the effect of different aggregation functions, of scales transformations, and of workers background / bias. We also analyze workers behavior, in terms of queries submitted, URLs found / selected, text justifications, and other behavioral data like clicks and mouse actions collected by means of an ad hoc logger.

@inproceedings{roitero2020covid,
title={The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?},
author={Roitero, Kevin and Soprano, Michael and Portelli, Beatrice and Spina, Damiano 
              and Della Mea, Vincenzo and Serra, Giuseppe  and  Mizzaro, Stefano 
              and Demartini, Gianluca},
booktitle={Proceedings of CIKM'20},
year={2020}
}
Damiano Spina