Building a Benchmark for Task Progress in Digital Assistants

  • Jonathan Liono
  • Johanne R Trippas
  • Damiano Spina
  • Mohammad S Rahaman
  • Yongli Ren
  • Flora D Salim
  • Mark Sanderson
  • Falk Scholer
  • Ryen W White
WSDM'19 Task Intelligence Workshop (TI@WSDM19), 2019

Understanding how tasks progress over time enables digital assistants to help with current activities and support future activities. Imbuing assistants with the ability to track task progress requires machine-learned models. In this paper, we describe an ongoing effort to collect signals from Cyber, Physical, and Social (CPS) activities, together with human assessments of task progression, to serve as a benchmark for training and testing such models. Collecting this data over time is inherently challenging in the daily sensing scenario. Consequently, lessons learned from our ongoing data collection are discussed to provide more insights for future research innovations in task intelligence.

author={Liono, Jonathan and Trippas, Johanne R. and Spina, Damiano and Rahaman, Mohammad S. and Ren, Yongli and Salim, Flora D. and Sanderson, Mark and Scholer, Falk and White, Ryen W.},
title={{Building a Benchmark for Task Progress in Digital Assistants}},
booktitle={{Proceedings of WSDM'19 Task Intelligence Workshop (TI@WSDM19)}},
Damiano Spina