Towards an automatic coding of observational studies: coding neurofeedback therapies of children with autism

Abstract

The coding of observational data is commonly used to analyse and evaluate human behaviours. The technique can help researchers inform the design and impact of, for example, an Ubicomp system by studying specific behaviours of interest. There are some tools that can alleviate the burden of observational coding, like those that help to collect and organise data, but can still be error‐prone and time‐consuming. Moreover, most of these tools lack automation, requiring intense human interaction. In order to mitigate these issues, computer vision (CV) and machine learning (ML) techniques could be used to automate observational coding, but little work has focused on analysing the feasibility of such an approach, with the goal of reducing the total coding time while maintaining accuracy. In this work, we address this question by proposing an automated approach for a real‐world case study and compare it to manual coding. The study is composed of 10 videos with an average duration of 17 min each, where the goal is to determine the attention of children with autism that participate in a neurofeedback therapy session. Each video was hand‐coded by three human observers to define the ground truth and to measure the manual coding time. Results show that it is feasible to automate the coding of observational behaviours and obtain a noticeable reduction in coding time, but with a slight loss in accuracy. Moreover, we illustrate that the best solution would be a hybrid approach, using a semi‐automated system that combines human expertise and ML predictions Keywords Observational studies, Coding behaviours, Automatic coding, Computer‐Vision, Machine Learning.