A Time Series Classification Pipeline for Detecting Interaction Ruptures in HRI Based on User Reactions

TY - CHAP

T1 - A Time Series Classification Pipeline for Detecting Interaction Ruptures in HRI Based on User Reactions

AU - Wachowiak, Lennart

AU - Tisnikar, Peter

AU - Coles, Andrew

AU - Canal, Gerard

AU - Celiktutan, Oya

PY - 2024/8/10

Y1 - 2024/8/10

N2 - To be able to react to interaction ruptures such as errors, a robot needs a way of realizing such a rupture occurred. We test whether it is possible to detect interaction ruptures from the user's anonymized speech, posture, and facial features. We showcase how to approach this task, presenting a time series classification pipeline that works well with various machine learning models. A sliding window is applied to the data and the continuously updated predictions make it suitable for detecting ruptures in real-time. Our best model, an ensemble of MiniRocket classifiers, is the winning approach to the ICMI ERR@HRI challenge. A feature importance analysis shows that the model heavily relies on speaker diarization data that indicates who spoke when. Posture data, on the other hand, impedes performance. Our code is available online (https://github.com/lwachowiak/HRI-Error-Detection-STAI).

AB - To be able to react to interaction ruptures such as errors, a robot needs a way of realizing such a rupture occurred. We test whether it is possible to detect interaction ruptures from the user's anonymized speech, posture, and facial features. We showcase how to approach this task, presenting a time series classification pipeline that works well with various machine learning models. A sliding window is applied to the data and the continuously updated predictions make it suitable for detecting ruptures in real-time. Our best model, an ensemble of MiniRocket classifiers, is the winning approach to the ICMI ERR@HRI challenge. A feature importance analysis shows that the model heavily relies on speaker diarization data that indicates who spoke when. Posture data, on the other hand, impedes performance. Our code is available online (https://github.com/lwachowiak/HRI-Error-Detection-STAI).

KW - hri

KW - robotics

KW - machine learning

KW - time series

KW - multimodal

U2 - 10.1145/3678957.3688386

DO - 10.1145/3678957.3688386

M3 - Conference paper

BT - Proceedings of the 26th International Conference on Multimodal Interaction (ICMI ’24)

PB - ACM

ER -