A Time Series Classification Pipeline for Detecting Interaction Ruptures in HRI Based on User Reactions

TY - CHAP

T1 - A Time Series Classification Pipeline for Detecting Interaction Ruptures in HRI Based on User Reactions

AU - Wachowiak, Lennart

AU - Tisnikar, Peter

AU - Coles, Andrew

AU - Canal, Gerard

AU - Celiktutan, Oya

N1 - Publisher Copyright: © 2024 Copyright held by the owner/author(s).

PY - 2024/11/4

Y1 - 2024/11/4

N2 - To be able to react to interaction ruptures such as errors, a robot needs a way of realizing such a rupture occurred. We test whether it is possible to detect interaction ruptures from the user’s anonymized speech, posture, and facial features. We showcase how to approach this task, presenting a time series classifcation pipeline that works well with various machine learning models. A sliding window is applied to the data and the continuously updated predictions make it suitable for detecting ruptures in real-time. Our best model, an ensemble of MiniRocket classifers, is the winning approach to the ICMI ERR@HRI challenge. A feature importance analysis shows that the model heavily relies on speaker diarization data that indicates who spoke when. Posture data, on the other hand, impedes performance. Our code is available online.

AB - To be able to react to interaction ruptures such as errors, a robot needs a way of realizing such a rupture occurred. We test whether it is possible to detect interaction ruptures from the user’s anonymized speech, posture, and facial features. We showcase how to approach this task, presenting a time series classifcation pipeline that works well with various machine learning models. A sliding window is applied to the data and the continuously updated predictions make it suitable for detecting ruptures in real-time. Our best model, an ensemble of MiniRocket classifers, is the winning approach to the ICMI ERR@HRI challenge. A feature importance analysis shows that the model heavily relies on speaker diarization data that indicates who spoke when. Posture data, on the other hand, impedes performance. Our code is available online.

KW - hri

KW - robotics

KW - machine learning

KW - time series

KW - multimodal

UR - http://www.scopus.com/inward/record.url?scp=85212589582&partnerID=8YFLogxK

U2 - 10.1145/3678957.3688386

DO - 10.1145/3678957.3688386

M3 - Conference paper

T3 - ACM International Conference Proceeding Series

SP - 657

EP - 665

BT - ICMI 2024 - Proceedings of the 26th International Conference on Multimodal Interaction

PB - ACM

ER -