TY - JOUR
T1 - Hierarchical attention-based temporal convolutional networks for EEG-based emotion recognition
AU - Li, Chao
AU - Chen, Boyang
AU - Zhao, Ziping
AU - Cummins, Nicholas
AU - Schuller, Björn W.
N1 - Funding Information:
The work presented in this article was substantially supported by the National Natural Science Foundation of China (Grant No: 62071330, 61702370, 61902282), the National Science Fund for Distinguished Young Scholars (GrantNo: 61425017),theKeyProgramoftheNationalNaturalScience Foundation of China (Grant No: 61831022), the Key Program of the Natural Science Foundation of Tianjin(Grant No.18JCZDJC36300), the technology plan of Tianjin (Grant No: 18ZXRHSY00100). Ziping Zhao is the corresponding author ([email protected]).
Publisher Copyright:
© 2021 IEEE
Copyright:
Copyright 2021 Elsevier B.V., All rights reserved.
PY - 2021
Y1 - 2021
N2 - EEG-based emotion recognition is an effective way to infer the inner emotional state of human beings. Recently, deep learning methods, particularly long short-term memory recurrent neural networks (LSTM-RNNs), have made encouraging progress for in the field of emotion recognition. However, the LSTM-RNNs are time-consuming and have difficulty avoiding the problem of exploding/vanishing gradients when during training. In addition, EEG-based emotion recognition often suffers due to the existence of silent and emotional irrelevant frames from intra-channel. Not all channels carry the same emotional discriminative information. In order to tackle these problems, a hierarchical attention-based temporal convolutional networks (HATCN) for efficient EEG-based emotion recognition is proposed. Firstly, a spectrogram representation is generated from raw EEG signals in each channel to capture their time and frequency information. Secondly, temporal convolutional networks (TCNs) are utilised to automatically learn more robust/intrinsic long-term dynamic characters in emotion response. Next, a hierarchical attention mechanism is investigated that aggregates the emotional information at both the frame and channel level. The experimental results on the DEAP dataset show that our method achieves an average recognition accuracy of 0.716 and an F1-score of 0.642 over four emotional dimensions and outperforms other state-of-the-art methods in a user-independent scenario.
AB - EEG-based emotion recognition is an effective way to infer the inner emotional state of human beings. Recently, deep learning methods, particularly long short-term memory recurrent neural networks (LSTM-RNNs), have made encouraging progress for in the field of emotion recognition. However, the LSTM-RNNs are time-consuming and have difficulty avoiding the problem of exploding/vanishing gradients when during training. In addition, EEG-based emotion recognition often suffers due to the existence of silent and emotional irrelevant frames from intra-channel. Not all channels carry the same emotional discriminative information. In order to tackle these problems, a hierarchical attention-based temporal convolutional networks (HATCN) for efficient EEG-based emotion recognition is proposed. Firstly, a spectrogram representation is generated from raw EEG signals in each channel to capture their time and frequency information. Secondly, temporal convolutional networks (TCNs) are utilised to automatically learn more robust/intrinsic long-term dynamic characters in emotion response. Next, a hierarchical attention mechanism is investigated that aggregates the emotional information at both the frame and channel level. The experimental results on the DEAP dataset show that our method achieves an average recognition accuracy of 0.716 and an F1-score of 0.642 over four emotional dimensions and outperforms other state-of-the-art methods in a user-independent scenario.
KW - EEG signals
KW - Emotion recognition
KW - Hierarchical attention mechanism
KW - Temporal convolutional networks
UR - http://www.scopus.com/inward/record.url?scp=85113347388&partnerID=8YFLogxK
U2 - 10.1109/ICASSP39728.2021.9413635
DO - 10.1109/ICASSP39728.2021.9413635
M3 - Conference paper
AN - SCOPUS:85113347388
SN - 1520-6149
VL - 2021-June
SP - 1240
EP - 1244
JO - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
JF - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
T2 - 2021 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2021
Y2 - 6 June 2021 through 11 June 2021
ER -