Deep learning analysis of mobile physiological, environmental and location sensor data for emotion detection

Kanjo, E ORCID: https://orcid.org/0000-0002-1720-0661, Younis, EMG ORCID: https://orcid.org/0000-0003-2778-4231 and Ang, CS, 2019. Deep learning analysis of mobile physiological, environmental and location sensor data for emotion detection. Information Fusion, 49, pp. 46-56. ISSN 1566-2535

Preview

Text 11864_Kanjo.pdf - Post-print Download (899kB) | Preview

Abstract

The detection and monitoring of emotions are important in various applications, e.g. to enable naturalistic and personalised human-robot interaction. Emotion detection often require modelling of various data inputs from multiple modalities, including physiological signals (e.g.EEG and GSR), environmental data (e.g. audio and weather), videos (e.g. for capturing facial expressions and gestures) and more recently motion and location data. Many traditional machine learning algorithms have been utilised to capture the diversity of multimodal data at the sensors and features levels for human emotion classification. While the feature engineering processes often embedded in these algorithms are beneficial for emotion modelling, they inherit some critical limitations which may hinder the development of reliable and accurate models. In this work, we adopt a deep learning approach for emotion classification through an iterative process by adding and removing large number of sensor signals from different modalities. Our dataset was collected in a real-world study from smart-phones and wearable devices. It merges local interaction of three sensor modalities: on-body, environmental and location into global model that represents signal dynamics along with the temporal relationships of each modality. Our approach employs a series of learning algorithms including a hybrid approach using Convolutional Neural Network and Long Short-term Memory Recurrent Neural Network (CNN-LSTM) on the raw sensor data, eliminating the needs for manual feature extraction and engineering. The results show that the adoption of deep-learning approaches is effective in human emotion classification when large number of sensors input is utilised (average accuracy 95% and F-Measure=%95) and the hybrid models outperform traditional fully connected deep neural network (average accuracy 73% and F-Measure=73%). Furthermore, the hybrid models outperform previously developed Ensemble algorithms that utilise feature engineering to train the model average accuracy 83% and F-Measure=82%)

Item Type:	Journal article
Publication Title:	Information Fusion
Creators:	Kanjo, E., Younis, E.M.G. and Ang, C.S.
Publisher:	Elsevier
Date:	September 2019
Volume:	49
ISSN:	1566-2535
Identifiers:	Number Type 10.1016/j.inffus.2018.09.001 DOI S1566253518300460 Publisher Item Identifier 679617 Other
Divisions:	Schools > School of Science and Technology
Record created by:	Jonathan Gallacher
Date Added:	05 Sep 2018 14:05
Last Modified:	21 Jan 2021 09:39
URI:	https://irep.ntu.ac.uk/id/eprint/34429

Actions (login required)

Edit View

Statistics

Views

Downloads