Combining deep learning with signal-image encoding for multi-modal mental wellbeing classification

Woodward, K ORCID: https://orcid.org/0000-0003-3302-1345, Kanjo, E ORCID: https://orcid.org/0000-0002-1720-0661 and Tsanas, A, 2023. Combining deep learning with signal-image encoding for multi-modal mental wellbeing classification. ACM Transactions on Computing for Healthcare. ISSN 2691-1957

Preview

Text 1831628_Kanjo.pdf - Post-print Download (1MB) | Preview

Abstract

The quantification of emotional states is an important step to understanding wellbeing. Time series data from multiple modalities such as physiological and motion sensor data have proven to be integral for measuring and quantifying emotions. Monitoring emotional trajectories over long periods of time inherits some critical limitations in relation to the size of the training data. This shortcoming may hinder the development of reliable and accurate machine learning models. To address this problem, this paper proposes a framework to tackle the limitation in performing emotional state recognition: 1) encoding time series data into coloured images; 2) leveraging pre-trained object recognition models to apply a Transfer Learning (TL) approach using the images from step 1; 3) utilising a 1D Convolutional Neural Network (CNN) to perform emotion classification from physiological data; 4) concatenating the pre-trained TL model with the 1D CNN. We demonstrate that model performance when inferring real-world wellbeing rated on a 5-point Likert scale can be enhanced using our framework, resulting in up to 98.5% accuracy, outperforming a conventional CNN by 4.5%. Subject-independent models using the same approach resulted in an average of 72.3% accuracy (SD 0.038). The proposed methodology helps improve performance and overcome problems with small training datasets.

Item Type:	Journal article
Publication Title:	ACM Transactions on Computing for Healthcare
Creators:	Woodward, K., Kanjo, E. and Tsanas, A.
Publisher:	Association for Computing Machinery
Date:	3 November 2023
ISSN:	2691-1957
Identifiers:	Number Type 10.1145/3631618 DOI 1831628 Other
Divisions:	Schools > School of Science and Technology
Record created by:	Jonathan Gallacher
Date Added:	06 Nov 2023 15:16
Last Modified:	06 Nov 2023 15:16
URI:	https://irep.ntu.ac.uk/id/eprint/50284

Actions (login required)

Edit View

Statistics

Views

Downloads