Combining deep learning with signal-image encoding for multi-modal mental wellbeing classification

Tools

Woodward, K. ORCID: 0000-0003-3302-1345, Kanjo, E. ORCID: 0000-0002-1720-0661 and Tsanas, A., 2023. Combining deep learning with signal-image encoding for multi-modal mental wellbeing classification. ACM Transactions on Computing for Healthcare. ISSN 2691-1957

Preview

Text
1831628_Kanjo.pdf - Post-print
Download (1MB) | Preview

Official URL: https://doi.org/10.1145/3631618

Abstract

The quantification of emotional states is an important step to understanding wellbeing. Time series data from multiple modalities such as physiological and motion sensor data have proven to be integral for measuring and quantifying emotions. Monitoring emotional trajectories over long periods of time inherits some critical limitations in relation to the size of the training data. This shortcoming may hinder the development of reliable and accurate machine learning models. To address this problem, this paper proposes a framework to tackle the limitation in performing emotional state recognition: 1) encoding time series data into coloured images; 2) leveraging pre-trained object recognition models to apply a Transfer Learning (TL) approach using the images from step 1; 3) utilising a 1D Convolutional Neural Network (CNN) to perform emotion classification from physiological data; 4) concatenating the pre-trained TL model with the 1D CNN. We demonstrate that model performance when inferring real-world wellbeing rated on a 5-point Likert scale can be enhanced using our framework, resulting in up to 98.5% accuracy, outperforming a conventional CNN by 4.5%. Subject-independent models using the same approach resulted in an average of 72.3% accuracy (SD 0.038). The proposed methodology helps improve performance and overcome problems with small training datasets.

Item Type:

Journal article

Publication Title:

ACM Transactions on Computing for Healthcare

Creators:

Woodward, K., Kanjo, E. and Tsanas, A.

Publisher:

Association for Computing Machinery

Date:

3 November 2023

ISSN:

2691-1957