PMH6 deriving clinical outcome measures from expression and speech: results of a proof of concept study

Siena, F., Vernon, M., Watts, P., Crundall, D., Byrom, B., Mowlem, F. and Breedon, P. ORCID: 0000-0002-1006-0942, 2020. PMH6 deriving clinical outcome measures from expression and speech: results of a proof of concept study. Value in Health, 23 (Supp 1), S200-S201. ISSN 1098-3015

Full text not available from this repository.

Abstract

OBJECTIVES: We have developed a mobile application and cloud analytics software solution to extract possible health outcome measures based on expression, voice acoustics and speech sentiment analysis from video diary footage collected using an Android app. This study aimed to assess the validity of expression detection.

METHODS: Healthy volunteers were requested to view 21 validated images from the International Affective Picture System (IAPS) database through a mobile app which simultaneously captured video footage of their face using the selfie camera. Images were selected to generate the following emotional responses: anger, disgust, sadness, contempt, fear, surprise and happiness. Expression analysis was performed using Microsoft Azure Cognitive services and the association with IAPS data was assessed for each image.

RESULTS: Forty participants (ages 21-57 years; Sex: M (20), F (19), Not stated (1)) provided informed consent and participated in this proof-of-concept study. Emotion was summarised into valence (unpleasant to pleasant) and arousal (calm to exciting) values. Both valence and arousal scores estimated by the average emotion estimated by the analysis of video footage were adequate predictors of the IAPS image scores (p < 0.001 and p=0.04 respectively), based on linear mixed-effects multi-level modelling using a chi-squared test to assess the significance of video analysis estimates of emotional response.

CONCLUSIONS: This proof-of-concept study provides early encouraging findings that facial expression derived from video footage may provide appropriate measures of expression. In combination with voice acoustical measures and speech sentiment analysis, this may lead to novel measures of health status in patients using a video diary in indications including depression, schizophrenia, autism spectrum disorder and PTSD amongst other conditions.

Item Type: Journal article
Publication Title: Value in Health
Creators: Siena, F., Vernon, M., Watts, P., Crundall, D., Byrom, B., Mowlem, F. and Breedon, P.
Publisher: Elsevier BV
Date: 1 May 2020
Volume: 23
Number: Supp 1
ISSN: 1098-3015
Identifiers:
NumberType
10.1016/j.jval.2020.04.627DOI
S1098301520308159Publisher Item Identifier
1329487Other
Divisions: Schools > School of Architecture, Design and the Built Environment
Record created by: Jill Tomkinson
Date Added: 30 Jun 2020 13:35
Last Modified: 30 Jun 2020 13:35
URI: http://irep.ntu.ac.uk/id/eprint/40142

Actions (login required)

Edit View Edit View

Views

Views per month over past year

Downloads

Downloads per month over past year