Episodes of change: emotion change in semantic trajectories of multimodal sensor data

Johnson, T. ORCID: 0000-0002-1702-0943 and Kanjo, E. ORCID: 0000-0002-1720-0661, 2023. Episodes of change: emotion change in semantic trajectories of multimodal sensor data. In: Proceedings of EmotionAware 2023: Seventh International Workshop on Emotion Awareness for Pervasive Computing Beyond Traditional Approaches, Atlanta USA, 13-17 March 2023. Institute of Electrical and Electronics Engineers (IEEE).

Text 1748320_Johnson.pdf - Post-print Restricted to Repository staff only Download (1MB)

Abstract

The ubiquity of location tracking on smartphones allows us to monitor, collect, and analyse large trajectory data in real-time. Time series classification and clustering is an efficient way to analyse trajectories. A remarkable amount of research on the relationship between urban environments, health or wellbeing has been conducted including our previous work. However, in this paper we will introduce semantic trajectories that use episodes as changes in emotion to enable scientists and healthcare professionals to assess the impact of surrounding environments and physiological responses directly to individuals' wellbeing. For the first time, the study explores how a trajectory can be enriched with several semantics, specifically made up of environmental (e.g. air pollution including: PM1, PM2.5, PM10, Oxidised, Reduced, NH3 and Noise, People Count in the vicinity) and physiological responses (including: EDA, HR, HRV, Body Temperature, BVP and movement), in addition to self-report wellbeing and location data collected in real-time. The proposed method divides the multi-modal sensor data semantic trajectory into individual episodes each time a change of emotion is detected. Statistical correlation analysis techniques have been applied to unravel the relationship between emotion episodes and semantics, highlighting that Electrodermal Activity (EDA), Heart Rate (HR) and self-labelled emotion are noticeably impacted by the level of air pollution in the environment. We adopted Dynamic Time Warping Algorithm (DTW) to classify the self-reported emotion as episodes which achieved an overall F1-Score of 0.88 using a KNN classifier.

Item Type:	Chapter in book
Description:	Paper presented at EmotionAware 2023: Seventh International Workshop on Emotion Awareness for Pervasive Computing Beyond Traditional Approaches, Atlanta USA, 13-17 March 2023.
Creators:	Johnson, T. and Kanjo, E.
Publisher:	Institute of Electrical and Electronics Engineers (IEEE)
Date:	13 March 2023
Identifiers:	Number Type 1748320 Other
Rights:	© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Divisions:	Schools > School of Science and Technology
Record created by:	Linda Sullivan
Date Added:	31 May 2023 08:50
Last Modified:	31 May 2023 08:50
URI:	https://irep.ntu.ac.uk/id/eprint/49085

Actions (login required)

Edit View

Views

Downloads