A human intention and motion prediction framework for applications in human-centric digital twins

Asad, U, Khalid, A ORCID: https://orcid.org/0000-0001-5270-6599, Lughmani, WA, Rasheed, S and Khan, MM, 2025. A human intention and motion prediction framework for applications in human-centric digital twins. Biomimetics, 10 (10): 656. ISSN 2313-7673

Preview

Text 2506856_Khalid.pdf - Published version Download (18MB) | Preview

Abstract

In manufacturing settings where humans and machines collaborate, understanding and predicting human intention is crucial for enabling the seamless execution of tasks. This knowledge is the basis for creating an intelligent, symbiotic, and collaborative environment. However, current foundation models often fall short in directly anticipating complex tasks and producing contextually appropriate motion. This paper proposes a modular framework that investigates strategies for structuring task knowledge and engineering context-rich prompts to guide Vision–Language Models in understanding and predicting human intention in semi-structured environments. Our evaluation, conducted across three use cases of varying complexity, reveals a critical tradeoff between prediction accuracy and latency. We demonstrate that a Rolling Context Window strategy, which uses a history of frames and the previously predicted state, achieves a strong balance of performance and efficiency. This approach significantly outperforms single-image inputs and computationally expensive in-context learning methods. Furthermore, incorporating egocentric video views yields a substantial 10.7% performance increase in complex tasks. For short-term motion forecasting, we show that the accuracy of joint position estimates is enhanced by using historical pose, gaze data, and in-context examples.

Item Type:	Journal article
Publication Title:	Biomimetics
Creators:	Asad, U., Khalid, A., Lughmani, W.A., Rasheed, S. and Khan, M.M.
Publisher:	MDPI
Date:	1 October 2025
Volume:	10
Number:	10
ISSN:	2313-7673
Identifiers:	Number Type 10.3390/biomimetics10100656 DOI 2506856 Other
Rights:	© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Divisions:	Schools > School of Science and Technology
Record created by:	Laura Borcherds
Date Added:	02 Oct 2025 08:55
Last Modified:	02 Oct 2025 08:55
URI:	https://irep.ntu.ac.uk/id/eprint/54492

Actions (login required)

Edit View

Statistics

Views

Downloads