Retrieval augmented large language model chatbots in higher education: a study on university open days

Shanthakumar, AK, Fassihi-Tash, F, Lotfi, A ORCID: https://orcid.org/0000-0002-5139-6565 and Bird, JJ ORCID: https://orcid.org/0000-0002-9858-1231, 2025. Retrieval augmented large language model chatbots in higher education: a study on university open days. In: Zheng, H, Glass, D, Mulvenna, M, Liu, J and Wang, H, eds., Advances in computational intelligence systems: contributions presented at the 23rd UK Workshop on Computational Intelligence (UKCI 2024), September 2-4, 2024, Ulster University, Belfast, UK. Advances in intelligent systems and computing (1462). Cham: Springer, pp. 32-44. ISBN 9783031788567

Text 2338719_Bird.pdf - Post-print Full-text access embargoed until 8 January 2026. Download (391kB)

Abstract

This study explores the implementation of Retrieval Augmented Large Language Models (LLMs) toward enhancing prospective student engagement during University open days. This work proposes the use of local, 4-bit quantised LLMs such as Microsoft Phi3, Meta’s LLaMa3, and Mistral AI’s Mistral to facilitate interactive dialogue about the Department of Computer Science at Nottingham Trent University. The proposed approaches are validated through the use of synthetic data generation via the RAGAS framework with additional expert human-in-the-loop oversight. We argue that the current state of the art which often involves ChatGPT as a sole validator is problematic, and we propose the use of an ensemble of multiple local validators that operate when a quorum is present to increase robustness. The results indicate that, while the chatbots are successful in providing the correct information, refining data relevance remains an open issue. Mistral demonstrated the highest performance in terms of information accuracy and coherence of responses, however, it was also the slowest at generating responses.

Item Type:	Chapter in book
Description:	Paper presented at the 23rd UK Workshop on Computational Intelligence (UKCI 2024), Ulster University, Belfast, 2-4 September 2024.
Creators:	Shanthakumar, A.K., Fassihi-Tash, F., Lotfi, A. and Bird, J.J.
Publisher:	Springer
Place of Publication:	Cham
Date:	2025
Number:	1462
ISBN:	9783031788567
Identifiers:	Number Type 10.1007/978-3-031-78857-4_3 DOI 2338719 Other
Divisions:	Schools > School of Science and Technology
Record created by:	Melissa Cornwell
Date Added:	13 Jan 2025 10:47
Last Modified:	13 Jan 2025 10:47
URI:	https://irep.ntu.ac.uk/id/eprint/52842

Actions (login required)

Edit View

Statistics

Views

Downloads