Evaluating XAI techniques under class imbalance using CPRD data

Rai, T ORCID: https://orcid.org/0000-0003-2299-0271, He, J ORCID: https://orcid.org/0000-0002-5616-4691, Kaur, J, Shen, Y, Mahmud, M, Brown, DJ ORCID: https://orcid.org/0000-0002-1677-7485, O'Dowd, E and Baldwin, D, 2025. Evaluating XAI techniques under class imbalance using CPRD data. Frontiers in Artificial Intelligence, 8: 1682919. ISSN 2624-8212

Preview

Text 2555346_He.pdf - Published version Download (3MB) | Preview

Abstract

Introduction: The need for eXplainable Artificial Intelligence (XAI) in healthcare is more critical than ever, especially as regulatory frameworks such as the European Union Artificial Intelligence (EU AI) Act mandate transparency in clinical decision support systems. Post hoc XAI techniques such as Local Interpretable Model-Agnostic Explanations (LIME), SHapley Additive exPlanations (SHAP) and Partial Dependence Plots (PDPs) are widely used to interpret Machine Learning (ML) models for disease risk prediction, particularly in tabular Electronic Health Record (EHR) data. However, their reliability under real-world scenarios is not fully understood. Class imbalance is a common challenge in many real-world datasets, but it is rarely accounted for when evaluating the reliability and consistency of XAI techniques.

Methods: In this study, we design a comparative evaluation framework to assess the impact of class imbalance on the consistency of model explanations generated by LIME, SHAP, and PDPs. Using UK primary care data from the Clinical Practice Research Datalink (CPRD), we train three ML models: XGBoost (XGB), Random Forest (RF), and Multi-layer Perceptron (MLP), to predict lung cancer risk and evaluate how interpretability is affected under class imbalance when compared against a balanced dataset. To our knowledge, this is the first study to evaluate explanation consistency under class imbalance across multiple models and interpretation methods using real-world clinical data.

Results: Our main finding is that class imbalance in the training data can significantly affect the reliability and consistency of LIME and SHAP explanations when evaluated against models trained on balanced data. To explain these empirical findings, we also present a theoretical analysis of LIME and SHAP to understand why explanations change under different class distributions. It is also found that PDPs exhibit noticeable variation between models trained on imbalanced and balanced datasets with respect to clinically relevant features for predicting lung cancer risk.

Discussion: These findings highlight a critical vulnerability in current XAI techniques, i.e., their interpretability are significantly affected under skewed class distributions, which is common in medical data and emphasises the importance of consistent model explanations for trustworthy ML deployment in healthcare.

Item Type:	Journal article
Publication Title:	Frontiers in Artificial Intelligence
Creators:	Rai, T., He, J., Kaur, J., Shen, Y., Mahmud, M., Brown, D.J., O'Dowd, E. and Baldwin, D.
Publisher:	Frontiers Media SA
Date:	2025
Volume:	8
ISSN:	2624-8212
Identifiers:	Number Type 10.3389/frai.2025.1682919 DOI 2555346 Other
Rights:	© 2025 Rai, He, Kaur, Shen, Mahmud, Brown, O'Dowd and Baldwin. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) (http://creativecommons.org/licenses/by/4.0/). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Divisions:	Schools > School of Science and Technology
Record created by:	Laura Borcherds
Date Added:	13 Jan 2026 09:12
Last Modified:	13 Jan 2026 09:12
URI:	https://irep.ntu.ac.uk/id/eprint/55019

Actions (login required)

Edit View

Statistics

Views

Downloads