Computational Evaluation of Model-Agnostic Explainable AI Using Local Feature Importance in Healthcare

Explainable artificial intelligence (XAI) is essential for enabling clinical users to get informed decision support from AI and comply with evidence-based medical practice. In the XAI field, effective evaluation methods are still being developed. The straightforward way is to evaluate via user feedback. However, this needs big efforts (applying on high number of users and test cases) and can still include various biases inside. A computational evaluation of explanation methods is also not easy since there is not yet a standard output of XAI models and the unsupervised learning behavior of XAI models. In this paper, we propose a computational evaluation method for XAI models which generate local feature importance as explanations. We use the output of XAI model (local feature importances) as features and the output of the prediction problem (labels) again as labels. We evaluate the method based a real-world tabular electronic health records dataset. At the end, we answer the research question: “How can we computationally evaluate XAI Models for a specific prediction model and dataset?”.