FaultLines - Evaluating the Efficacy of Open-Source Large Language Models for Fault Detection in Cyber-Physical Systems

Cyber-physical systems are integral to the infrastructure of global communication and transportation networks, which makes it crucial to detect faults, prevent cyber attacks, and ensure operational safety. Although machine learning techniques, including large language models (LLMs), have been explored for fault detection, the efficacy of open-source LLMs remains underexplored. In this work, we assess the capabilities of eight open-source LLMs in identifying faults in cyber-physical systems using a simulation dataset from monitoring an electrified vehicle's battery management system. By applying pretrained LLMs without fine-tuning and incorporating retrieval augmented generation (RAG) techniques alongside textual encoding methods, our study aims to explore the potential of open LLMs in fault detection. Our results show that open LLMs can effectively identify faults, with Mistral out-performing alternative models such as Mixtral, codellama, and Gemma in precision, recall, and Fl-score metrics. Furthermore, our results highlight the importance of textual encoding strategies in enhancing the fault detection capabilities of LLMs, which possess a degree of explanatory power with respect to the detected anomalies. This work demonstrates the feasibility of using open LLMs for fault detection in cyber-physical systems and opens avenues for future research to enhance fault detection and fault localization.