AI-Powered Immersive Assistance for Interactive Task Execution in Industrial Environments

Many industrial sectors rely on well-trained employees that are able to operate complex machinery. In this work, we demon- strate an immersive assistance system powered by Artificial Intelli- gence (AI) that supports users in performing complex tasks in in- dustrial environments. Our system leverages a Virtual Reality (VR) environment that resembles a juice mixer setup. This digital twin of a physical setup simulates complex industrial machinery used to mix preparations or liquids (e.g., similar to the pharmaceutical indus- try) and includes various containers, sensors, pumps, and flow con- trollers. This setup demonstrates our system’s capabilities in a con- trolled environment while acting as a proof-of-concept for broader industrial applications. The core components of our multimodal AI assistant are a large language model and a speech-to-text model that process a video and audio recording of an expert performing the task in a VR environment. The video and speech input extracted from the expert’s video enables it to provide step-by-step guidance to support users in executing complex tasks. This demonstration showcases the potential of our AI-powered assistant to reduce cognitive load, in- crease productivity, and enhance safety in industrial environments.