AI-Powered Immersive Assistance for Interactive Task Execution in Industrial Environments

Tomislav Duricic, Nicole Weidinger, Neven El Sayed, Dominik Kowald, Eduardo Enrique Veas

Publikation: Beitrag in Buch/Bericht/KonferenzbandBeitrag in einem KonferenzbandBegutachtung

Abstract

Many industrial sectors rely on well-trained employees that are able to operate complex machinery. In this work, we demon- strate an immersive assistance system powered by Artificial Intelli- gence (AI) that supports users in performing complex tasks in in- dustrial environments. Our system leverages a Virtual Reality (VR) environment that resembles a juice mixer setup. This digital twin of a physical setup simulates complex industrial machinery used to mix preparations or liquids (e.g., similar to the pharmaceutical indus- try) and includes various containers, sensors, pumps, and flow con- trollers. This setup demonstrates our system’s capabilities in a con- trolled environment while acting as a proof-of-concept for broader industrial applications. The core components of our multimodal AI assistant are a large language model and a speech-to-text model that process a video and audio recording of an expert performing the task in a VR environment. The video and speech input extracted from the expert’s video enables it to provide step-by-step guidance to support users in executing complex tasks. This demonstration showcases the potential of our AI-powered assistant to reduce cognitive load, in- crease productivity, and enhance safety in industrial environments.
Originalspracheenglisch
Titel27TH EUROPEAN CONFERENCE ON ARTIFICIAL INTELLIGENCE
PublikationsstatusVeröffentlicht - 20 Okt. 2024

ASJC Scopus subject areas

  • Artificial intelligence
  • Information systems

Dieses zitieren