In-Situ Virtual Reality Tour using a real-time 360° Video Stream

With our project, we developed a system capable of live streaming immersive video footage from a simple handheld 360◦ camera to a modern VR Headset. The chosen hardware focuses on providing a lightweight, small-footprint solution allowing us to mount the camera on an RC-Car, thus providing a first-person view of the cockpit. Additionally, the steering servo and cruise control of the RC-Car are controllable via an ESP32 listening to MQTT messages of the VR Headset’s controller input. The implemented software uses highly efficient-hardware accelerated encoding and decoding solutions featuring H.264 video compression standard for rendering. The live stream from the camera is delivered via RTSP(Real Time Streaming Protocol) to the end user’s Oculus Quest 2 VR Headset. The client decodes the video and stitches the equirectangular projected footage into a sphere that fully encloses the Viewpoint in the virtual world. Thus, via simple head movement, the user can look around in his new virtual world represented by the live video footage of the camera.