Instant Mixed Reality Lighting from Casual Scanning

Thomas Richter-Trummer, Denis Kalkofen, Jinwoo Park, Dieter Schmalstieg

Research output: Chapter in Book/Report/Conference proceedingConference paperpeer-review


We present a method for recovering both incident lighting and surface materials from casually scanned geometry. By casual, we mean a rapid and potentially noisy scanning procedure of unmodified and uninstrumented scenes with a commodity RGB-D sensor. In other words, unlike reconstruction procedures which require careful preparations in a laboratory environment, our method works with input that can be obtained by consumer users. To ensure a robust procedure, we segment the reconstructed geometry into surfaces with homogeneous material properties and compute the radiance transfer on these segments. With this input, we solve the inverse rendering problem of factorization into lighting and material properties using an iterative optimization in spherical harmonics form. This allows us to account for self-shadowing and recover specular properties. The resulting data can be used to generate a wide range of mixed reality applications, including the rendering of synthetic objects with matching lighting into a given scene, but also re-rendering the scene (or a part of it) with new lighting. We show the robustness of our approach with real and synthetic examples under a variety of lighting conditions and compare them with ground truth data.
Original languageEnglish
Title of host publicationProc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR'16)
Place of PublicationMerida, Mexico
PublisherIEEE Computer Society
ISBN (Electronic)978-1-5090-3641-7
Publication statusPublished - 1 Sept 2016
Event15th IEEE International Symposium on Mixed and Augmented Reality: ISMAR 2016 - Merida, Mexico
Duration: 19 Sept 201623 Sept 2016


Conference15th IEEE International Symposium on Mixed and Augmented Reality

Cite this