Analyzing the Internals of Neural Radiance Fields

Research output: Chapter in Book/Report/Conference proceedingConference paperpeer-review

Abstract

Modern Neural Radiance Fields (NeRFs) learn a mapping from position to volumetric density leveraging proposal network samplers. In contrast to the coarse-to-fine sampling approach with two NeRFs this offers significant potential for acceleration using lower network capacity. Given that NeRFs utilize most of their network capacity to estimate radiance they could store valuable density information in their parameters or their deep features. To investigate this proposition we take one step back and analyze large trained ReLU-MLPs used in coarse-to-fine sampling. Building on our novel activation visualization method we find that trained NeRFs Mip-NeRFs and proposal network samplers map samples with high density to local minima along a ray in activation feature space. We show how these large MLPs can be accelerated by transforming intermediate activations to a weight estimate without any modifications to the training protocol or the network architecture. With our approach we can reduce the computational requirements of trained NeRFs by up to 50% with only a slight hit in rendering quality. Extensive experimental evaluation on a variety of datasets and architectures demonstrates the effectiveness of our approach. Consequently our methodology provides valuable insight into the inner workings of NeRFs.
Original languageEnglish
Title of host publicationProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops
Pages2822-2831
Publication statusPublished - 2024
Event2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024 - Seattle, United States
Duration: 16 Jun 202422 Jun 2024

Conference

Conference2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024
Abbreviated titleCVPR 2024
Country/TerritoryUnited States
CitySeattle
Period16/06/2422/06/24

Cite this