Mega-NeRF:
Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs
CVPR 2022


Haithem Turki1
Deva Ramanan1,2
Mahadev Satyanarayanan1


Carnegie Mellon University
Argo AI





Training: Data Partitioning

Inference: View Synthesis



Abstract

We use neural radiance fields (NeRFs) to build interactive 3D environments from large-scale visual captures spanning buildings or even multiple city blocks collected primarily from drones. In contrast to single object scenes (on which NeRFs are traditionally evaluated), our scale poses multiple challenges including (1) the need to model thousands of images with varying lighting conditions, each of which capture only a small subset of the scene, (2) prohibitively large model capacities that make it infeasible to train on a single GPU, and (3) significant challenges for fast rendering that would enable interactive fly-throughs.
To address these challenges, we begin by analyzing visibility statistics for large-scale scenes, motivating a sparse network structure where parameters are specialized to different regions of the scene. We introduce a simple geometric clustering algorithm for data parallelism that partitions training images (or rather pixels) into different NeRF submodules that can be trained in parallel.
We evaluate our approach on existing datasets (Quad 6k and UrbanScene3D) as well as against our own drone footage, improving training speed by 3x and PSNR by 12%. We also evaluate recent NeRF fast renderers on top of Mega-NeRF and introduce a novel method that exploits temporal coherence. Our technique achieves a 40x speedup over conventional NeRF rendering while remaining within 0.8 db in PSNR quality, exceeding the fidelity of existing fast renderers.




Citation


@misc{turki2021meganerf,
  title={Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs},
  author={Haithem Turki and Deva Ramanan and Mahadev Satyanarayanan},
  year={2021},
  eprint={2112.10703},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}




Longer Fly-Throughs




Data


Mill 19 - Building

Mill 19 - Rubble



Acknowledgements

This work was supported by the National Science Foundation under grant number CNS-2106862, the CMU Argo AI Center for Autonomous Vehicle Research, the Living Edge Lab, and the Open Edge Computing Initiative at Carnegie Mellon University. Webpage template