Nvidia releases source code for its Neuralangelo AI model
Originally posted on 1 June 2023. Scroll down for news of the release of the source code.
Nvidia has posted a demo video for Neuralangelo, a new AI model for reconstructing 3D models of real-world objects or environments from video footage.
The framework, which uses the same underlying technology as Nvidia’s Instant NeRF, reconstructs scenes with a “complexity and fidelity [that] far surpass prior methods”.
The research could pave the way for new tools that reconstruct real-world objects for game development, visual effects or visualisation work more accurately than traditional photogrammetry.
So what is Neuralangelo?
Neuralangelo is the result of Nvidia’s latest work in neural surface reconstruction, an AI-based alternative to the multi-view stereo (MVS) approach used in traditional photogrammetry tools.
The new methods train a neural network to generate and progressively optimise a volumetric representation of a scene from a set of source images.
Although neural surface reconstruction avoids some of the drawbacks of MVS – like its difficulty in resolving regions of an object with strong colour variation or large areas of homogenous colour – current methods struggle to recover detail from real-world scenes.
How does Neuralangelo differ from existing neural surface reconstruction methods?
Neuralangelo itself uses Instant Neural Graphic primitives (Instant NGP), the same neural representation of the underlying 3D scene used in its downloadable Instant NeRF toolset.
The representation is refined using a new coarse-to-fine optimisation strategy that Nvidia’s blog post compares to a sculptor chipping away at a stone block to create a statue.
The metaphor is picked up in the name of the AI model, and in one of the objects reconstructed for its demo video: Michaelangelo’s David.
How is Neuralangelo better than existing techniques?
Nvidia says that Neuralangelo reconstructs surface detail more accurately than existing approaches: a claim supported by the mathematical analysis in its research paper Neuralangelo: High-Fidelity Neural Surface Reconstruction, due to be presented at the CVPR 2023 conference this month.
You can gauge the results visually from the footage in the video, and the close-up images in the paper.
As well as the statue of David, Nvidia’s test objects include a flatbed truck, and its own Santa Clara campus, reconstructed by Neuralangelo from drone footage.
Updated 15 August 2023: Licence and system requirements
Nvidia has now released the source code for Neuralangelo. The GitHub repository includes prebuilt Docker images, so it should be possible to run them on Windows, Linux or macOS.
The code is released under a custom Nvidia Source Code Licence. It’s a modified Apache 2.0 licence that limits use to the development of non-commercial applications, and only for Nvidia GPUs.
Read more about new AI scene reconstruction model Neuralangelo on Nvidia’s blog
Get the source code for Neuralangelo from Nvidia Research’s GitHub repository