Use NeRFs in Unreal Engine with Luma AI’s new plugin
Introducing the Luma✨Unreal Engine alpha! Fully volumetric Luma NeRFs running realtime on Windows in UE 5 for incredible cinematic shots and experiences, starting today!
Try now: https://t.co/mdyGTWhhgN pic.twitter.com/Lj1Lm3F2P4— Luma AI (@LumaLabsAI) April 3, 2023
3D capture specialist Luma AI has released its new Unreal Engine plugin, for importing and rendering volumetric NeRFs inside Unreal Engine 5 in real time.
The plugin, which is free in alpha, makes it possible to display volumetric captures of real-world objects or environments generated using Luma AI’s web app inside UE5 for use in games and virtual production.
Capture real-world 3D objects and environments using Neural Radiance Fields
Luma AI is a pioneer in the field of 3D capture based on Neural Radiance Fields (NeRF): a new method for training a neural network to optimise a volumetric representation of a scene from a set of source images.
Although its creators focused on NeRF as a means of synthesising new views of a scene, the volumetric representation can be converted into a 3D mesh, making it an alterative to photogrammetry for 3D scanning.
In the case of Luma AI, captures can be generated from video captured on an iPhone using its iOS app or from video or zip files of images uploaded to its web app, both currently free to use in beta.
Users of the web app can download captures of objects in glTF, OBJ or USDZ format and captures of scenes in PLY format – and now, as Luma Field files for use with the new Unreal Engine plugin.
Import your volumetric captures into Unreal Engine 5 for real-time rendering
The plugin makes it possible to import those Luma Field files into Unreal Engine to use as environments, automatically generating a set of Blueprints to light and display the captures.
Users can choose baked lighting from the captured scene or dynamic lighting from the Unreal Engine scene.
Other Blueprints automatically crop in on the central object in the capture, and generate a sky environment.
The plugin is provided as part of two sample Unreal Engine projects, one tailored to games and real-time projects, and one to cinematics and movie projects, although it works perfectly well outside of them.
It currently doesn’t generate colliders, so to use captures in simulations, or with in-game physics, you will need to add collider objects manually, but that may change in future updates.
Price and system requirements
Luma AI’s Unreal Engine plugin is compatible with Unreal Engine 5.0+ running on Windows only. It’s currently free in alpha.
The web app should work with any standard desktop browser, and is currently free in beta.
Read more about Luma AI’s Unreal Engine plugin in the online documentation
(Includes links to download the .uproject files containing the plugin)