Sneak peek: Adobe’s Project Scene Change
The Sneaks session at Adobe MAX 2023. See Project Scene Change at 00:25:40.
Adobe previewed an interesting experimental AI-powered compositing technology during the Sneaks session at Adobe MAX 2023, its user conference.
Project Scene Change integrates an actor from one video into the environment from another, without the need for rotoscoping or tracking work.
Composite an actor into a new environment without roto or tracking
Seen at 00:25:40 in the video above, Project Scene Change integrates an actor from one video into the environment from another without the need for 3D tracking to match the cameras in the two shots.
It generates a 3D point cloud of the background scene, then uses AI to align it to the actor.
Spot the difference: Adobe scientist Zhan Xu composited into a miniature set, before and after using Project Scene Change to synchronize the camera motion of the two shots.
The process synchronises the camera settings and motion of the two shots – in the demo shown above, wildly different camera motions – with the actor automatically being obscured by foreground objects when they pass behind them, and casting shadows on the ground in their new environment.
It isn’t going to replace a traditional rotoscoping-and-tracking workflow for VFX or motion graphics work just yet, but it does look like it could be a quick way for video editors to composite videos for less demanding online or social media work.
So when will I be able to use this technology in Adobe software?
While Project Scene Change is still a research project, not a commercial tool, technologies shown in Sneaks sessions can make their way into Adobe products quite quickly.
One of the highlights of the 2020 session, physics-based scene layout system Physics Whiz became part of Substance 3D Stager, released less than a year later.
Other new AI-powered technologies shown at Adobe Sneaks 2023
Project Scene Change was just one of 10 experimental new technologies shown off during the Adobe MAX 2023 Sneaks, ranging from typography tools to this amazing digital dress.
Project Stardust makes it possible to edit still images just by dragging objects in them around, while Project Fast Fill is pitched as Firefly’s Generative Fill for video.
And while it’s a better fit for Lightroom than the Adobe apps we usually cover on CG Channel, do check out Project See-Through in the recording of the event.
It’s an AI tool for removing glass reflections in photos, and it does an amazingly good job at recreating fine detail, even in shots almost completely obscured by the reflections.
Read more about the projects featured in Adobe MAX 2023’s Sneaks session