How new AI workflows helped create indie VFX short BYE-BYE

Artist and director Freddy Chávez Olmos has released an interesting technical breakdown showing how he used new AI-based tools to create the VFX of indie horror film BYE-BYE.

In order to be able to complete the two-minute short – shot in just two hours on a virtual production stage – the former Image Engine compositing supervisor relied on generative AI platform Leonardo.ai and new machine learning features in Nuke and Photoshop for rotoscoping and paint work, and to create an unnerving facial transformation effect.

We contacted Chávez Olmos to find out a little more about how the work was done, and how AI workflows could help other indie film-makers trying to create complex VFX on a limited budget.

Turning an urban legend into an unnerving VFX short
Based on an urban legend, BYE-BYE shows two women, Karla and Hannah, waiting for a train on a deserted subway platform when chilling whispers emerge from the darkness.

Although the film itself is currently screening on the festival circuit, which means that only brief clips are available online, you can get a flavor of what happens next from the breakdown.

Using AI to create effects too time-consuming for a brief live shoot
BYE-BYE came about through an initiative from the Director’s Guild of Canada, giving eight directors the chance to direct short film projects in virtual production at Vancouver’s Lumostage.

With just two hours available for the shoot, Chávez Olmos – who was previously mentored by special effects make-up legend Dick Smith – turned to new AI-based workflows to create the complex prosthetic make-up look he had in mind for one of the characters.

“I knew that achieving this on set within the two-hour window was impossible, so the idea of using AI and machine learning to streamline post-production became everything,” he says.

AI tools: Nuke’s CopyCat for roto and matte work
For the work, Boxel Studio, the VFX and animation studio where Chávez Olmos is Creative Director for Artifical Intelligence, used a number of new AI and machine learning tools.

One of the most significant was CopyCat, part of Nuke, Foundry’s industry-standard compositing software, which lets users train their own neural networks to automate VFX tasks.

Boxel used CopyCat in conjunction with real-time portrait matting model MODNet to isolate the two actors from the backgrounds in the live footage.

“CopyCat proved to be a powerful tool [and one we used] extensively used in this project,” says Chávez Olmos. “We relied on it to assist in the rotoscoping process, and to generate all the DI mattes required for the color grading session.”

Some of Chávez Olmos’s early tests for using generative AI platform Leonardo.ai and Nuke’s CopyCat system to create digital make-up concepts for VFX: a workflow used on BYE-BYE.

AI tools: Leonardo.ai for generating digital make-up looks
Boxel also used CopyCat in conjunction with Leonardo.ai to create the crucial facial transformation effect, which sees the live actor changed into a much older woman.

As well as providing a range of AI models tuned for production work, the generative AI platform lets users train customized AI models using their own source images.

“We [used Leonardo.ai to create] our own personalized model to generate looks consistent with the style I envisioned for the decrepit old lady character,” says Chávez Olmos. “By establishing consistent ‘style keyframes’ in Leonardo, we were able to use CopyCat to generate in-betweens and map them to the plate.”

According to Chávez Olmos, the use of ControlNet features for image-to-image guidance was crucial for the work. “It allowed us to stay true to the [actor’s] performance and have her movements and poses drive the generative process,” he says.

AI tools: Photoshop’s Generative Fill for background paint-out
Boxel also used Generative Fill, the AI-based object removal and background replacement tool introduced in Photoshop earlier this year for paint-out work on BYE-BYE.

Chávez Olmos describes it as “quite handy” for the project, adding: “It’s a tool I look forward to seeing evolve in the near future, especially once it’s released for the Adobe video packages.”

Virtual production and AI: opening up new possibilities for indie film-making?
Set extension work on the short was done in Unreal Engine, and while Boxel didn’t use AI for this part of the work, Chávez Olmos says that the team is now “testing some exciting AI/ML tools in Unreal” for his next directorial project with the studio.

Juan Carlos Galindo, Unreal Engine VP Supervisor on BYE-BYE, and Director of New Technologies at Boxel Studio, says that the combination of virtual production and AI-based workflows of the type used on BYE-BYE open up new ways for film-makers to work.

“Emerging technologies are transforming the landscape,” he says. “Directors now have an entirely new toolbox to enhance their visions.”

Read more about BYE-BYE on Freddy Chávez Olmos’s website

See more of Chávez Olmos’s technical tests on using AI in indie VFX productions

Have your say on this story by following CG Channel on Facebook, Instagram and X (formerly Twitter). As well as being able to comment on stories, followers of our social media accounts can see videos we don’t post on the site itself, including making-ofs for the latest VFX movies, animations, games cinematics and motion graphics projects.