Saturday, August 7th, 2021 Posted by Jim Thacker

Omniverse Audio2Face now supports Reallusion characters


Nvidia has added presets for characters generated by Reallusion’s Character Creator software to Omniverse Audio2Face, its experimental free software for generating facial animation from audio sources.

The functionality makes it possible for Reallusion users to generate lip sync animation automatically for their 3D characters from voice recordings in any language, for use in game engines like Unity and Unreal Engine.

The workflow is still early in development, with Reallusion flagging it as a preview. According to the firm, “practical solutions for production environments will come in the following months”.

Generate automatic lip-sync and facial animation for Character Creator characters from audio files
Released in free open beta earlier this year, Audio2Face is an AI-trained tool for generating facial animation for a 3D character from audio sources: either offline recordings of speech, or a live audio feed.

Along with sister app Omniverse Machinima, the software is one of a set of new games tools Nvidia is developing around Omniverse, its new USD-based real-time collaboration platform.

The latest update, Audio2Face 2021.2.6, adds presets for characters created with Character Creator, Reallusion’s software for generating 3D characters for games or real-time applications.

Users can export the head mesh from both standard Character Creator CC3 Base+ and lower-resolution Game Base characters to Audio2Face, and generate lip-sync and facial animation automatically.

The resulting animation can be exported from Audio2Face as either a Maya point cache or a USD point cache, for import into applications that support the USD file format, including Unity and Unreal Engine.

Works with dialogue in any language, but limited options for editing the animation
Unlike AccuLips, the new automated lip sync system in iClone, Reallusion’s own character animation software, Audio2Face works with speech in any language, not just English.

However, the options for editing the resulting animation are currently more limited than with iClone.

The latest release provides slider controls to adjust the strength and smoothing of the animation of a character’s upper and lower face, but doesn’t support blendshape editing.

In the comments on this YouTube video, Reallusion says that it is working on a Python script to bring the blendshapes back to iClone.

Pricing and system requirements
The workflow requires the Pipeline edition of Character Creator 3.43+, Omniverse and version 3.43+ of its Character Creator connector, Omniverse Machinima 2021.2+, and Omniverse Audio2Face 2021.2.6+.

Omniverse, Omniverse Machinima and Omniverse Audio2Face are available for Windows 10. To use all three, you will need a Nvidia GeForce RTX 3060 or Quadro RTX 4000 GPU or higher.

All of the Omniverse tools are in open beta, and are free to individual artists.

Character Creator is available for Windows 7+. The Pipeline edition costs $199.

Read more about Character Creator to Omniverse Audio2Face workflow on Reallusion’s forum