Cubic Motion ships Persona
Siren, Epic Games’ demo of a real-time photorealistic digital human from GDC 2018. Cubic Motion has just launched Persona, a new off-the-shelf markerless facial capture system based on the tech used in the demo.
Originally posted on 1 May 2019. Scroll down for news of the latest hardware update.
Facial mocap specialist Cubic Motion has launched Persona, an intriguing new markerless facial capture system based on the technology used in many of Epic Games’ amazing recent GDC demos.
The system comprises a head-mounted infrared camera system, a wireless processing pack worn on an actor’s body, and accompanying software.
It uses machine learning techniques to create a customised facial solve for individual actors, and can stream the resulting animation data to Unreal Engine in real time, or export it to Maya offline.
The facial capture technology behind Epic Games’ amazing recent GDC demos
To many CG Channel readers, Cubic Motion will be most familiar through its work with Epic Games on Epic’s eye-catching recent GDC demos of real-time digital humans.
The companies teamed up in 2016 with facial rigging specialist – and recent Epic acquisition – 3Lateral to create a demo based on Ninja Theory’s Hellblade: Senua’s Sacrifice.
The trio reunited last year for Siren, in which Cubic Motion’s tech was used to transfer facial expressions from footage of the live actor to a 3D character at 60fps.
However, the company also has a strong track record as a facial animation service bureau, particularly for the games industry: as well as Hellblade, it worked on Call of Duty: WW2, Metro Exodus and 2018’s God of War.
An all-in-one markerless facial capture system capable of streaming data to UE4 in real time
Persona makes that technology – or a version of it, at least – available as an off-the-shelf markerless facial motion capture system.
It’s an all-in-one solution, comprising a head-mounted camera system, a dedicated hardware processing unit worn on the actor’s body, and accompanying software.
Unlike Faceware‘s rival products, the HMC is a dual-camera system, with front and side infra-red cameras both capturing at 1K resolution.
The helmet on which they are mounted comes in a range of sizes, and looks highly adjustable.
Persona’s other hardware component is the ‘Performance Pipeline’ a dedicated processing and data storage unit worn on the actor’s body.
It’s capable of tracking and solving facial expressions at 60fps and streaming the animation data wirelessly to Unreal Engine, or storing it on-board for offline use.
Both head-mounted camera and Performance Pipeline can be worn alongside a full-body motion-capture suit: Cubic Motion’s marketing materials show them in use in a Vicon optical mocap volume.
The hardware is accompanied by Performance Manager, Cubic Motion’s desktop software for launching and monitoring capture sessions, capable of capturing multiple actors simultaneously.
Further applications handle streaming of animation data to Unreal Engine, and importing it into Maya offline.
Facial solves customised for individual actors and characters
The really interesting feature of Persona is that the system is customised for individual actors.
Before a capture session, users record the actor performing a set of standard actions, then send the training footage to Cubic Motion along with the 3D character onto which animation will be transferred.
Cubic Motion’s machine-learning-based system processes the data to create a custom ‘identity’ for the actor: a personalised profile that can be loaded onto the Performance Pipeline prior to capture.
The profile tailors the tracking model used by Persona to the actor’s face, and the solver model to the capabilities of the 3D character rig, creating a custom set-up for each “human and digital character pairing”.
Updated 2 May 2019: We contacted Cubic Motion to find out how an identity is created.
The firm told us that the process is partly automated, but still requires some manual input: in the case of generating the tracking model, to mark up the footage of the actor; in the case of the solver model, to create poses or short animated sequences for the character matching the live footage.
Product manager Tony Lacey commented that how long the work takes depends on both actor and character, but “in many cases, we can create an initial identity within a couple of days”.
There are no restrictions on the topology or rig structure of the 3D character used.
Updated 9 October 2019: Cubic Motion has released a hardware update to Persona, reducing the size of the head-mounted camera and on-body system by 70%, and reducing the weight from 2.3kg to 1.25kg.
Other new features include a physical record button for “quick, manual capture in emergency situations” and basic audio recording capabilities.
Pricing and system requirements
Persona is available now, priced on enquiry. As well as the initial cost of the head-mounted camera, there are separate licence costs for the software, and for generating identities.
Updated 2 May 2019: Cubic Motion told us that pricing for Persona is “definitely high-end”, with the cost of the head-mounted camera exceeding that of both Faceware and Dynamixyz‘s models, but that it is “considerably less than a Fox/Technoprops system”.
The Performance Manager software is available for 64-bit Windows 10 Professional only. Processed animation data can be streamed to Unreal Engine 4.19+ or imported into Maya 2018.