How North of Now created holo doubles for Kaytranada’s ‘Witchy’
Software developer KeenTools and production company North of Now have released a VFX breakdown for the video for Kaytranada’s Grammy Award-nominated track Witchy.
The video, which features Childish Gambino, features 3D holograms of both musicians’ heads inside a crystal ball, created using KeenTools’ FaceBuilder and FaceTracker plugins for Nuke.
Other software used on the project included Cinema 4D, 3DEqualizer and RealityCapture.
One of the custom 3D head meshes created using KeenTools’ FaceBuilder plugin.
Creating the 3D heads from reference photos
The live shoot took place in New York, with North of Now collecting on-set measurements, HDRs and photogrammetry, plus the reference photos used to reconstruct Kaytranada and Childish Gambino’s heads in 3D, and video footage used for facial motion capture.
The photos were shot using Sony’s ZV-E10 camera, which has no lens or sensor stabilization.
“Getting photos of both artists took around 30 minutes, most of the time spent on … setting up the lighting,” says VFX Supervisor Igor Eyth. “There were around 10 shots for each actor, the most important ones being frontal, two three-quarter views, both profiles, and higher and lower front.”
Eyth uploaded the photos to FaceBuilder for Nuke, matching the neutral 3D head mesh generated by the plugin to the reference images to create custom 3D head models for both musicians: a process that took around an hour per head.
“My favorite feature is the Align Face button,” he says. “It automatically matches [the] mesh to [the] face and actually does a pretty good job, saving you a lot of time.”
A reference plate compared to the 3D face track generated in FaceTracker.
Tracking and texture projection
For facial tracking, Eyth used FaceTracker for Nuke, which tracked the 3D head models to match the facial performances in the video shot on set.
“It was almost one-click,” he says. “I plugged [each head model] into the geometry input, lined it up in the initial frame, and I think there were only four adjustments during the actual tracking.”
The tracks were exported in .abc format, with 3D modeler Sergey Malov adding hair to the 3D heads – done using static meshes, rather than simulation – and CG sunglasses for Kaytranada.
Skin textures were taken from plates shot on set with an Arri Alexa: two of each musician.
Eyth projected the technical plates onto the animated 3D heads inside Nuke, using KeenTools’ TextureBuilder node to fill in gaps on the necks and ears.
The base 3D scene, and the completed hologram effect, developed in Cinema 4D.
Developing the hologram effect
With the plate, the camera track, and the animated and textured heads in hand, Eyth created the hologram effect in Cinema 4D, Maxon’s 3D animation and rendering software.
“I like to have flexibility [during] compositing [so I created] quite a number of render passes,” he says. “There was a bubblegram pass, a more common diffuse pass using the projection from the main plate, and a number of experimental passes, including a scanline pass.”
Eyth worked closely with director Bethany Vargas to develop the look of the hologram, eventually coming up with a “less digital, more analog” look that matches the “old-schoolish” feel of the video.
In total, the effect took four days of work, including compositing.
Other effects and clean-up work for the video
As well the holograms, North of Now worked on around half the shots in the video, with a team of 13 creating set extensions, clock animation, and other effects, using 3DEqualizer for camera tracking, and RealityCapture – since renamed RealityScan – for photogrammetry.
“I’m absolutely happy with the result,” says Eyth. “I think Bethany incorporated visual effects in this music video in a very subtle way so they’re not distracting but adding a new dimension to it.”
Have your say on this story by following CG Channel on Facebook, Instagram and X (formerly Twitter). As well as being able to comment on stories, followers of our social media accounts can see videos we don’t post on the site itself, including making-ofs for the latest VFX movies, animations, games cinematics and motion graphics projects.