Friday, October 5th, 2012 Posted by Jim Thacker

Weta’s Matthias Menz on new trends in CG lighting

From its work on the Lord of the Rings trilogy to recent successes like Avatar, Weta Digital has continually broken new ground in visual effects. But while the studio’s innovations in creating realistic digital characters and environments have often – and rightly – been celebrated, the same is true of other parts of the production pipeline.

For 2011’s The Adventures of Tintin, Weta moved towards a model of CG lighting informed by traditional cinematography and visual art. We caught up with Matthias Menz, a key player in those developments, and one of the movie’s four VFX Supervisors, to discuss the shift from technically to artistically oriented lighting, the impact of new technology, and how the role of the lighting artist will evolve in future.

CG Channel: Let’s start with your own pipeline. How have Weta’s tools changed technically in recent years?

Matthias Menz: We rolled straight from the technology developed for Avatar – including area lights and Spherical Harmonics-based Lighting (SH) – into the new developments for The Adventures of Tintin.

For Tintin, we knew that we had to do quite a few indoor scenes and SH, as it turns out, is more suited for outdoor lighting, so we needed to innovate around indirect illumination.

We have had ways to compute indirect for a while, but not ones fast enough to be used on a broader scale. Typically, we either approximated the look, or used it only in selected scenes to avoid a massive render hit.

Indirect illumination consists of two things: indirect diffuse and a specular component dependent on the camera angle. Indirect diffuse has been used for some time, but indirect specular isn’t typically accounted for by the renderer and makes computation much more involved. But without it, images look wrong.

Our implementation is based on optimized precomputed incident light caches. Separating static and dynamic (per frame) caches allowed for good speed improvements. We introduced a new refinement-dicing pipeline, which basically means that the tessellation is only what it needs to be for any object inside or outside of the camera frustum. All this allowed us to use indirect for the majority of the film and not just for selected scenes.

Another fantastic tool we developed is PantaRay shadows. Using our in-house PantaRay raytracer, we were able to compute realistic raytraced shadows for area lights. It correctly computes the penumbra or shadow falloff based on light’s size and distance from an object for much greater realism.

In creative hands, all of these developments added greatly to the realism and visual richness of Tintin’s world.

The look of The Adventures of Tintin is informed by classical cinematography and visual art. The work of artists such as Rembrandt and Caravaggio informs the way that characters are lit, particularly their faces.

CGC: You looked to the work of Janus Kami?ski, Steven Spielberg’s regular cinematographer, for reference. What lessons did you learn from it?

MM: We were wondering: is there a difference between the way live-action DPs approach lighting and what we do in the computer? That is to say: is there a difference in approach between DP vs TD lighting?

We found that being grounded in cinematography resonated really well with the director, and we were getting first-look approvals faster.

It’s actually quite interesting, if you think about it. In everything else we tried to honor the look and feel of Hergé’s original comics. But for lighting, we went the other way. The comics were drawn quite flat, in simple pastel colors, but for the movie we went for dramatic lighting: almost a film noir style in areas. It’s a gutsy, more dramatic look for CG film lighting.

Grounded in reality: another source of reference for Tintin was stock footage of opera stars such as Maria Callas.

CGC: Let’s talk about how that translates to the role of the artist. What do you look for in a lighter?

MM: It depends on the job. The skills suited to live-action work and full CG are slightly different.

Lighting to live-action plates, you’re dealing with chrome ball extractions, plate matching, scripting, and so on. It requires skillful choices, but it leaves less room for creative freedom, because you’re matching a reference.

On the other hand, lighting for a full CG film can be more creatively demanding, because it’s more open. You’re working with much less reference, and there isn’t a plate to use for guidance.

For full CG, I’d look for good artistic sensibility along with the technical skills you look for in any lighter.

?CGC: So is it harder to recruit one type of lighter than the other?

There certainly seem to be more technically skilled people than artistically talented ones, because historically CG has been such a technical endeavor.

Most of CGI is a very left-brained activity; it’s very analytical. But lighting, just as much as acting, is a right-brained activity. It’s often counter-productive to apply too much analytical logic, or you run the risk of getting a result that looks like a documentary.

The meeting between art and science. “Lighting is a right-brained activity,” says Menz. “Apply too much analytical logic, and you run the risk of getting a result that looks like a documentary.”

CGC: Pixar’s Jeremy Vickery told me that he’d like to get to the point where he could recruit people with a background in cinematography or illustration, not CG software.

I agree: for full CG films, in particular. After all, we’re in the business of making a good image. Which implies that software tools need to be easy enough for creatives to handle scene complexity, and workflows made easier so they can focus on what they do best.

CGC: So how long will it be before the tools become that transparent?

It’s well underway. Our Weta Research team is pushing in this direction, just like many other visual effects houses and software companies.

We also found that you can help by synergistic team building: for example, having separate technical and artistic leads, not trying to combine them into one person.

CGC: How close are we to having truly usable real-time feedback within lighting tools?

MM: I’d say quite close, particularly if you include ‘nearly real-time’. The latter is certainly fast enough now for many tasks, like previs, shot lighting, fur grooming and texturing.

What might take a little longer is true global illumination in real time, at least for anything more complex than a Cornell box. GPU memory is an issue with complex scenes.

However, if the goal is to show the director more realistic lighting earlier in production, it may not be beneficial to take too many visual shortcuts for the sake of speed. If the look of the previs diverges greatly from that of the final render, it defeats the object of the exercise.

A painterly approach: today’s lighting artists need to move from the ‘how’ of their craft to the ‘why’, says Menz.

CGC: How has the role of a lighting artist changed over the past decade?

MM: Scene complexity has gone way up. We are creating worlds now that were unthinkable ten years ago. As lighters, we have to grapple with that increased technical detail.

For full CG films, I’d also add a requirement for more cinematic lighting. It’s beneficial for a lighting artist to know more about color, composition and light motivation; to move from the ‘how’ to the ‘why’.

I’ve been very fortunate to be able to light and lead lighting at Weta for 11 years, but it was only when we started to work on Avatar, half of which is completely CG, that it fully occurred to me: “You know, maybe we are little cinematographers here”. A few of us then started to look to classical cinematography seriously for guidance.

The technical evolution of lighting tools will feed into a more artistically driven approach to the job. “Lighting is make believe, but you don’t want to have to cheat in the computer for any other reason,” says Menz.

CGC: How do you think things will change in future?

MM: The tools around us are going to be far more interactive. GPU-based lighting will speed things up massively and lighting discussions will move further up the pipeline. More realistic results are going to be possible with less iteration.

On the rendering side, we’re going to see more correctly behaving materials and luminaires. Rendering will be mostly in raytrace mode and will mimic the look of the real world a lot more closely than in the past.

CGC: What does that mean for the lighting artist?

MM: Lighting workflow wants to mimic live-action cinematography more closely. Fewer CG cheats will be required, and you will want to constrain yourself to what’s possible on a real set. In other words, you won’t want to place your area light underneath the floor and turn shadows off: that sort of thing.

Cinematography is by definition creating an illusion; writing with light. It’s make believe, but you don’t want to have to cheat in the computer for any other reason.

CGC: So what’s the most important piece of advice you could give someone starting out today?

To learn about lighting motivation: to move from the ‘how’ to the ‘why’. Lights are brush strokes on a canvas, not things to be placed at random.

A good exercise might be to find live-action film references and see if you can replicate/relight them in the computer. Try to guess where DPs placed the lights, and learn from their choices. Grab random models, pretend they are in a film, and light them.

Imagine that a whole virtual film set for a sequence is handed to you: that you are sat in front of a blank sheet of paper, lighting-wise. What would you do next? Where would you place your brush strokes – and for what reason? And there has to be a reason.

This, for me, is a remarkable moment. It’s magical to be so close to the look of a picture.

Visit Weta Digital online