How they did it: the visual effects team on Avatar: The Way Of Water gives us the 101

Pretty much everything you see in Avatar: The Way Of Water is there thanks to Wētā FX. When James Cameron gets an idea, senior animation supervisor Dan Barrett and senior visual effects supervisor Eric Saindon are among those tasked with finding a way to make it possible. Judging by reactions so far, they’ve succeeded, but in a conversation with The A.V. Club, the two technicians explain the challenges built into the process, reveal the visual cheat in the first film you probably didn’t notice, share the way of de-aging, discuss whether Avatar counts as an animated film, and tell us what the shoot had in common with NFL games. If you’ve ever wondered, for example, how nine-foot-tall CG characters match their eyelines and blocking with mere humans, read on.

Dan Barrett: That’s great!

AVC: But it made me wonder: to what extent, if any, did you try to make any of the creatures cute? Is there a degree to which you can push it and a degree beyond which you can’t?

DB: Well, I think that’s certainly the case. You can get pretty broad pretty quickly if you’re not careful. A creature that I find cute is the Ilu, you know, the domesticated mount the Metkayina people use. I mean, it can look pretty vicious when it opens its mouth and shows its teeth, but a lot of that is the creature design, and then also the character, and the way it moves, and how playful they can be. But I think you do have to be careful that you don’t go too far with a creature in a film like this.

AVC: Eric, you were instrumental in the creation of The Lord Of The Rings’ Gollum back in the day, and there’s a direct line from that to Avatar, where it’s the transition from dead eyes to soulful eyes in motion capture. What was the big leap from the uncanny valley to the windows to the soul that we’re at now?

Eric Saindon: I think it’s the performance itself, right? The ability to capture the performance and put that performance into our characters in a way that we’ve never done before. In fact, the system we started on Avatar, way back when, was a FACS-based system [Facial Action Coding System]. And Dan’s going to be able to describe this a lot better than I can, but it was really a sort of model-based system. So you went from shape to shape to shape. And the animator had to translate the performance a lot more. So back on Gollum, we didn’t have the ability to take the performance directly from Andy [Serkis] and put it on to Gollum quite as easily, so it required a lot more artistry from the animators.

But on something like Avatar: The Way Of Water, this is the first film [where] we’ve used a new system that allows us to really take that performance directly from the actors, and translate it back into the characters in such a way that we’ve never done before, which just gives you so much more information.

DB: I think the other really important thing about Andy’s performance in The Lord Of The Rings was that was really the moment when motion capture became performance capture. It was when the actor was given the confidence to fully go there, to be fully behind the process and understand that this was not about mechanics, this was about emotion. That was a big moment for films like these, and since then you’ve seen more and more, both directors and actors having faith and really going for it, where it becomes the performance and understanding that it’s not just about the way a body moves; it’s about the way a character feels.

ES: It’s also sort of a theatrical thing, right? Like, the great thing with Avatar is Jim [Cameron] had all the actors working all together, so it was like a big stage performance. And all of the scenes were shot like this, without worrying about the cameras, without worrying about a camera stuck in your face. The actors were allowed to just work together and get a really great performance. A lot of times if you get someone in a mo-cap suit, you tend to get overacting very quickly. Because they feel like they need to show their performance, so they overact. But if all of the performers are working together, they just go into the scenes, and they give you a great scene, and that really shows through in movies like Avatar where you really get that great performance coming through in a group, in everyone working together.

AVC: With a sequel like this, are you able to use any of the motion capture you got on Sam Worthington and Zoe Saldaña last time, or do you have to do all new scans of them, so to speak?

DB: In terms of the performances, no. The performances are all new. There’s nothing that would stop us from putting motion and performances from the first film on to this new film, but that’s not the way it was done. It was all new performances. And then … everything’s a rebuild. Everything has moved on so far, so much, that there was nothing we could really take, except for the fact that we had sculpts. We knew what they look like. So they were aged somewhat, to account for the years that had passed. But I think it’s fair to say everything’s new: new models, new shaders, new everything.

ES: Yeah, with all the new technology, I mean, we’re always upgrading. We don’t want to stick with the same tech we had 13 years ago. Everything has moved on: the way we build models, the way we put our characters together, the textures, everything. On the first film, the hair for Neytiri and Jake was solved as big strips of cloth. So if you really go back and look back, the hair-solves were … ehhh. But in this film, every individual hair on their heads was solved, and colliding with one another, colliding with the water, interacting with the water, interacting with other characters. It’s things like that, all those small details that people won’t even notice that are different, that make all the difference in the world.

AVC: Do you motion capture the water at all? James Cameron has said that all the water on screen was animated, but then I’ve also seen behind-the-scenes stuff of motion capture balls in the water.

ES: Well, the floating balls are a different thing. One of the big challenges that we’ve never had to deal with was performance capture underwater, and on the surface of the water. And the way we did that is we created two different performance volumes. One that basically gets sunk into a tank and shot underwater, and one that’s above the water. And they are sitting one on top of another, but the problem is the water’s surface causes all kinds of reflections, so those balls are actually to stop all those reflections, and allow us to do the performance capture above and below. It stops the reflections from below bouncing up to the surface, and the ones from above bouncing down. So that’s what those balls were for. That said, we were able to use that as really good reference for the types of waves in those shots, and the types of swells in those pools, because our characters had to sit on the same waves. Because the performance capture was done on the same wave patterns, and such, we had to regenerate the same waves.

AVC: In the L.A. Film Critics Association year-end vote, we recently had a big discussion about whether Avatar should be eligible for Best Animated Feature. Dan, as an animator on Avatar you’d be the best person to ask: what would your perspective be?

DB: It’s a debate I would like to be involved in; I would like to have been there that day, because I think it is something that needs to be talked about. Right now I would say no, the fact that it is digital doesn’t necessarily make it animated. I think that when we talk about feature animation, it’s a different thing. We talk about animators using acting skills to do the work, often driven by lead character animators. In a case like this, it’s wholly driven by the performances of the actors. So I would say no.

AVC: Do you motion capture any animals, like fish, or is that even possible? What’s the reference point for that?

DB: That is pure animation, [except] for horses. We do motion capture for horses. I’m just trying to make sure there’s nothing else … in the past, for other films, we’ve done dogs, I know. But no, everything else is animated based on reference from the real world.

AVC: Was any of the reference material stuff that Cameron actually shot from some of his deep dives, like footage from Aliens Of The Deep?

DB: I think “yes” is the answer to that. We take reference from wherever we can find it. That’s our bread and butter when doing creature work. I think any animator, whatever creature, when you’re faced with a creature for the first time, whether it’s a fantastical alien creature or something that is very much more like something on our planet, you look to reference. A tulkun is very close to being a whale, you know, and so whales are a big thing for us. So it’s been helpful, some of the work that Jim has done underwater with cameras.

AVC: What was the hardest thing to animate that people wouldn’t normally notice? You mentioned hair in the first film, which I don’t think anyone thought was particularly lacking. Was there something like that here, where it took so much work to do and you don’t think audiences will even notice?

DB: The corners of the mouth. From an animation perspective, that’s the hardest thing to get right where it comes to motion and the model. Perhaps not something that audiences would immediately recognize or understand, but that was something that everybody was looking at, so there was a huge amount of work done there to be as anatomically plausible as we could possibly be, especially in those two small areas of every face.

ES: For me, obviously you’d say the water is the hardest, right? The one people wouldn’t notice as being hard is the rain. The rain is something we didn’t have in the first movie. We had a couple shots of rain; didn’t really have a whole lot. But in this movie Jim used rain to tell emotion a little bit more, to help sell some of these scenes and get the feel for these scenes a lot more, so we had a lot more rain, we had a lot more detail, we had a lot more closeups, so we had rain falling on characters, we had rain hitting characters, creating a little crown splash on their shoulder, and then dripping down them, or getting on their costume, or all this interaction that we never really thought were gonna be quite as difficult to get right. And it’s just something we didn’t account for quite as much; when we had all these huge water sims, rain ended up being one of those hard things to do.

Same with bubbles. That’s another thing we didn’t really think was gonna be as difficult to get right as it really was. There’s so many states of bubbles, right? Like when the kids jump in the water, they generate this really milky, fine diffusion of bubbles that really is difficult to get working and feeling correct. The good thing is we had great reference, because all the performance capture was shot of the kids jumping in the water, and we had the bubbles that they generated so we knew what it should look like, and it just took time to get it right.

AVC: Production had several of the sequels going on at the same time. Was post like that, or was your work just on The Way Of Water? Or have you started on any of the others at the same time?

ES: We’ve really just done [The Way Of Water] at this stage. When we shot live action, we shot part 2, and 3, and the first act of 4, mainly because Jack [Champion, who plays Spider] grew like a tree over the course of this, so he shot up quite a lot, and we really wanted to make sure we got Jack’s size to be consistent across all the movies. We’ve gotten some of 3 to work on, but we’re really concentrating on 2, to get that right.

AVC: Given that you have the technology to de-age Sigourney Weaver into a teenager, does it really matter too much if Jack has a growth spurt? Could you digitally “de-spurt” him if you needed to?

ES: Well, the reality, yes. But it would be better to have Jack’s performance as Jack. The more you screw with someone’s face, the more “botox” they look. You’re gonna lose the subtleties of their performance, so you really don’t want to do that too much. So with Sigourney, because she was a full CG character, we were able to take her performance and translate it, and use reference from when Sigourney was 18, because she’s been onscreen for so long, we have a lot of performance of her, so we have great reference of things that her face would have done then that it might not do now.

AVC: Is Spider always Jack, or was there a CG double ever created for him?

ES: There’s a couple scenes where he’s doing things that we really couldn’t get a 16-year-old kid to do—falling down cliffs, and things like that, but for the most part it’s Jack, or, there’s a couple scenes with a double, just where you don’t want to put your main actor through certain stunts.

DB: Just like in any other film, he has a stunt double. And there are shots in there with digi-double versions of Spider, but in those cases it would either be that we’ve done it to help with integration, and we’ve match-moved Jake’s performance from the day, so his body looks exactly the same, or it’s that earlier the performance capture was done with Jack in a suit, and then later the live-action was done with Jack in his costume and makeup. There’s plenty of digital versions of him, but on the whole, it’s Jack.

AVC: We’ve all seen behind-the-scenes footage of when actors play taller CG characters, like She-Hulk, and they have reference faces on posts above their head. When they’re playing smaller CG characters, like Sigourney, what does that look like? How do you get the eyelines?

ES: Well, Kiri’s actually seven and a half feet tall. That’s the thing that you don’t think about when you’re watching these movies. She’s quite a lot taller than Sigourney was, and to interact with Spider, because we didn’t actually have anyone for him to interact with on set, we had a new thing, a fly-by wire camera. Basically from sporting events, you see that camera that’s flying overhead, over NFL games. We had a very similar setup, where we had the wire cameras over set, and hanging off that, off a pole, we basically had a very small HD monitor, and we had Sigourney’s performance on that monitor, and it tracked to the location of where the performance was that Sigourney did, so that when Jack was there interacting with Sigourney, he had an eyeline to where Sigourney was supposed to be from the performance, moving around set and with Sigourney’s face on it, so he had something to interact with for timing, for dialogue, all those things.

DB: Yeah, because the performance happened before the live-action shoot, it was a very accurate eyeline. The screen would move exactly where the CG head of Kiri was ultimately going to be.