While people think of visuals as being the biggest achievement of Avatar, it would be more appropriate to call it immersion – the feeling of being there, without any distracting clues that the world and its characters are computer-generated.

Bringing characters to life in a convincing manner is a daunting task. Final Fantasy – The Spirits Within was the first to try and they failed spectacularly. The reason is psychological: while the brain can interpret a simple stick figure as a human, as the complexity of the increases, perceived realism improves until a point where the character is almost – but not quite – alive. Such a character looks real but dead. This dread area is called the Uncanny Valley. After the Final Fantasy flop, few have ever attempted to cross the Uncanny Valley; although LoTR’s Gollum is often cited as a realistic digital character, he was supposed to look repulsive, so the challenge wasn’t that big.

Performance capture

James Cameron wanted to create beautiful, sexy characters you could fall in love with (and judging from various talks on forums, it seems that Neytiri had that effect) and for this, rough motion capture that is usually employed was not enough.

Cameron and his team coined the term performance capture – that is capturing all the nuances, body language and feelings of an actor and translate them on the digital counterpart. For this they used innovative solutions from Giant Studios (that company was responsible for all motion capture) and they added a helmet-mounted camera attached to a rod, that sit in front of the actors’ heads; this way, the facial expressions could be captured along with the whole body motion. The helmets also featured ears and braids, helping actors remain aware of them. In addition to that, they used up to 12 HD cameras to capture the actors from different angles in wide and closeups to preserve all the fine details.

This approach helped the animators at Weta  to concentrate only on bringing up every nuance of the original performance, like small winks, swaying of the hair or movement of the ears. It was also a boon for actors, now freed of tedious details like markers, lighting and costumes, now being able to concentrate on what they do best – conveying emotions.

They also used motion capture for creatures (using actors for stand-ins), and even for helicopters and banshees. This allowed for a better interaction with the actors and more natural motion paths (especially tricky for helicopters, which sway in their path in a way only a helicopter pilot can describe). The Weta animators still had to refine the motion and add all the necessary details, but the general motion path was already given to them in the way the filmakers wanted.

Real-Time feedback

Everybody has their own camera eye and sensibility and to translate that through an intermediary is very difficult

Another problem of working with CGI-heavy pictures is that the actors play in an empty space and when standing on the floor it’s difficult for everyone (actors, cameramen, director) to imagine how the end result will look like. So James Cameron and his team devised a “virtual camera” hooked to Motion Builder, allowing them to see the virtual environments in real-time.

“Everybody has their own camera eye and sensibility and to translate that through an intermediary is very difficult“, said Rob Legato, referring to the traditional workflow that required the animators to handle camera movement, with variations taking weeks to complete.

James Cameron holding the virtual camera

For Avatar, they would move the camera in the empty set and the virtual camera would do the same inside the simulated environment, all while actors movements were being captured and applied to their digital models. This way, instead of doing a scene with actors, having the motion interpreted later and receive an animation weeks later, the production team could see the results instantly, providing them with the essential feedback needed to refine the details. This approach allowed them to stop using storyboards and just experiment. For camera work alone, they could try 15-20 variations – wider, lower – to get the best look.

Viewfinder shows low-res CGI with real-time motion capture applied

We gave Weta what was essentially a rough videogame version of the scene

When a scene was ready, it was given to Weta and they replaced the low-res models with high-res ones, did the tweaks and small details and rendered it.

Another clever thing was the Simulcam – a rig that allowed them to do real-time compositing over green-screen. So they’d film actors in areas that required digital extensions (for example the hangar bay) and the image in the viewfinder would show the composited result, with the digital backgrounds instead of the green screen. Needless to say, this is a huge help for people behind the camera, now able to see how the scene is going to look like and how to frame it.

Low-res realtime footage

Attention to detail

The creatures started as sketches on paper and were then digitally sculpted with Z-Brush. Vehicles were modeled with Maya, XSI and 3ds max.

ZBrush study

For the avatars, Stan Winston Studios built full-size models of generic Na’vi male and female characters that were laser-scanned and tweaked.  Incorporating features from the human characters was a challenge in itself as the avatars had to have some resemblance to their ‘operators’. Moreover, a face conformation too different from the actor’s face did not translate well for face performance capture, this is why the Na’vi ended up looking quite similar to the actors portraying them.

Direhorse muscle system

Speaking of performance capture, to fully retain all the nuances of the body language, the Weta artists had created all the face muscles, fat and tissue and used state-of-the-art shaders and light models to give the sense of depth beneath the skin, with the usual sub-surface scattering model replaced with a sub-surface absorption one.

The modeling and texturing of the models and characters had to go through several update cycles, because as the one character model was made more realistic, all other CGI near it started looking obviously fake and had to be improved as well.

Someone actually modeled all the stitches and tiny details

For the vegetation, Weta created a library of almost 3000 separate plants and tress, that enabled them to “decorate” the Pandora jungle in a realistic fashion. By the way, initially the vegetation was supposed to be cyan, but the jungle was too alien, so they bough back some green for a more familiar look.

Further reading

You can find out more about the companies and software involved in the making of Avatar. Weta did most of the work, but even they were overwhelmed by the sheer number of CG scenes, so received help from ILM for the animations involving humans’ aircraft and from Framestore for the Hell’s Gate compound.

PS: Congratulations on winning the Golden Globes Award. Despite its detractors, I believe Avatar is strong film both on technical and human level.

Updated on January 16 with more details. Also, in a new article I’m providing some details on the challenges and innovations involved in the making of Avatar.

Although usually only big names like Weta Digital or ILM are mentioned in the making of high-budget CGI, there are always tens of companies involved (possibly hundreds considering that work often gets outsourced), with thousands of digital artists, 3d modelers, animators and compositors working to bring the script to live.

It’s also too bad that many of the companies (the ones involved in the production and the ones making the software) do not promote their software more. Adobe was the quickest to show how their line of software was involved, even though it wasn’t really crucial to the project and this is why I decided to make a list and try to give proper credit to everyone involved.

Wireframe of a Na'vi

For Avatar, these are the companies credited for the visual effects:

• Weta Digital, New Zealand (most of the work, character modeling, animation, rendering)
• Industrial Light and Magic, USA (180  non-character shots, mostly the human aircraft in the final battle)
• Stan Winston Studios (most of the props, including the Ampsuit and the Samson helicopter)
• Framestore, UK (70 shots of the Hell’s Gate)
• Hybride, Canada (graphics for the link room)
• Prime Focus, USA (design and compositing of the control room screens, HUDs, etc)
• Look Effects, USA (compositing)
• Hydraulx, USA  (digital extension shots of crematoria and cryovault)
• Giant Studios, USA (motion capture)
• Blur, USA (4 space shots of ISV Venture Star)
• Pixel Liberation Front, USA (screens and HUD design)
• BUF (the conscience-transfer tunnel)
• Lola VFX, USA (digital cosmetic)

ZBrush study

And here are the main software used:

• Autodesk Maya
• Pixar Renderman for Maya
• Autodesk SoftImage XSI
• Luxology Modo (model design, e.g. the Scorpion)
• Lightwave (low-res realtime environments)
• Houdini (unknown area)
• ZBrush (creature design)
• Auodesk 3ds max (space shots, control room screens and HUD renderings)
• Autodesk MotionBuilder (for real-time 3d visualisations)
• Eyeon Fusion (image compositing)
• The Foundry Nuke Compositor (previz image compositing)
• Autodesk Smoke (color correction)
• Autodesk Combustion (compositing)
• Massive (vegetation simulation)
• Mudbox (floating mountains)
• Avid(video editing)
• Adobe After Effects (compositing, real-ime visualizations)
• PF Track (motion tracking, background replacement)
• Adobe Illustrator (HUD and screens layout)
• Adobe Photoshop (concept art, textures)
• Adobe Premiere (proofing, rough compositing with AE)
• many tools developed in-house
• countless plugins for each platform, some of them Ocula for Nuke, Ktakatoa for 3ds max, Sapphire for Combustion/AE.

The list of tools is not exhaustive and you can see there’s overlap in capabilities, depending on each company’s pipeline.