Sidebars
The Producer's Perspective
DP's Double Duty

Massive Effects Pipeline Evolves for Sequels

The so-called "burly brawl" (top) and the freeway sequence (middle and bottom) were two of the most complicated scenes produced for Matrix Reloaded..

Back in the late '90s, when they were trying to get The Matrix made, the Wachowski brothers — visual bibles in hand, filled with creature concepts and storyboards by Geof Darrow and Steve Skroce — showed their proposal to veteran visual effects supervisor John Gaeta. That meeting inadvertently set wheels in motion to develop not only a surprise box-office hit, but also an Academy Award-winning visual effects team that has been very busy the last two years on two new Matrix movies.

The effects' work in Matrix Reloaded and Matrix Revolutions attempts to continue the legacy of the franchise's award-winning original film. Ghostly twins materialize from flesh with decreased opacity, emulating a cybernetic form. Characters fly through cars to poised attack positions. Wide camera shots show the city of Zion with squid-like robotic machines digging deep into the Earth while attacking the human fortress. And hundreds of replicated versions of Agent Smith (Hugo Weaving) furiously battle Neo (Keanu Reeves) in Reloaded's centerpiece — a CG/martial arts extravaganza dubbed “the burly brawl.”

To understand the massive digital effects effort (approximately 2,500 shots between the two new movies on an effects budget surpassing six figures, according to producer Joel Silver), Gaeta insists it is necessary to understand the nature of the team behind the job. Although the two films rely on work from several different facilities, Gaeta says that the core group responsible for effects on all three Matrix films has been together since they met while he still worked for industry legend Douglas Trumbull in the early '90s. That group collaborated at now-defunct Manex Visual Effects, which produced the first movie's effects, and many of them are now continuing the collaboration at ESC Visual Effects, Alameda, Calif., lead facility on the new films.

During the early Trumbull phase of his career, Gaeta says he began to connect with like-minded, talented people on similar tracks. These were people who, like him, had become obsessed with the notion of seamlessly integrating digital and real-world images.

“[At Trumbull's company, Mass.Illusions], I got connected to a fellow named Joel Hynek [Predator, Oscar-winner for What Dreams May Come],” Gaeta says. “I worked with him on that project for a while, but left to work on Matrix. Before all that, when I hooked up with Joel at Mass.Illusions, the pair of us decided to steal the CAD department away from Doug's art department and repurpose it to do 3D visualization for shot planning and for exporting NURBS that we made in the 3D animation environment into motion-control systems, so that we could get precisely what we were designing in 3D in advance. That was our real beginning of being basically connected between the three-dimensional computer graphics rules and the rules of the real world as it pertains to photography. We began speculating about how you define physical space so that you can integrate computer graphics.”

Around this time, Gaeta also encountered other key influences responsible for helping to develop the famous “bullet-time” effects that won the first Matrix movie an Oscar.

“During What Dreams and into the beginning of Matrix, I became very interested in meeting people who were doing similar types of experimentation of photogrammetry and shape extraction,” he says. “[During this period], I eventually hooked up with Kim Libreri, George Borshukov, and Dan Piponi.”

Gaeta credits that trio for its groundbreaking, develop-mental work on the principals that later allowed them and other colleagues to formulate the “bullet-time” effect. At ESC, they are trying to surpass that accomplishment with work on the new films. Gaeta refers to Libreri, currently ESC's visual effects supervisor, as “probably the most knowledgeable person in the entire industry when it comes to image quality and understanding thresholds of resolution enhancement, understanding the properties of film, and how to simulate them.”

Meanwhile, Gaeta calls Piponi, ESC's current R&D chief, “a brilliant R&D innovator,” and he credits Borshukov, ESC's effects' technical supervisor, with pioneering some of the original algorithms, which have emerged as the foundation of contemporary image-space rendering methods.

“[This has led to the development of] the most realistic renderer that has ever been completed for computer graphics,” Gaeta says. “It was basically the beginning of projecting textures onto forms and extracting the forms by way of multiple camera views of those forms — suddenly getting this enhanced 3D that people had not seen before, photographic-based texturing.”

Actors in this year's two Matrix releases appear in both live-action and animated form throughout the two films.

Gaeta says this team is now evolving into the world of true “virtual cinematography.”

“[That group] embodied what [has now become] this whole new realm of what we call ‘reality capture,’” he says. “Other terms we've used are ‘virtual cinema’ and ‘virtual cinematography.’ That was the beginning of that phase of doing ‘bullet-time’ stuff — trying to represent the concept of working with a virtual camera and a percentage of it with still cameras, all planned in 3D in advance. We are using locations, floating interests points, and things that real cameras do, as opposed to fixed interest points in frozen time. Instead, we are doing floating interest points and dynamic time, because we could figure it out by way of 3D visualization, and then we were doing some of the very first virtual backgrounds ever for feature films.”

Gaeta hastens to add, however, that the new films should not become known for any one visual effect approach. “They are two of the most diverse films that you will ever find in terms of effect styles, looks, and themes. We have everything from creatures to procedural animation to procedural destruction that looks really cool and stylized to simulated phenomenon like plain water, weather — all that stuff — done in really interesting ways. There are totally psychedelic, full-on, head-banging, tripping, freaked-out visuals that are there because the directors wanted to pursue a surreal flavor of action.”

As the sequels evolved, Gaeta had to form a much larger team to handle the load. He spent months hand-picking the talent — a mixture of longtime colleagues, well-established supervisors, emerging leaders in the effects world, and young, newly discovered code-crankers, all collaborating globally between a mix of well-known effects facilities and ESC.

To put all this in perspective, keep in mind that this team was responsible not only for the simultaneous effects work on the two films, but also the parallel video game release and material for the anime-styled, animated prequels that are now being released on DVD (see “Final Fantasy Live On,” Millimeter, March 2003). This volume of work on multiple releases from the same franchise is almost unprecedented. Only the Lord of the Rings trilogy and, years ago, the second and third installments of the Back to the Future franchise have ever attempted a production schedule approaching this scope. In the end, the team Gaeta formed consisted of more than 500 postproduction artists at eight facilities on three continents, not to mention on-set visual effects unit crews and other crews from related departments, all working simultaneously on Reloaded and Revolutions — creating, tracking, and wrangling thousands of digital assets.

“I am the effects designer/director and senior visual effects supervisor, working for [production company EON Enter-tainment], the creative interface between Larry and Andy and all the other vendors on this job,” Gaeta says. “I have a staff and other small flex departments that go along with that. I have two associates, [visual effects supervisors] Dan Glass and John Des Jardin. These guys are basically my right and left arms for all manner of logistics management, image quality, and trying to basically create a continuity of communications once I have found a basic directive/path, helping me sort it all out with either Larry or Andy, or other designers, like [production designer] Owen Paterson or [DP] Bill Pope.”

Gaeta says Glass and Des Jardin, along with visual effects producer Terry Clotiaux, were key players in helping him keep track of more than 2,500 shots — “all manner of scanning and layout and quality control, while I am working on the animation direction and look and concept of the shots.”

Beyond his army of animators and compositors, Gaeta also had data wranglers, a LIDAR (Light Detection and Ranging) laser scanning team, an aerial unit, a live-action motion control unit, a pyro unit, and a miniatures unit collaborating with him. Gaeta also had a mo-cap unit called “universal capture,” which was a proprietary performance-capture system that Libreri developed at ESC.

“We also built a huge motion-control stage,” Gaeta adds, “to do a lot of the stuff that we had to do running in parallel with first-unit photography on the same stages with all the primary actors.”

Gaeta says the many effects vendors came next in that structure. He emphasizes, in particular, ESC's work as a lead vendor, with approximately 300 people working on the project. “That's a facility funded specifically for Matrix,” he says. “[I consider] ESC a nexus of talents from the best companies.”

That group was responsible, among other things, for a variety of virtual humans, car-chase effects, and what Gaeta calls “stylized destruction and stylized phenomenon,” as well as extensive creature-animation work under the supervision of former ILMer George Murphy, much of which won't be seen until Revolutions debuts.

Gaeta also brought Sony Pictures Imageworks into the mix, giving that facility “a large package of shots that are quite involved.” In particular: The City of Zion, a labyrinth of tunnels in both films, sewer crafts, and the squid-like creature scenes were all handled at SPI.

Simultaneously, in Paris, BUF Compagnie toiled away on stylized, perceptual effects under the supervision of Stephane Ceretti and Pierre Buffin, according to Gaeta.

Other key vendors on the project included Giant Killer Robots (San Francisco) which had supervisor Mike Schmidt oversee the creation of what Gaeta calls “scenes beneath the Earth, new spaces we visit.” Under Lynn Cartwright's supervision, Animal Logic (Sydney, Australia) created the twin dreadlocked ghosts effect seen in trailers for Reloaded.

The freeway scene was shot at the beginning of the mammoth project, on a small stretch of specially built freeway in Alameda, Calif.

Tippett Studio (Berkeley, Calif.) added important creature environment shots on Revolutions, under Craig Hayes' supervision, but was not involved on Reloaded. (Other shots began at now-defunct Centropolis Visual Effects, but were transferred to SPI when Centropolis closed during production.)

Gaeta is particularly interested in promoting both the quality and quantity of the work done by these companies, pointing out that despite the huge combined budget for the two movies, the 2,500-plus shots might be considered a bargain, when one does the math.

“We're working multiple miracles in parallel here because of the amount of content and the quality that is insisted upon,” he says. “People like to think, ‘oh yeah, they get to spend a lot of money.’ But if you look at how much content the Wachowski brothers are creating, and you actually run the numbers, an argument could be made that we have even less money per shot than the first movie, and what we have is a huge amount of content to accomplish, and very complex at that. We had to be very smart in amortizing the R&D and how we use it.”

Gaeta and his team of supervisors worked two years on evolving a process that began with primordial concepts during a February 2000 meeting with the Wachowski brothers and other key participants, and grew through a pseudo-Darwinian path to its full realization a week and a half before delivery in the first quarter of 2003. From those early previz meetings, the team grew from a small, tight group of artists and the two directors to meetings that eventually included close to 50 people breaking down shots in detail.

But Gaeta has an almost religious devotion to that early CG previz stage, something he believes essential to the success of a project with such a large scope. All effects scenes were laid out both in storyboards and 3D form to lock down timing, angles, action, and so forth. Entire sequences were completed and cut together in previz form, allowing Gaeta's team to troubleshoot thoroughly before spending millions on any particular shot.

Some previz versions of the elements, created by Pixel Liberation Front (PLF), Venice, Calif., working as an EON subcontractor, were then used as references on set during shooting to help the directors execute live-action scenes. Those digital pieces were also integrated into an on-set compositing system, allowing the Wachowski brothers to get good compositional previews of final effects shots long before they were final.

Plates were then shot, scanned, and supplied to whichever vendor was in charge of that sequence. Vendors also received the 3D previz elements, allowing them to create early rough scene comps. Each vendor then puzzled these various pieces together with their corresponding animated elements, replacing over time the previz proxies with higher-resolution versions. Each stage of development added more detail to the scenes, with Gaeta's team often changing parameters and adding levels of complexity as they went along. “We were never completely out of the previsualization stage until final post,” says Gaeta.

During production, Gaeta's team had a cornucopia of tools at its disposal to make sure the production could figure out real-world parameters for the many 3D elements.

Most of the facilities involved used Maya-based pipelines, with rendering approaches varying between RenderMan, Mental Ray for its global illumination rendering capabilities, and a handful of custom solutions. Shake was widely used for compositing, with both films featuring some Inferno work as well. ESC used its proprietary, photographically based pipeline for acquiring textures for synthetic humans and virtual backgrounds, while other facilities also used Maya 3D StudioPaint for texture work. Filmbox processed motion-capture data, and LightWave performed some modeling work. ESC and most of the other facilities involved also relied heavily on a wide range of proprietary plug-ins.

The toolbox also included laser scanning systems, photogrammetry, tracking markers, tracking cameras, observation cameras, optical trackers, and 3D lasers. All of these provided data to permit Gaeta and his colleagues to create a viable relationship between the 3D world and the physical world. Also, the production extensively used miniatures, optical tracking, a complete package of coders, and what Gaeta claims are probably today's largest blue- and greenscreens.

Glass and Des Jardin were in charge of the complex bluescreen and greenscreen setups. Glass supervised bluescreens for all the green-hued subjects photographed inside the Matrix set and on the film's motorcycle stunt course, while Des Jardin handled greenscreens for all the blue-hued subjects filmed in real-world locations.

Glass's team created, among other setups, a 50'×80' bluescreen and another completely immersive bluescreen environment called the “blue egg,” essentially a hollowed-out, egg-shaped rig built on Stage 1 at Sydney's Fox Studios, painted blue on the inside. The unit had arms on each side, to which actors could be attached and rotated while being filmed for what visual effects coordinator Dennis Cooper calls “crazy camera moves.”

Des Jardin, meanwhile, supervised the largest greenscreen on the production, and maybe the planet — a 131'×279' soundstage covered entirely in greenscreen.

“These were mostly custom screens imported from Thailand and covering approximately the entire width and length of the stage,” says Des Jardin. “We also had perimeter screens that were [about] 39ft. to 46ft. tall, depending on certain walls or ceiling obstructions, and you add to that the additional smaller greenscreens [20'×20'] scattered around to take care of custom edges on these huge sets, and we're talking about quite a lot of greenscreen. Most of the sets were pretty much surrounded by green to allow [the Wachowski brothers] to move the camera all around their subjects, with plenty of coverage for future background replacements.”

The effects team also utilized a digital daily system at ESC, which Gaeta calls “one of the best digital daily screening systems ever devised.”

“We played the images off hard drives, using an Iridas FrameCycler [a PC-based, uncompressed video playback system designed for digital daily use by German manufacturer Iridas],” explains Cooper. “We used in-house software to create playlists on the fly to generate dailies, allowing us to put them into the editorial cuts we got from production, trim the shots, and compare new versions of shots to older ones by easily searching and playing off hard drives. We viewed the clips through a JVC DLA-QX1G projector in our main theater, and we also watched dailies in breakout rooms using two JVC DLA-G150CLU projectors. It was an impressive setup because we were able to see different shots, comparing different versions, just sitting there in the theater, without significant delay.”

It wasn't easy keeping tabs of 2,500-plus visual effects shots. EON engineers therefore devised a proprietary media management and relational database to keep track of digital assets, data, records, and mundane details. That system, dubbed “the Zion Mainframe,” stores virtually all of the digital assets pertaining to the Matrix movies. Plates, 3D models, textures, previz animation, work in progress, and virtually every single asset that has been drawn, painted, modeled, and rendered dwells inside the Zion Mainframe.

The system's proprietary software features a node for each department under the artistic umbrella, and those departments looking for specific data can each query the system by department, artist, shot number, vendor, scene, and keywords. The project spanned three continents — North America, Australia, and Europe — and the Zion Mainframe was designed to be accessible from EON offices in Sydney, Los Angeles, and Alameda.

The movies, though, aren't all about CG creatures and environments. They also have a human side, with well-known characters returning, played by Keanu Reeves, Lawrence Fishburne, Carrie Ann Moss, Hugo Weaving, and others. Producers made those actors available to the effects team during preproduction for a series of facial-capture sessions, using the proprietary Universal Facial Capture System. Libreri, Borshukov, and Piponi wrote the system's optical flow code, and ESC CTO Paul Ryan wrote the actual capture software. That, in turn, permitted the effects unit to apply facial performances directly to 3D versions of the characters.

The system used five Sony HDW-900 HD cameras, taping facial performances from actors sitting in chairs. The cameras were modified to offload the images as uncompressed data directly to a disk array for later use. That storage system consisted of a series of arrays from Huge Systems, Agoura Hills, Calif. The mo-cap unit ran three capture systems simultaneously in Australia, with each relying on two 1TB disks for storage, giving a total of 6TB of storage at any given moment.

“The way it was set up, we could take those six disks offline to a [Sony] Tape Robot system, which pulled data off the disks onto tape, and replace them with six other disks,” says Cooper. “We also had a backup capture station always running, and a couple of extra disks, so we had about 20TB of storage on stage during the capture shoot.”

That data was later fed into computer systems running proprietary algorithms designed to analyze facial movement and calculate 3D information, triangulating points on the face from the positions of the five cameras. This allowed Gaeta's artists to view the animated face from nearly any angle.

Reloaded features an out-of-control, over-the-top chase down a busy freeway, shot at an abandoned U.S. Naval base in Alameda. To realize what had been previsualized for the complicated sequence, producers gave the production permission to build a quarter-mile piece of real highway in Alameda so that full control could be maintained for indefinite periods of time. Gaeta says that investment paid off because it would have been even more costly to build the full highway in CG. There are, according to Gaeta, “hundreds and hundreds of shots” in the sequence.

The sequence itself features a potpourri of CG cars, live-action stunts, real backgrounds, and CG backgrounds, typifying the project's sophisticated effects approach. In particular, the freeway sequence features a CG version of Agent Smith leaping from car to car. The sequence mixes-and-matches two complete CG shots, extensive live-action plates, CG cars, CG overpasses, and much more.

Likewise, the “burly brawl” mixes-and-matches freely before climaxing with an all-CG shot that epitomizes Gaeta's beloved “virtual cinematography” by replicating the real actors and location doing seemingly impossible things. The sequence includes hundreds of CG replicants of Weaving, his face placed onto the bodies of martial artists (all choreographed by Hong Kong legend Yuen Woo-Ping). As they attack Neo, a virtual camera flies in, around, above, and through the scene.

“The ‘burly brawl’ is three different things: some live action, some head and face replacements, and some pieces that are completely virtual,” says Cooper. “The martial arts moves in the CG shots, though, are still Woo-Ping's choreography. They mo-capped, at the most, nine guys at any one time, doing martial arts moves, and then built all that up. The cool thing is, you end up with something you can't possibly film in real life, yet it is still a vision of what the directors wanted in their photography — an extension of their photography. So while it is synthetic at some points, the idea is to achieve the vision of what they were trying to accomplish on set.”

The Producer's Perspective

Because the directors of the Matrix franchise — the brothers Wachowski — have elected not to discuss their work with the media, it's left to producer Joel Silver to explain this year's two sequels. The first thing Silver wants to make clear about Matrix Reloaded and Matrix Revolutions: the two films are not back-to-back sequels at all.

“They're one giant movie that we are cutting in half to show each half separately,” Silver explains. “We shot for 270 days in Australia and California, making not only this giant movie, but also shooting footage and creating effects for the videogame and the Matrix prequel animation [coming soon on DVD], as well as the website, the sound track, all that stuff. We spent well over $100 million, much of it on visual effects. The difference between these sequels and, say, when I produced the Die Hard sequels, is that most sequels, including Die Hard, get made because the first one is a box-office hit. In this case, this was a complete, huge story in the heads of the [Wachowski] brothers, and they agreed to do all this work, writing and directing the three films, if I would work to get Warner Brothers to commit to all three. These two films were always necessary to complete the story.”

From a producer's point of view, Silver makes no bones about why most of that story was produced in Australia: Runaway production is a necessary evil these days, in his view.

“It's just not financially feasible to shoot something of this scope in the U.S. right now,” he says. “It's not a coincidence we went to Australia. The fact is down there, the cost is less, and the government helps us with subsidies. It's that simple. We didn't want to go there, but the truth is people here in California don't seem to care. The aircraft industry is gone, and I guess they want us gone, too. Our state and national governments don't seem to care about keeping productions in the U.S., even though entertainment is the single biggest export this country has.”

In any case, the entire project has been a “freight train, and we've just managed to stay in front of it,” says Silver. “The key was a cohesive team — one editor, one visual effects supervisor, one costume designer, great crews in Australia, and all that. But it has definitely been the most complicated project I've ever worked on.”

And finally, why won't the Wachowski brothers talk publicly about what has, in essence, become their life's work?

“They don't want to explain it,” says Silver. “They want people to talk about it and figure it out for themselves. They figure that if they comment, that gives final resolution to many questions, and they don't want those questions resolved. I tried to get them to talk, but they wouldn't do it. That's the deal they made with me, so I said OK.”

DP's Double Duty

DP Bill Pope is evasive when asked if he would ever again agree to shoot two epic films simultaneously as he did for the two Matrix sequels.

“I'd rather not answer that,” he says. “But I will say this is easily the most grueling project I've ever done. It was spiritually and physically difficult.”

From a DP's point of view, the shoot Pope conducted in Australia and Alameda, Calif., over the course of 18 months (with a break in between locations) is a testament to the art of balancing preparation with flexibility. The frantic pace, however, was necessary for Matrix Reloaded and Matrix Revolutions to premiere this year.

“In a sense, we had plenty of prep, and in a sense, we had no prep at all,” he says. “The preproduction time was largely spent preparing to film two sequences in Alameda — the freeway sequence and the [‘burly brawl’] fight between Keanu Reeves and Hugo Weaving. Those two sequences alone took 65 days of first-unit work and almost the same number of days by the second unit. That came first, so our prep focused on that. Then, we had a month break to travel to Australia and set up. By the time we got there, I thought, ‘We are under-prepared for the rest of the shoot.’ In that sense, we needed a lot of day-to-day planning, so we set up a process to shoot and plan at the same time.”

That process came in the form of what Pope calls “a shadow government” that set things up for the first unit as production progressed.

“The shadow government had a grip, a gaffer, and a full production team from every relevant department that shadowed us,” he explains. “The first unit would talk with them at some point during the day, and then they would go off and prep the next thing we'd have to shoot. It was exhausting because I had to leave the set, watch dailies, and then go talk to them about lighting and things, but it was the only way to get it all done.”

Also important was finding the time to attend daily meetings to plan future shots, which, as Pope points out, might have been just a day or two away. “In my particular case,” he says, “my key grip, Ray Brown, and my gaffer, Reg Garside, got me through it. They would often prep for me while I was dealing with other things.”

Pope emphasizes that the job shooting effects plates was a big part of the load because there were more than 2,500 digital effects shots created for the two films. “Most of those shots had dozens of layers, so you might have 2,500 shots, but it probably took around 7,500 plates to create those shots,” he says. “It was pretty intense.”

The cinematography team also had to contend with maintaining the Matrix look pioneered in the first movie while at the same time moving beyond it.

“The first movie dictated the notion that there is the ‘real world’ and there is the ‘Matrix,’” he says. “That automatically divided the films into different colors, with the real world being cold and blue because it is a world with no sun, and the Matrix being green and sunless, because it is manufactured by a machine. That held true for these two movies — you use wider lenses for the Matrix scenes because you want to be further away from the actors. In the real world, you want to be closer because, there, they are real people. At the same time, we have new characters and locations and different themes in these two films. That means we present some new colors, and we light some scenes differently. There is also a difference between the two sequels. The second movie gets darker, and we move outdoors more. I can't give away the story, but things get ‘worse’ in the second movie, before they ever might get better later on, so the lighting and color schemes reflect that.

“Most of the two movies was shot on Kodak [Vision] 5279, except the effects plates, for which I mainly used Vistavision. The freeway scene, though, I used 5274. The reason I used a different stock there was for continuity — that sequence was shot over 40-some days, and I knew we'd have to scan the whole thing to digitally treat the sky and sunlight to match it all up since the weather changed during that period. I figured 5279 would give me a finer grain structure since we had to scan it and treat it anyway.”