A trio of IDM alumni share three Emmys for their work on the Super Bowl LVIII broadcast

What if your job involved coating a stadium in virtual slime or helping a loveable cartoon starfish give the play-by-play at a major football game in real-time? If you’re a graduate of NYU Tandon’s Integrated Design & Media (IDM) program — adept at augmented and virtual reality technologies, motion capture, and other emerging media skills — you might be called upon to do exactly that.

If you happened to be one of the 123.4 million viewers who tuned in to Super Bowl LVIII (a figure that made it the most-watched telecast in history), you saw the power of an IDM degree in action, thanks to a small group of alumni working for Silver Spoon, a Brooklyn-based animation studio. 

The game was played at Allegiant Stadium in Las Vegas, televised nationally by CBS, streamed on Paramount+, broadcast on the kid-friendly network Nickelodeon, and shown on the Spanish-language network Univision, marking just the second simulcast in Super Bowl history. Silver Spoon had been tapped to work on Nickelodeon’s offering, Super Bowl LVIII: Live from Bikini Bottom, which featured an augmented reality SpongeBob SquarePants doing gametime analysis and slime cannons going off over the endzone when a player made a touchdown.

Silver Spoon (now owned by AE Live) deployed 18 tracked cameras; dozens of graphics built in an Unreal Engine, a popular 3D computer graphics game engine; and live motion capture that allowed SpongeBob and his friends to interact seamlessly with each other and with actual human commentators. 

Their efforts won them three Sports Emmys, and Melissa Canavan, Christian Hadjigeorgiou, and Isabella Heron — a trio of IDM-trained Unreal Engine artists, motion-capture experts, and graphic designers — shared the prize.

Below, we catch up with Melissa and Christian to hear how NYU Tandon helped them attain red-carpet success and where their skills might take them next.

Melissa Canavan (B.S. ‘21)

Q: Besides Super Bowl LVIII: Live from Bikini Bottom, what type of projects have you worked on since graduating?

A: My first big job was working on season 3 of The Mandalorian, which is the first live-action series in the Star Wars franchise. I got to be on site in LA, so that was exciting. I also worked on Toy Story Funday Football. For that broadcast, we handled the live performance/motion capture of the commentators Drew Carter and Booger McFarland, as well as the virtual environment that they were commentating from. That project also won an Emmy.

Q: How instrumental was NYU Tandon in helping you forge your career?

A: It’s no exaggeration to say I wouldn’t be where I am today without it. Todd Bryant taught me Unreal Engine, and I couldn’t do my work without it. I learned that and so much more in the IDM program. I minored in Disability Studies and worked with Tech Kids Unlimited, which is a great program that teaches neurodiverse students to create technology rather than being passive consumers of it, so I feel that shaped me as someone who can be called on to train others with the tools and techniques they need to be a success in this field. Recently I had the chance to return to Tandon during Design Week at the MakerSpace. I helped lead a workshop on Unreal Engine, so that was a real full-circle moment for me. 

Q: Where do you store your Emmys?

A: First of all, when you win an Emmy as a team, each individual doesn’t automatically receive a statuette; they have to be purchased. ESPN has announced that they’re going to purchase one for each of us, so I guess we’ll see. I also happen to live in a tiny Brooklyn apartment, so that’s a factor. It’s enough, though, to just know I was part of such a great team.

Christian Hadjigeorgiou (M.S. ‘22) 

Q: What led you to NYU Tandon?

A: I earned an undergraduate degree in psychology and worked for a few years, but I really wanted to do something creative. I had loved video games and music my whole life, and after reading about the IDM master’s program, I realized that would provide a path. I loved NYU’s diversity, and my classmates turned out to be the coolest, most creative people I know.

Q: How did you get involved with Silver Spoon?

A: Todd Bryant recommended that I apply for an internship. Melissa was working there full-time at that point, and she really taught me a lot. I ended up working on AR effects for a League of Legends event.

Q: How did you contribute to the SpongeBob simulcast?

A: I was part of the team that executed body, face, and finger motion capture for SpongeBob and Patrick (minus the fingers for Patrick), using Unreal Engine, Movella Entertainment | Xsens, and StretchSense. It had its challenges!  Before this event, the voice actors have never had to physically embody their characters. It was entirely new to them. Many times during rehearsal, the actor’s natural physical movements wouldn’t reflect how their characters in the show would normally move. So at times we had to help prompt them on where their arms should rest, and the general positioning of their bodies. For example, I knew that the character of Sandy Cheeks can be more physically expressive with her body and her arms than the other characters, so we advised the actress about that. It was actually very helpful to have this vast knowledge of SpongeBob I’ve been building since I was about five years old.

It was definitely a highlight of my professional life, especially since I got to meet actors who were — literally — the voices of my childhood. It was surreal to see them and interact with them in real life. They’re also extremely kind and hilarious and easy to work with. They had us all laughing pretty much the entire time and had an amazing attitude even though we had to subject them to not-the-most-comfortable skin-tight suits and bulky helmets for hours at a time. And they were also super-stoked to see themselves fully controlling the characters they’ve been playing for decades. It was the first time they had experienced that.

Q: That must have been gratifying for them as actors; what stands out to you as a technologist? 

A: The motion capture process we used was pretty impressive. You start with calibrating the three different systems (body, face, and gloves). Typical systems use optical tracking for mocap (camera systems), but because we had such a tiny space to work in, we actually used an inertial motion capture system that uses accelerometers, gyroscopes, and magnetometers at each of its sensors along the body. Then the software processes this data to understand where each body part is at any given time and creates a skeleton for you to see. Then our software maps that moving skeleton data onto a (rigged) 3D model of our character, and combined with the face and finger data we’re also getting, we end up with a fully animated character. For the face data, we used Unreal Engine Live Link, which connects to an app on the iPhone, so we actually had helmets with a pole that attached the iPhone a foot and a half from the actors’ faces that captured data from the expressions they were making. And for fingers for SpongeBob, we used Stretchsense gloves that worked similarly to the mocap suits.

The envelope, please ... 

While you may not think of an engineering school as a hotbed of Emmy Award-winning talent, Canavan, Hadjigeorgiou, and Heron are far from the only members of the Tandon community to have that brush with glamor. Media executive Stephanie Mitchko, who graduated in 1987 with a degree in electrical engineering took home an Emmy for Best Interactive Television Platform during her tenure with Cablevision, and Professor Claudio Silva was honored for his development of a visual analytics tool used in the broadcast of Major League Baseball games. And like Canavan, Hadjigeorgiou, and Heron, Mahe Dewan (B.S. ‘17) graduated with enough skill using Unreal Engine to garner an Emmy of his own, thanks to the uncannily realistic depiction of Washington, D.C., against which NBC displayed eye-catching data visualizations during the 2022 midterm elections.