“Bodies in Motion” Melds Virtual Reality and Motion Capture Engineering with Art

In NYU Tandon’s Integrated Digital Media (IDM) program, engineering students are getting creative with cutting-edge motion capture technology, thanks to a class that pairs Tandon students with students of NYU Tisch School of the Arts’ Interactive Telecommunications Program (ITP). 

The fall semester course, Bodies in Motion, taught by Todd Bryant, a postdoctoral researcher at ITP and adjunct professor at both IDM and ITP, and Kat Sullivan, an adjunct professor at IDM, held its final showcase on December 10, at the NYU Media and Games Network (MAGNET) at NYU Tandon. Dozens of people braved the chilly weather on Sunday evening to watch and interact with five installations that merged technology with live performance to question our modern-day surveillance state, criticize technological dependence, invite us to a world of monsters and magic, explore memories, and examine digital communication.

Bryant pointed out that the evening was about more than putting on a show. “The idea of class is that it’s both a showcase and a fantastic opportunity for students to get great documentation so that they have the materials to submit their works to other venues outside of school,” said Bryant, who added that to produce their projects students got full run of the motion capture studio at MAGNET.

“In the first half of the semester, everyone was encouraged to work solo to hone their basic skills. Prior to mid-terms, when we brought in eight guest critics to critique student proposals, [Sullivan] and I divided the class into groups so that they could brainstorm and present as their final group,” he said. “We gave them huge scope to realize their vision without limitations at the beginning, and then reeled it in with what was feasible with the equipment we had.”

To bring their visions to life the designers employed such technology as animation rendering tool Unreal Engine, motion capture equipment Optitrack, digital signal processing interface Max/MSP, and Arduino open-source microcontrollers, which use sensory data to control lights, motors, and other actuators.

The Projects

Elsewhere in the Woods, by Christina Elizabeth Hall, Lin Yang, Isabella Heron, and Larysa Hrabowych, was a live action installation in miniatures about a teenager lost in the woods and seeking a way home. Inspired by Theatre Optique de Emile Reynaud, a 19th Century animated moving picture system, the piece employed actors in motion capture gear who embodied the lost girl and the characters she meets. As the actors performed in the motion capture studio, the audience huddled around a black box display watched her adventure, interacting to assist her when she pleaded for help, thanks to various interactive devices.

A screenshot of the Elsewhere in the Woods trailer

Isabella Heron, an IDM sophomore who helped create the project’s video display and animations, explained that the box projector was designed to add visual depth. “We used a sheet of acrylic to reflect what was happening on a monitor installed out of view, so what you saw where characters that seem to float.”

Project Memory, an exploration of life moments revisited, created by Alex Fast, Ife Lawal, and Shuang Liang, and performed by Elizabeth White, required the audience to stand in the temporary motion capture space, between three wall-sized projection screens. As Fast, Lawal and Liang alternatively recited stories from their past, the videos changed to reflect the narratives.

Combining live dance with audience interaction, Vessel was a commentary on digital communication overload. The project, by Casey Judge, Manning Qu, Spencer Capiello and Yuhan Zhang, featured a mobile app developed by Capiello, a senior in the IDM program at NYU Tandon who is also partnering with Bryant and Sullivan on a motion capture-based VR platform for stroke victims called Relevant Motion. Once audience members had downloaded Capiello’s chat application by capturing a QR code, they used the app to answer a series of personal questions such as “What do you value most in others?” and “What is too serious to joke about?” 

A dancer in a motion-capture outfit — and her animated avatar, projected onto a wall-sized screen — performed movement improvisation reflecting the audience’s texted responses.

Time Machine, by Vhalerie Lee, Jing Huang, Alexia Kyriakopoulos and Luqian Chen, is an immersive installation featuring a video projection of Times Square in the past, present and future to explore the idea that technology is leading to a convergence of people and machines. The audience was ushered into the studio in small groups, and members were chosen to place glowing orbs on a pedestal, launching different elements of the experience.

“We fabricated a podium that housed circuitry that was able to recognize each crystal ball by color,” said Lee, who has now completed her graduate program at IDM. “Circuits sent wireless data to a computer that then loaded the virtual environment.” A dancer delivered the third orb, representing the future. When it was placed on the podium a futuristic New York materialized, with the performer represented by a phalanx of robotic dancers moving in unison.

While students at Tandon, Lee and fellow IDM graduate student Subigya Basnet created, among other projects, a Game of Thrones mobile AR app that brings to life the fictive world’s stories and settings.

Since the focus for Privacy Statement was individual space and boundaries, only one person at a time could experience the piece, which ran continuously throughout the evening. The 360-motion capture installation, by Valentina Escotet, Teresa Yang, Jixuan Sun and Dimos Markopoulos, delved into the fundamentals of privacy and personal space: participants were “trapped” inside a virtual space that initially promoted a sense of comfort and ease, until it faded into a mass of bodies and movement that created a feeling of objectification and an invasion from the virtual spectators inside the world.