Visiting Research Scholar Spotlight: Dr. Ellen Pearlman

What do you think of when you hear the word opera? Soaring arias? Dramatic storytelling? Lavish costumes? Chances are that neither artificial intelligence, electromyography (EMG) nor brain computer interfaces (BCI) will immediately spring to mind.

Dr. Ellen Pearlman, a Visiting Research Scholar in NYU Tandon’s Integrated Design & Media (IDM) program, creates works that challenge traditional perceptions.

In 2011, Pearlman earned a master’s degree from the University of Calgary under the National Canadian Research Chair in Telematic Art, while simultaneously attending an artist residency at Videotage in Hong Kong. She established the world’s first summer institute in Telematic Art between Canada and Hong Kong using a professional motion capture studio with high speed fiber optic telecommunications networks for creative projects. This resulted in I Move In Decades, which data mapped dancers from the Hong Kong Center for Performing Arts in the motion capture studio so their choreography changed video effects playing thousands of miles away in Canada. In 2014 she returned to Asia for a doctoral program at Hong Kong City University’s School of Creative Media, which provided a fertile environment for developing Noor, which she created in 2016 and which has been widely described as the world’s first interactive immersive “brainwave opera.”

The piece — which premiered at the International Society of Electronic Arts (ISEA) in Hong Kong, in a 360-degree theater using 15-foot tall projector screens — tells the story of Noor Inayat Khan (also known as Nora Baker), a covert British agent who sent vital wireless transmissions from Nazi-occupied France to London during World War II, before being captured and dying in Dachau.

“I wanted to explore the question of whether there was a place in human consciousness where surveillance cannot reach,” Pearlman explains. For the piece, she hooked a performer up to an electroencephalogram (EEG) headset that measures electrical activity in the brain; as the performer’s emotional state changed, their brainwaves were used to trigger video, sound, a libretto and a series of colored bubbles: red when the performer felt frustrated, yellow signifying excitement, pink for interest; and blueish-green for calm. There were no seats in the theater, and the performer wandered amongst the audience, setting up a real-time human computer feedback loop.

When GPT-2, an early large language model (LLM), was released by OpenAI in February 2019, Pearlman said, “the ability to mimic human dialogue and produce fake, but believable interactions between humans and computer-based agents is fully upon us.” In March 2019 she began harnessing that ability and created AIBO, (Artificial Intelligence Brainwave Opera), the world’s first emotionally intelligent artificial intelligence brainwave opera. The project received additional support from the Estonian Academy of Art, the Estonian Academy of Music, and the Human Computer Interaction Lab of Tallinn University, premiering in early February 2020 in Estonia with a second showcase in Paris two weeks before COVID-19 lockdown.

AIBO  was shown well before ChatGPT made AI accessible to lay users, but Pearlman (who in her earlier years had funded her art projects with technical jobs) built a feedback loop between EEG signals from her performer’s brain with a built-from-scratch AI entity running in the computing cloud — emotionally intelligent, but twisted. Her purpose was to interrogate the disturbing idea that an AI could indeed be fascist: AIBO’s human character was modeled on Eva von Braun, and the AI character, who ran in real time in the cloud, was modeled on Adolf Hitler.  

Among her most recent works is the Lumen Prize short-listed Language Is Leaving Me: An AI Cinematic Opera Of The Skin, which draws upon her experience with immersive technologies, AI, and biometrics; this time, she placed EMG sensors on an audience volunteer’s face and instructed them to react to a multilingual film in order to explore the question of whether an AI entity can have epigenetic (coded into DNA and inherited) trauma, such as that experienced in cultures forced into diaspora.The volunteer’s reactions drove the sonic landscape affecting the rest of the audience.

Pearlman’s development in AI cinema, (which did not exist at the time) began in early 2021 and was supported by a 2022/23 Fulbright Scholar Award to the Department of Mathematics and Informatics at the University of Warsaw, Poland. It had its world premier at the Copernicus Science Center, one of the top three art/science centers in Europe on October 7, 2023.

While Pearlman has been celebrated for her biometric operas, she is also well known in the emerging-arts world as the founder of the cleverly named Art-A-Hack™, an initiative that matches potential partners from various disciplines, including art, technology, hardware and software development, immersive environments, music, theater, animation, social justice and interactivity, and provides them with the resources to collaborate on rapid prototyping projects.  

Pearlman’s laurels are too numerous to list in their entirety, but in addition to recognition from the Lumen Prize committee, they have included a Leonardo LABS Abstract Award for best doctoral thesis globally, a residency at Germany’s Mir Lab, a research fellowship at MIT, multiple Vermont Studio Center Special President's Fellowships, an Asian Cultural Council award, a Vertigo S.T.ARTS residency (an E.U. initiative that funds work at the intersection of science, technology, and the arts), a Harvestworks New Works grant and a total of four Fulbright Awards that have taken her around the globe. The most recent Fulbright of the four awards, given in 2025, allows her to serve as a Research Team Leader at the Center for Environmental and Technology Ethics-Prague (CETE-P) as part of a Horizon Europe (EU) grant, working on rapid prototyping methodologies in AI art and technologies in collaboration with FAMU, the number one filmmaking academy in the Czech Republic.

Here in Brooklyn, Pearlman, a native New Yorker, is working with the powerful, high-speed NVIDIA computers at Tandon @ The Yard on a proof-of-concept process that uses AI and biometric devices to evolve audio and motion-reactive 3D images from earlier-stages as drawings, photographs, and films. “AI is moving at an astonishing pace,” she says, “and engineers like those responsible for NVIDIA chips and OpenAI are now developing the technologies I need to keep innovating in my art.”

Update September 19, 2025

Witnessing what its members call “a veritable boom of fully or partially AI-generated art,” the German Federal Cultural Foundation recently launched a €3.68 million initiative to explore the artistic potential of AI technologies, reflect on their possible social impact, and gain a deeper understanding of the subject.

To that end, they provided funding to a select cohort of creative organizations with both artistic expertise and technical know-how, charging them with producing works that can serve as reference points for future engagement with AI.

Among those groups is the digital lab of the Musiktheater im Revier (MiR.LAB), in Gelsenkirchen, Germany, which is working with Pearlman to create a new production, “The System Cannot Fail.” Awarded the sum of € 240,000, the piece will delve into how memory and identity unfold in bodies, code, and collective imagination, and how an AI system can shape this process in real-time and make it visible.

The premiere is scheduled for late 2027, but before then, the public will have the opportunity to learn more via workshops, work-in-progress performances, and public discussions.