Two nonverbal actors star in a new opera — with an assist from AI

A team from NYU Tandon's Ability Project, including Associate Professor Luke DuBois, developed novel AI voice technology for composer Paola Prestini's opera "Sensorium Ex," which features a non-verbal character. Using neural vocoders and gesture sensors, the system creates personalized synthetic voices from natural sounds, allowing real-time emotional expression. DuBois described the breakthrough: "We had this pretty profound eureka moment... because it no longer sounds like their off-the-shelf commercial speech synthesizer. It sounds like they're talking." The open-source technology aims to benefit the broader non-verbal community.