National Science Foundation taps NYU Tandon with $5 Million grant to advance accessibility

A team at NYU Tandon School of Engineering will kick off the second phase of an ambitious research project that aims to transform navigation and accessibility for many of the 285 million people with blindness and low vision (pBLV) worldwide.

The National Science Foundation (NSF) awarded the project a three-year $5 million grant last month.

Led by John-Ross Rizzo — an associate professor in NYU Tandon’s Biomedical Engineering department, associate director of NYU WIRELESS, affiliated faculty at the NYU Tandon Center for Urban Science and Progress (CUSP) and associate professor in the Department of Rehabilitation Medicine at NYU Grossman School of Medicine — the team will continue developing VIS⁴ION (Visually Impaired Smart Service System for Spatial Intelligence and Onboard Navigation), a wearable technology platform designed to help pBLV better understand and navigate their local environments.

VIS⁴ION works by using miniaturized sensors including cameras, microphones, GPS, and motion sensors on wearable devices to collect data about the user's environment. Artificial Intelligence (AI) services, running locally within the platform itself and remotely in the cloud, process the sensor data to interpret the environment and tell the user where to walk, what to avoid and how to maneuver through hazards.

“People with blindness and low vision have unacceptably high unemployment rates, with some studies showing levels at about eighty percent,” said Rizzo, who is the project’s Principal Investigator (PI). “A critical obstacle to employment is commuting difficulties and navigation within the workplace itself. This project takes a fundamental step in solving this problem.  We believe, if successful, it can significantly improve pBLV’s quality of life and unlock their potential to contribute fully to the communities in which they live.”

In 2022, the NSF awarded the team a Phase One Convergence Accelerator grant to support its early VIS⁴ION work. These highly competitive grants fund research to address disability challenges. The team’s promising preliminary results prompted the NSF to advance the VIS⁴ION project to its phase two funding round.

In the first phase, the researchers successfully built a lightweight wearable VIS⁴ION backpack prototype with cameras, on-board processing, and audio and haptic feedback. They subsequently demonstrated the backpack could be used for successful and safe navigation through a variety of environments.

The team also developed VIS⁴ION mobile, a standalone mobile platform for use entirely on a smartphone. The signature microservice — offered on both the wearable and mobile platform — is UNav, a software that creates camera-based, infrastructure-free “digital twins” or 3D maps of complex indoor and outdoor environments, supporting wayfinding to close- or far-range destinations through audio and haptic user prompts.

In collaboration with New York City’s Metropolitan Transit Authority (MTA), the VIS⁴ION researchers also unveiled an early version of Commute Booster, a smartphone app that guides users to their destinations in subway stations by “reading” the directional signage it encounters. 

Moving forward, the team will continue to improve VIS⁴ION services while reducing the wearable’s size and weight. The goal is a commercially available wearable product, a breakthrough that user advocates have sought for years.

“NYU has the rare ability to bring faculty from across schools and departments to collaborate on the creation of technologies that help create greater accessibility,” said NYU Tandon Dean Jelena Kovačević.  “Engineering can even the playing field and it is research like this, led by JR Rizzo, that ensures that everyone is able to navigate both urban and rural communities.”

Vital to the work is the team’s collaborations with industry leaders, including Qualcomm, whose ongoing R&D efforts contribute broadly to the wearable technology ecosystem. Qualcomm is an affiliate of NYU WIRELESS, a 6G wireless research center at NYU Tandon. 

“Snapdragon platforms enable compelling new applications that require high performance compute, wireless and AI capabilities,” said Junyi Li, vice president, engineering at Qualcomm Technologies, Inc. “Qualcomm is looking forward to the potential collaboration with NYU in this NSF-awarded project to enable powerful wearable devices with advanced assistive technologies for people with disabilities.”

Dell Technologies, another NYU WIRELESS affiliate, will provide critical Dell Precision rack workstations to enable the cloud-based AI microservices for the wearable.

“Collaborating with NYU on this project is an incredible opportunity to use our technology to advance and transform accessibility,” said Charlie Walker, senior director and GM, Precision Workstations, Dell Technologies. “Using Dell Precision rack workstations equipped with powerful GPUs, NYU can run computing-intensive tasks, such as real-time object detection and vision-language models, to expand and improve the AI services on the wearable.”

The NYU Tandon senior personnel involved in the project reflect an integrated effort that crosses disciplines and engages multiple departments and Centers:

• Chen Feng (Co-PI), assistant professor Department of Mechanical and Aerospace Engineering (MAE) and in the Department of Civil and Urban Engineering (CUE), an associated faculty at CUSP and the director of the AI4CE lab, will lead all AI-based visual navigation (localization and mapping) development and evaluation work.
• Maurizio Porfiri (Co-PI), Institute Professor in Biomedical Engineering (BME) and MAE and Director of NYU Tandon’s CUSP, will lead optimization and evaluation of the haptic feedback belt. 
• Sundeep Rangan (Co-PI), professor of Electrical and Computer Engineering (ECE) and Associate Director of NYU WIRELESS, will lead wireless connectivity development.
• Yao Wang (Co-PI), professor of BME and ECE and a member of NYU WIRELESS, will lead optimization of the machine vision algorithms and video adaptation development.
• Yi Fang, associate professor of ECE at NYU Abu Dhabi and NYU Tandon, will direct the development of foundational vision and language models for the project, enhancing environmental interaction for persons with blindness and low vision.
• Marco Mezzavilla, research scientist in NYU WIRELESS, will lead the development of the test lab, direct the wearable software implementation, and coordinate connectivity. 

From NYU Grossman School of Medicine, team members include Mahya Beheshti, Research Scientist; Giles Hamilton-Fletcher, post-doctoral fellow; Todd Hudson, Research Assistant Professor, Department of Rehabilitation Medicine at Rusk Rehabilitation; and Joseph Rath, Assistant Professor of Rehabilitation Medicine at Rusk Rehabilitation. 

Tactile Navigation Tools (TNT), the  team’s start-up partner, will drive the project’s commercialization efforts, beginning with increasing the applications of the technology to improve accessibility and quality in NYC real estate for customers with and without disabilities.

TNT is led by its CEO Van Krishnamoorthy, a physician entrepreneur. 

The project’s nonprofit collaborators include Lighthouse Guild — whose Chief Research Officer William H. Seiple will help direct and develop protocols for all workplace studies — and VISIONS. Key government agencies including the MTA and NYC Department of Transportation are also involved. 

In addition to Qualcomm and Dell, corporate collaborators include Google, AT&T and TaggedWeb. 

Other people from NYU Tandon involved in the project include: Anbang Yang, MAE Ph.D. candidate, and the main software developer and maintainer for the vision-based navigation system UNav; Tommy Azzino, graduate research assistant at NYU WIRELESS; Junchi Feng, BME Ph.D. candidate;  Fabiana Ricci, BME Ph.D. candidate; and Jacky Yuan, ECE Ph.D. candidate.  

Ashish Bhatia, Clinical Associate Professor of Management & Entrepreneurship at NYU Stern School of Business will also contribute.

The VIS⁴ION project reflects NYU Tandon’s commitment to solving problems related to health, one of the seven interdisciplinary “Areas of Excellence” that define the School’s research and educational priorities.