Research News

NYU Tandon School of Engineering researchers developed a new artificial intelligence technique to change a person’s apparent age in images while maintaining their unique identifying features, a significant step forward from standard AI models that can make people look younger or older but fail to retain their individual biometric identifiers.

In a paper published in the proceedings of the IEEE International Joint Conference on Biometrics (IJCB), Sudipta Banerjee, the paper’s first author and a research assistant professor in the Computer Science and Engineering (CSE) Department, and colleagues trained a type of generative AI model – a latent diffusion model – to “know” how to perform identity-retaining age transformation.

To do this, Banerjee – working with CSE PhD candidate Govind Mittal and PhD graduate Ameya Joshi, under the guidance of Chinmay Hegde, CSE associate professor and Nasir Memon, CSE professor – overcame a typical challenge in this type of work, namely assembling a large set of training data consisting of images that show individual people over many years.

Instead, the team trained the model with a small set of images of an individual, along with a separate set of images with captions indicating the age category of the person represented: child, teenager, young adult, middle-aged, elderly, or old. This set included images of celebrities captured throughout their lives.

The model learned the biometric characteristics that identified individuals from the first set. The age-captioned images taught the model the relationship between images and age. The trained model could then be used to simulate aging or de-aging by specifying a target age using a text prompt.

Researchers employed a method called "DreamBooth" for editing human face images by gradually modifying them using a combination of neural network components. The method involves adding and removing noise – random variations or disturbances – to images while considering the underlying data distribution.

The approach utilizes text prompts and class labels to guide the image generation process, focusing on maintaining identity-specific details and overall image quality. Various loss functions are employed to fine-tune the neural network model, and the method's effectiveness is demonstrated through experiments on generating human face images with age-related changes and contextual variations.

The researchers tested their method against other existing age-modification methods, by having 26 volunteers match the generated image with an actual image of that person, and with ArcFace, a facial recognition algorithm. They found their method outperformed other methods, with a decrease of up to 44% in the rate of incorrect rejections.

UPDATE March 4, 2024: The data set that Faghih’s lab collected for this research is now available to the global research community on the PhysioNet platform. This dataset is unique, offering real-world insights into how common pleasures affect our physiological responses and cognitive performance.

The potential of this dataset is vast. It opens new avenues for research into the influence of everyday experiences on cognitive performance, potentially leading to smarter work environments or personalized life-enhancing strategies. Imagine tailoring your work environment with specific sounds or scents to boost productivity and creativity. By analyzing this dataset, researchers can discover patterns and connections previously unseen. This could lead to breakthroughs in understanding how to harness everyday experiences to enhance cognitive abilities. Ultimately, this research could pave the way for innovative applications in workplace productivity enhancement and educational method improvement.

“This dataset is more than a collection of data points; it is a window into the intricate relationship between daily pleasures and our brain's performance,” says Fekri Azgomi, Faghih’s former PhD student who collected this data. “As our lab, the Computational Medicine Laboratory, shares this dataset with the world, we are excited about the endless possibilities it holds for advancing our understanding of the human mind and enhancing everyday life.”

Original story below.

Listening to music and drinking coffee are the sorts of everyday pleasures that can impact a person’s brain activity in ways that improve cognitive performance, including in tasks requiring concentration and memory.

That’s a finding of a new NYU Tandon School of Engineering study involving MINDWATCH, a groundbreaking brain-monitoring technology.

Developed over the past six years by NYU Tandon's Biomedical Engineering Associate Professor Rose Faghih, MINDWATCH is an algorithm that analyzes a person's brain activity from data collected via any wearable device that can monitor electrodermal activity (EDA). This activity reflects changes in electrical conductance triggered by emotional stress, linked to sweat responses.

In this recent MINDWATCH study, published in Nature Scientific Reports, subjects wearing skin-monitoring wristbands and brain monitoring headbands completed cognitive tests while listening to music, drinking coffee and sniffing perfumes reflecting their individual preferences. They also completed those tests without any of those stimulants.

The MINDWATCH algorithm revealed that music and coffee measurably altered subjects’ brain arousal, essentially putting them in a physiological “state of mind” that could modulate their performance in the working memory tasks they were performing.

Specifically, MINDWATCH determined the stimulants triggered increased “beta band” brain wave activity, a state associated with peak cognitive performance. Perfume had a modest positive effect as well, suggesting the need for further study.

“The pandemic has impacted the mental well-being of many people across the globe and now more than ever, there is a need to seamlessly monitor the negative impact of everyday stressors on one's cognitive function,” said Faghih. “Right now MINDWATCH is still under development, but our eventual goal is that it will contribute to technology that could allow any person to monitor his or her own brain cognitive arousal in real time, detecting moments of acute stress or cognitive disengagement, for example. At those times, MINDWATCH could ‘nudge’ a person towards simple and safe interventions — perhaps listening to music — so they could get themselves into a brain state in which they feel better and perform job or school tasks more successfully.”

The specific cognitive test used in this study — a working memory task, called the n-back test — involves presenting a sequence of stimuli (in this case, images or sounds) one by one and asking the subject to indicate whether the current stimulus matches the one presented "n" items back in the sequence. This study employed a 1-back test — the participant responded "yes" when the current stimulus is the same as the one presented one item back — and a more challenging 3-back test, asking the same for three items back.

Researchers tested three types of music - energetic and relaxing music familiar to the subject, as well as novel AI-generated music that reflected the subject’s tastes. Consistent with prior MINDWATCH research, familiar energetic music delivered bigger performance gains — as measured by reaction times and correct answers — than relaxing music. While AI-generated music produced the biggest gains among all three, further research is needed to confirm those results.

Drinking coffee led to notable but less-pronounced performance gains than music, and perfume had the most modest gains.

Performance gains under all stimulations tended to be higher on the 3-back tests, suggesting interventions may have the most profound effect when “cognitive load” is higher.

Ongoing experimentation by the MINDWATCH team will confirm the efficacy of the technology’s ability to monitor brain activity consistently, and the general success of various interventions in modulating that brain activity. Determining a category of generally successful interventions does not mean that any individual person will find it works for them.

The research was performed as a part of Faghih’s National Science Foundation CAREER award on the Multimodal Intelligent Noninvasive brain state Decoder for Wearable AdapTive Closed-loop arcHitectures (MINDWATCH) project. The study's diverse dataset is available to researchers, allowing additional research on the use of the safe interventions in this study to modulate brain cognitive states.

Faghih served as the senior author for this paper. Its first author is Hamid Fekri Azgomi, who earned his Ph.D. under Faghih and is now a postdoctoral scholar of neurological surgery at the University of California San Francisco School of Medicine.

Fekri Azgomi, H., F. Branco, L.R., Amin, M.R. et al. Regulation of brain cognitive states through auditory, gustatory, and olfactory stimulation with wearable monitoring. Sci Rep 13, 12399 (2023). https://doi.org/10.1038/s41598-023-37829-z

A new trip-planning app has shown encouraging results in improving navigation inside subway stations, according to a study published in IEEE Journal of Translational Engineering in Health and Medicine, promising the possibility of easier commutes for people who are blind and low-vision.

Designed by researchers at NYU Tandon School of Engineering and NYU Grossman School of Medicine, Commute Booster routes public-transportation users through the “middle mile” — the part of a journey inside subway stations or other similar transit hubs — in addition to the “first” and “last” miles that bring travelers to and from those hubs.

“The ‘middle mile’ often involves negotiating a complex network of underground corridors, ticket booths and subway platforms. It can be treacherous for people who cannot rely on sight,” said John-Ross Rizzo, MD, who led the research team that includes advisors from New York City’s Metropolitan Transit Authority (MTA). Rizzo is an associate professor in NYU Tandon’s Biomedical Engineering department and is on the faculty of NYU Grossman. “Most GPS-enabled navigation apps address ‘first’ and ‘last’ miles only, so they fall short of meeting the needs of blind or low-vision commuters. Commute Booster is meant to fill that gap.”

Subway signs are typically graphical or text-based, creating challenges for the visually impaired to recognize from distances and reducing their ability to be autonomous in unfamiliar environments.

Commute Booster automatically figures out what signs a traveler will encounter along the way to a specific subway platform. Then, it uses a smartphone’s camera to recognize and interpret signs posted inside transit hubs, ignoring irrelevant ones and prompting users to follow relevant ones only.

In the recent study, researchers tested Commute Booster’s interpretation of signage from three New York City subway stations — Jay Street-Metrotech, Dekalb Avenue and Canal Street — that a traveler would encounter on a specific journey. The app proved 97 percent accurate in identifying signs relevant to reach the intended destination.  

Testing inside those three subway stations also revealed that Commute Booster could “read” signs from distances and at angles that reflect expected physical positioning of travelers.

The Commute Booster system relies on two technological components. The first, general transit feed specification (GTFS), is a standardized way for public transportation agencies to share their transit data with developers and third-party applications. The second, optical character recognition (OCR), is technology that can translate images of text into actual editable text.

The GTFS dataset contains descriptions for locations and pathways within each subway station. Commute Booster’s algorithm uses this information to generate a comprehensive list of wayfinding signage within subway stations that users would encounter during their intended journey. The OCR functionality reads all texts presented to users in their immediate surroundings. Commute Booster’s algorithm can identify relevant navigation signs and locate the position of signs in the immediate environments. By integrating these two components, Commute Booster provides real-time feedback to users regarding the presence or absence of relevant navigation signs within the field of view of their phone camera during their journey.

Researchers plan to conduct a human subject study of Commute Booster in the near future. The app could be available for public use in the near term.

Rizzo, who was named to MTA’s board in June 2023, has a long track record of research that applies engineering solutions to challenges faced by people with disabilities, particularly those with visual disability .

In addition to Rizzo, the team involved in the Commute Booster study are NYU Tandon PhD candidate Junchi Feng; Physician-Scientist at NYU Langone’s Rusk Rehabilitation Mahya Beheshti; MTA Senior Innovation Strategist Mira Philipson; MTA Senior Accessibility Officer Yuvraj Ramsaywack; and NYU Tandon Institute Professor Maurizio Porfiri.

This research was supported by the National Science Foundation, the National Eye Institute and Fogarty International Center, as well as by the U.S. Department of Defense.

Researchers at NYU Tandon School of Engineering’s C2SMART Center have developed an algorithm to plan the most efficient routes for modular vehicle (MV) fleets — specially-designed vehicles that attach and detach from one another as they move people around cities — removing a significant obstacle to making this type of transportation system a reality.

In a paper published in Transportation Research Part C: Emerging Technologies, the researchers employ a mathematical model called MILP (Mixed Integer Linear Programming) to optimize the service time for the passengers and the travel cost for the vehicles in an MV system. The model factors in passenger pickups and deliveries, en-route transfers, and variable capacity of the MVs to identify the best routes and schedules for the attachments and separations of the vehicles.

Conventional mass transit and demand-responsive transportation systems can face challenges accommodating fluctuations in traveler demand, leading to long travel times, energy inefficiencies, traffic congestion and financial waste.  

Low-capacity vehicles like vans may be slow and overcrowded in peak times. High-capacity vehicles like buses may be largely unoccupied when demand is low. On-demand services like paratransit often deliver only one passenger at a time, making them expensive to operate.

MVs offer a flexible and efficient alternative. The independent vehicles in MV fleets can connect while in motion, creating platoons that travel as one unit until the vehicles detach. According to research lead Joseph Chow, Institute Associate Professor in the Department of Civil & Urban Engineering and the Deputy Director of C2SMART, MVs can move people faster, with less energy consumption and operational expenses than many conventional systems.

“MVs offer a promising alternative to move people more efficiently in certain situations,” said Chow, who collaborated on the research with NYU Tandon Ph.D. student Zhexi Fu. “Imagine, for instance, employees at the same company. The individual vehicles could pick up people who live within similar enclaves, and join together in a platoon to deliver the entire group to its workplace. MVs also have significant potential to improve on-demand transportation that delivers people door-to-door, including those that serve people with disabilities.”

Currently, no city has an MV system in use, although Next Transportation Systems is piloting a MV test in Dubai now. According to Chow, the inability to track and route MV fleets has been a significant roadblock to potential real-world adoption. To build its routing model, the C2SMART team used the Anaheim network, a traffic simulation of Anaheim, California.

The research on MV routing is the latest in a long series of studies Chow has conducted around urban mobility. Among his previous studies include examinations of Dial-a-Ride programs, e-scooter usage, and urban bus networks. Chow’s new research also advances the mission of C2SMART, a U.S. Department of Transportation (US DOT) Tier 1 University Transportation Center (UTC) designated to address the US DOT priority area of Congestion Reduction.

A new paper from researchers at NYU Tandon School of Engineering and the NYU Grossman School of Medicine explores how interdisciplinary programs connecting medical and engineering education may foster innovation and prepare students in both disciplines for more successful careers.

The paper, published in the Technology and Innovation journal of the National Academy of Inventors, describes initiatives at NYU as a case study, along with similar programs at Johns Hopkins University, Stanford University, Harvard and Massachusetts Institute of Technology.

NYU Tandon and Grossman have partnered on educational programs for about a decade. But for most of this time, the skill-sharing only went in one direction, explained lead author John-Ross Rizzo, a rehabilitation medicine specialist and professor at both schools. “It dawned on us that we spend a ton of time bringing engineers to the medical school, but almost zero time trying to get our doctors immersed in the engineering world,” Rizzo said.

By bringing medical students to the engineering field, as NYU has done in recent years, educators can enable a shared understanding of engineering concepts that contributes to more effective problem-solving, Rizzo and his colleagues argue. Clinicians and engineers are more capable of collaboration if they speak each other’s languages; new innovations that result from these partnerships may be better set up for long-term success.

Rizzo compared this interdisciplinary learning to earning belts in martial arts. A medical student might not become a “black belt in computer science,” but might learn enough for a “yellow belt” — a lower level of understanding, but enough to enable collaboration with the true experts. “We’re creating a smarter generation of students,” Rizzo said.

One way NYU students may gain this expertise is through participation in the NYU HealthTech Transformer Challenge, which pairs engineers and clinicians to work on “healthcare’s most pressing problems.” Finalists from the program have won funding from NYU and other sources to pursue their ideas at new startups. Other challenges and grant-funded research projects at NYU Tandon and Grossman have allowed graduate students to receive co-advising from engineering and medical professors.

The researchers also discussed barriers to setting up these interdisciplinary programs. Early initiatives may require extensive effort, including dedicated advocacy to bring different school administrators on board. It may be especially tough to convince medical school leaders to devote student time to engineering work outside their typical course load. Part of the challenge is a lack of data: while NYU and similar programs have produced some clear success stories among individual students and startups, universities are not tracking their results in a comprehensive manner.

In the new paper, Rizzo and colleagues share lessons from NYU’s leadership in this interdisciplinary space and from programs at other institutions. The findings may provide inspiration for more universities to consider connecting medicine and engineering education. “I think this is a trend we’ll hear more about over the next decade,” Rizzo said.

LinkedIn Recruiter — a search tool used by professional job recruiters to find candidates for open positions — would function better if recruiters knew exactly how LinkedIn generates its search query responses, possible through a framework called “contextual transparency.”

That is what a team of researchers led by NYU Tandon’s Mona Sloane, a Senior Research Scientist at the NYU Center for Responsible AI and a Research Assistant Professor in the Technology, Culture and Society Department, advance in a provocative new study published in Nature Machine Intelligence.

The study is a collaboration with Julia Stoyanovich, Institute Associate Professor of Computer Science and Engineering, Associate Professor of Data Science, and Director of the Center for Responsible AI at New York University, as well as Ian René Solano-Kamaiko, Ph.D. student at Cornell Tech; Aritra Dasgupta, Assistant Professor of Data Science at New Jersey Institute of Technology; and Jun Yuan, Ph.D. Candidate at New Jersey Institute of Technology.  

It introduces the concept of contextual transparency, essentially a “nutritional label” that would accompany results delivered by any Automated Decision System (ADS), a computer system or machine that uses algorithms, data, and rules to make decisions without human intervention. The label would lay bare the explicit and hidden criteria — the ingredients and the recipe — within the algorithms or other technological processes the ADS uses in specific situations.

LinkedIn Recruiter is a real-world ADS example — it “decides” which candidates best fit the criteria the recruiter wants — but different professions use ADS tools in different ways. The researchers propose a flexible model of building contextual transparency — the nutritional label — so it is highly specific to the context. To do this, they recommend three “contextual transparency principles” (CTP) as the basis for building contextual transparency, each of which relies on an approach related to an academic discipline.

• CTP 1: Social Science for Stakeholder Specificity: This aims to identify the professionals who rely on a particular ADS system, how exactly they use it, and what information they need to know about the system to do their jobs better. This can be accomplished through surveys or interviews.
  
• CTP 2: Engineering for ADS Specificity: This aims to understand the technical context of the ADS used by the relevant stakeholders. Different types of ADS operate with different assumptions, mechanisms and technical constraints. This principle requires an understanding of both the input, the data being used in decision-making, and the output, how the decision is being delivered back.
  
• CTP 3: Design for Transparency- and Outcome-Specificity: This aims to understand the link between process transparency and the specific outcomes the ADS system would ideally deliver. In recruiting, for example, the outcome could be a more diverse pool of candidates facilitated by an explainable ranking model

Researchers looked at how contextual transparency would work with LinkedIn Recruiter, in which recruiters use Boolean searches — AND, OR, NOT written queries — to receive ranked results. Researchers found that recruiters do not blindly trust ADS-derived rankings and typically double-check ranking outputs for accuracy, oftentimes going back and tweaking keywords. Recruiters told researchers that the lack of ADS transparency challenges efforts to recruit for diversity.

To address the transparency needs of recruiters, researchers suggest that the nutritional label of contextual transparency include passive and active factors. Passive factors comprise information that is relevant to the general functioning of the ADS and the professional practice of recruiting in general, while active factors comprise information that is specific to the Boolean search string and therefore changes.

The nutritional label would be inserted into the typical workflow of LinkedIn Recruiter users, providing them information that would allow them to both assess the degree to which the ranked results satisfy the intent of their original search, and to refine the Boolean search string accordingly to generate better results.

To evaluate whether this ADS transparency intervention did achieve the change that can reasonably be expected, researchers suggest using stakeholder interviews about potential change in use and perception of ADS alongside participant diaries documenting professional practice and A/B testing (if possible).

Contextual transparency is an approach that can be used for AI transparency requirements that are mandated in new and forthcoming AI regulation in the US and Europe, such as the NYC Local Law 144 of 2021 or the EU AI Act.

Wearable monitoring is likely to play a key role in the future of healthcare. In many cases, wearable devices may monitor our physiological signals that can indicate mental states, such as emotions. The lab of Rose Faghih has been developing a system called MINDWATCH, algorithms and methods for wearable sensors that collect information from electrical signals in the skin to make inferences about mental activity. While their lab has been successful in translating these physiological signals quickly and effectively, they didn't incorporate direct feedback from the individual’s subjective experiences.

Now, the researchers are incorporating feedback and labels from the users, enhanced with machine-learning, and combining it with their existing model for a fuller, more accurate picture of mental states.

At a high level, the human body can be viewed as a complex dynamical system. It is a complex conglomeration of control systems that each works in turn to manage different variables or states. Unfortunately, a number of the states researchers are interested in — particularly the ones that are more abstract — cannot be measured directly. These include states of emotion, cognition and consciousness. Nevertheless, changes in these unobserved states do give rise to corresponding changes in different physiological signals that can be measured more easily. For instance, we may not be able to directly observe or measure a person’s emotional state, but we can measure subtle changes in a person’s heart rate, breathing or sweat secretions (which in turn affects the conductivity of the skin). These signals can then be used to estimate the states we wish to.

However, the signals researchers use to estimate these unobserved states are “spikey” or “pulsatile” in nature. These spikey signals can be used to estimate the various states of the human body and brain without direct observation. With already-existing methods, you could obtain state estimates, but still wouldn’t have any means to have those estimates agree with the more direct state-related information in possession. This is especially important in experiments involving human subjects where the subjects can indeed provide information related to the unobserved state. This type of more-direct state-related information can be called a “label.”

Incorporating direct feedback from users offers information that can’t be gleaned from biological data alone. For instance, a person with PTSD could have their skin conductance continuously monitored to provide an emotion estimate, but ideally, the final estimate should rely both on these signals and information perhaps obtained on rating scales or through regular questionnaires. It is likewise the case for patients with hormone disorders. Hormone measurements do provide valuable information, but should likely be combined with personal feedback (e.g. regarding feelings of energy/lethargy) to obtain a single complete picture. The authors met this need through a proof-of-principle work on a hybrid type of estimator.

Performing estimation on some data such that what is predicted agrees with available labels falls within the domain of supervised machine learning. This work adapted an existing neural network method for state estimation by adding a penalization term for not agreeing with the labels to enable a hybrid estimator. The proposed hybrid estimator was utilized to determine an aspect of emotion tied to changes in skin conductance (through changes in sweat secretions) and to determine energy states within the body based on pulsatile hormone secretions. A wearable monitoring system that incorporates verbal feedback from the user with physiological signals for hybrid estimation can eventually provide a more complete picture of the user to eventually provide more comprehensive closed-loop care.

Stress has a negative impact on physical health, reduces work productivity, and results in significant annual costs for industries and healthcare. While high stress is known to raise the risk of cardiovascular disease and have negative effects on mental health, It also has key effects on the ability of one to complete tasks by both excessively high or excessively low stress. There has been growing research interest on understanding how real-world stress impacts our body and performance, at work and across life activities

Unfortunately, attempts to simulate their impact in the laboratory or elsewhere are less useful than datasets gathered in real-world circumstances. As a result, researchers have access to fewer real-world stress datasets. Even rarer indeed are such datasets used in longitudinal investigations on the same subjects over time.

Real-world situations are also unrestricted environments. Research-grade equipment is frequently inaccessible, and motion artifact contamination is pervasive. These continue to be some of the biggest barriers to automated emotion decoders outside of the research labs in daily life.

To address the above-mentioned gap, Rose Faghih and her former PhD students Md. Rafiul Amin and Dilranjan Wickramasuriya performed an experiment, in which a set of students' physiological data was gathered over the course of three exams. They used a smartwatch-like wearable device and collected multimodal physiological data. The use of the smartwatch-like wearable device was to provide a seamless data collection experience for the students participating in the experiment.

The investigation shows that it is possible to link the variations in the physiological signals to the exam performance. More details about this study can be found in the corresponding publication titled "A Wearable Exam Stress Dataset for Predicting Grades using Physiological Signals."

To enable other researchers, use this dataset for additional investigations, the research team has made the de-identified data publicly available on the PhysioNet platform. A Wearable Exam Stress Dataset for Predicting Cognitive Performance in Real-World Settings is available at: physionet.org/content/wearable-exam-stress

Ultimately, the researchers believe it would be extremely beneficial to consider how exam performance and the stress that goes along with it interact. It will allow for a wide range of potential applications with the aim of enhancing personal performance. This may, for instance, assist scientists in developing effective interventions to improve each person's performance and increase productivity within a company. Additionally, the knowledge may be used in online and remote learning contexts to connect with students effectively and improve learning outcomes.

One in six deaths from cardiovascular disease is due to stroke. Caused by a blood clot in the brain, a stroke can have severe consequences, even minutes after initially occurring. That is what makes prevention so important.

Now, thanks to NYU researchers, including a collaborative effort between NYU Assistant Professor S. Farokh Atashzar, and NYUAD Assistant Professor Farah Shamout, early monitoring may soon be available at an unlikely location: your eye doctor.

Their project, called EyeScore, is developing a technology that uses non-invasive scans of the retina to predict the recurrence of stroke in patients. They use optical coherence tomography — a scan of the back of the retina — and track changes over time. The retina, attached directly to the brain through the optic nerve, can be used as an indicator for changes in the brain itself.

Atashzar and Shamout are currently formulating their hybrid AI model, pinpointing the exact changes that can predict a stroke and recurrence of strokes. The outcome will be able to analyze these images and flag potentially troublesome developments. And since the scans are already in use in optometrist offices, this life-saving technology could be in the hands of medical professionals sooner than expected.