Research News
High-profile incidents of police brutality sway public opinion more than performance of people’s local law enforcement, new study from NYU Tandon reveals
National media coverage of police brutality influences public perceptions of law enforcement more than the performance of people’s local police departments, according to data analysis from NYU Tandon School of Engineering, challenging the assumption that public confidence in police depends mostly on feeling safe from local crime.
In a study published in Communications Psychology, a NYU Tandon research team tracked media coverage of police brutality in 18 metropolitan areas in the United States — along with coverage of local crimes — and analyzed tweets from those cities to tease out positive attitudes from negative ones towards the police.
Led by Maurizio Porfiri, Institute Professor and Director of the Center for Urban Science and Progress (CUSP), the team found when high-profile cases of police brutality make the news, negative sentiment and distrust towards police spikes across cities, even if the incident occurred in another state.
In contrast, local media coverage of crimes in people's own cities had little sway over their views of the police. Porfiri discussed the research and its implications in a blog post.
“Our research shows that police misconduct occurring anywhere reverberates across the country, while performance of police in their own communities contribute minimally towards attitudes around those local police departments,” said Rayan Succar, a Ph.D. candidate in Mechanical Engineering and CUSP who is the paper’s lead author. “The pattern holds steady across diverse cities.”
To reach their conclusions, researchers employed transfer entropy — an advanced statistical technique that allowed them to detect causal relationships within complex systems that change over time — in their analysis of more than 2.5 million geo-localized tweets. The approach allows for significantly more time-sensitive analysis of public sentiment than standard surveys which are constrained to the point in time at which they are fielded.
“By comparing this time series tracking shifts in sentiment to parallel time series documenting volumes of media coverage about local crime and national police brutality news, transfer entropy quantified causal relationships between media coverage and Twitter discourse about law enforcement,” said Salvador Ramallo, Fulbright Scholar from the University of Murcia in Spain and a visiting member of CUSP who is part of the research team.
The researchers assembled their data from the period October 1, 2010 to December 31, 2020. With a time resolution of one minute, the team collected tweets in each metropolitan area that contained the words “police,” “cop,” or the local police department name abbreviation of the main city in the metropolitan area (“NYPD” for New York Police Department).
In that same time frame, researchers collected coverage of police brutality and of local crime from 17 of the 20 most circulated newspapers.
To better detail the interplay between media coverage and public sentiment, the researchers also zeroed in on a two-week period around the heavily-covered George Floyd murder, a notorious example of extreme police brutality. Specifically, they scraped the Twitter feeds of the top 10 most-followed newspaper profiles and created a time series of police brutality coverage from May 29, 2020 until June 13, 2020.
This highly resolved time series was examined in conjunction with the time series of negative tweets about the police for each of the 18 metropolitan areas during the same two-week time window.
“The research reveals how profoundly a single incident of police violence can rupture public trust in police everywhere,” said CUSP postdoctoral fellow Roni Barak Ventura, a member of the research team. “The findings suggest that to improve perceptions, police departments may need to prioritize transparency around misconduct allegations as much as local crime fighting. More community dialogue and balanced media coverage may also help build understanding between police and the public they serve.”
This study is the latest in a series that Porfiri is pursuing under a 2020 National Science Foundation grant awarded to study the “firearm ecosystem” in the United States. His research employs sophisticated data analytics to investigate the firearm ecosystem on three different scales. On the macroscale, research illuminates cause-and-effect relationships between firearm prevalence and firearm-related harms. On the mesoscale, the project explores the ideological, economic, and political landscape underlying state approaches to firearm safety. On the microscale, research delves into individual opinions about firearm safety.
Porfiri’s prior published research has focused on motivations of fame-seeking mass shooters, factors that prompt gun purchases, state-by-state gun ownership trends, and forecasting monthly gun homicide rates.
CUSP postdoctoral fellow Rishita Das also contributed to the study.
Succar, R., Ramallo, S., Das, R. et al. Understanding the role of media in the formation of public sentiment towards the police. Commun Psychol 2, 11 (2024). https://doi.org/10.1038/s44271-024-00059-8
Asylum seekers’ mental health benefits from sheltering in refugee centers, new study reveals
Sheltering in refugee centers can positively impact asylum seekers’ mental health, according to a new study published in Communications Medicine, underscoring the benefits of providing migrants safe and welcoming transitional environments in which professionals in the host countries monitor their psychological and physical needs.
The study’s multidisciplinary research team, coordinated by Emanuele Caroppo — Head of International Projects and Researches at the Department of Mental Health Asl Roma 2 — administered a battery of six questionnaires, ranging from demographic surveys to comprehensive psychological assessments, to a cohort of 100 asylum-seekers in 14-day COVID-19-related quarantines in Italy between August 2020 and September 2021.
Maurizio Porfiri, NYU Tandon Institute Professor and Director of the Center for Urban Science and Progress (CUSP), designed the framework for the statistical analysis and led the interpretation of the results. He and Pietro De Lellis, associate professor in the Department of Electrical Engineering and Information Technology at University of Naples Federico II, are the corresponding authors on the paper.
The study’s aim was to understand the impact of the first contact with the reception system on the mental health of asylum-seekers, and to delve into predictors of Post Traumatic Stress Disorder (PTSD) among that population.
Twenty-three percent of asylum-seekers in the study had PTSD — higher than the 4 to 10% incidence previously reported among the general global population. Pre-migration traumatic experiences were the key influencers in the the development of PTSD, including the infliction of bodily injury and torture, and witnessing violence. The study found no specific demographic factors that played a crucial role in predicting PTSD. Social ties and education levels did not emerge as salient features to predict the onset of PTSD.
Despite the incidence of PTSD, the authors also observed that a 14-day stay in reception facilities appeared to positively impact asylum-seekers’ mental health, with the proportion of participants needing to undergo further psychological assessments decreasing from 51% to 21% throughout the quarantine period.
The study offers a significant step towards understanding the relationship between migration, mental health, and the reception environment. Asylum-seekers, who have already endured tremendous hardship, may find a glimmer of hope in the notion that a supportive and secure environment can significantly contribute to their psychological well-being.
Along with Porfiri, De Lellis and Caroppo, the study’s researchers are Carmela Calabrese, Department of Electrical Engineering and Information Technology, University of Naples Federico II and the Institut de Neurosciences des Systèmes (INS), Aix Marseille Université; Marianna Mazza, Institute of Psychiatry and Psychology, Department of Geriatrics, Neuroscience and Orthopedics, Fondazione Policlinico Universitario A. Gemelli IRCCS and Università Cattolica del Sacro Cuore, Department of Psychiatry, Università Cattolica del Sacro Cuore; Alessandro Rinaldi, Migrant Health Unit, Local Health Authority Roma; Daniele Coluzzi, Migrant Health Unit, Local Health Authority Roma; Pierangela Napoli, Migrant Health Unit, Local Health Authority Roma; Martina Sapienza, Department of Life Sciences and Public Health, Università Cattolica del Sacro Cuore; and the UOC Salute Mentale Asl Roma 2 in Rome, Italy.
Caroppo, E., Calabrese, C., Mazza, M. et al. Migrants’ mental health recovery in Italian reception facilities. Commun Med 3, 162 (2023)
New research reveals alarming privacy and security threats in Smart Homes
An international team of researchers, led by IMDEA Networks and Northeastern University in collaboration with NYU Tandon School of Engineering, Universidad Carlos III de Madrid, IMDEA Software, University of Calgary, and the International Computer Science Institute, has unveiled groundbreaking findings on the security and privacy challenges posed by the ever-growing prevalence of opaque and technically complex Internet of Things (IoT) devices in smart homes.
Smart Homes: Trusted and Secure Environments?
Smart homes are becoming increasingly interconnected, comprising an array of consumer-oriented IoT devices ranging from smartphones and smart TVs to virtual assistants and CCTV cameras. These devices have cameras, microphones, and other ways of sensing what is happening in our most private spaces — our homes.
An important question is: can we trust that these devices in our homes are safely handling and protecting the sensitive data they have access to?
“When we think of what happens between the walls of our homes, we think of it as a trusted, private place. In reality, we find that smart devices in our homes are piercing that veil of trust and privacy — in ways that allow nearly any company to learn what devices are in your home, to know when you are home, and learn where your home is. These behaviours are generally not disclosed to consumers, and there is a need for better protections in the home,” said David Choffnes, Associate Professor of Computer Science and Executive Director of the Cybersecurity and Privacy Institute at Northeastern University.
The research team's extensive study, titled "In the Room Where It Happens: Characterizing Local Communication and Threats in Smart Homes," was presented this week at the ACM Internet Measurement Conference (ACM IMC'23) in Montreal (Canada).
The paper delves for the first time into the intricacies of local network interactions between 93 IoT devices and mobile apps, revealing a plethora of previously undisclosed security and privacy concerns with actual real-world implications.
While most users typically view local networks as a trusted and safe environment, the study's findings illuminate new threats associated with the inadvertent exposure of sensitive data by IoT devices within local networks using standard protocols such as UPnP or mDNS.
These threats include the exposure of unique device names, UUIDs, and even household geolocation data, all of which can be harvested by companies involved in surveillance capitalism without user awareness.
According to Vijay Prakash, PhD student from NYU Tandon who co-authored the paper, "analyzing the data collected by IoT Inspector, we found evidence of IoT devices inadvertently exposing at least one PII (Personally Identifiable Information), like unique hardware address (MAC), UUID, or unique device names, in thousands of real world smart homes. Any single PII is useful for identifying a household, but combining all three of them together makes a house very unique and easily identifiable. For comparison, if a person is fingerprinted using the simplest browser fingerprinting technique, they are as unique as one in 1,500 people. If a smart home with all three types of identifiers is fingerprinted, it is as unique as one in 1.12 million smart homes."
These local network protocols can be employed as side-channels to access data that is supposedly protected by several mobile app permissions such as household locations.
“A side channel is a sneaky way of indirectly accessing sensitive data. For example, Android app developers are supposed to request and obtain users’ consent to access data like geolocation. However, we have shown that certain spyware apps and advertising companies do abuse local network protocols to silently access such sensitive information without any user awareness. All they have to do is kindly ask for it to other IoT devices deployed in the local network using standard protocols like UPnP,” said Narseo Vallina-Rodriguez, Associate Research Professor of IMDEA Networks and co-founder of AppCensus.
“Our study shows that the local network protocols used by IoT devices are not sufficiently protected and expose sensitive information about the home and our use of the devices. This information is being collected in an opaque way and makes it easier to create profiles of our habits or socioeconomic level,” adds Juan Tapiador, professor at UC3M.
The Wider Implications
The impact of this research extends far beyond academia. The findings underscore the imperative for manufacturers, software developers, IoT and mobile platform operators, and policymakers to take action to enhance the privacy and security guarantees of smart home devices and households. The research team responsibly disclosed these issues to vulnerable IoT device vendors and to Google's Android Security Team, already triggering security improvements in some of these products.
About IMDEA Networks: IMDEA Networks is a non-profit, independent research institute located in Madrid (Spain). Its multinational research team conducts cutting-edge fundamental research in the fields of computer and communication networks to develop future network principles and technologies; designing and creating today the networks of tomorrow.
About Northeastern University: Founded in 1898, Northeastern is a global research university and the recognized leader in experience-driven lifelong learning. Our world-renowned experiential approach empowers our students, faculty, alumni, and partners to create impact far beyond the confines of discipline, degree, and campus.
About NYU Tandon School of Engineering: The NYU Tandon School of Engineering is home to a community of renowned faculty, undergraduate and graduate students united in a mission to understand and create technology that powers cities, enables worldwide communication, fights climate change, and builds healthier, safer, and more equitable real and digital worlds. NYU Tandon dates back to 1854 and is a vital part of New York University and its unparalleled global network.
About UC3M: The Universidad Carlos III de Madrid (UC3M) is a Spanish public university that excels in research, teaching and innovation. It is among the best universities in the world in the QS World University Rankings 2024 and among the best universities for the employability of its graduates, according to the latest edition of the Times Higher Education (THE) Global University Employability Ranking.
About IMDEA Software: The IMDEA Software Institute is a non-profit, independent research institute promoted by the Madrid Regional Government to perform research of excellence and technology transfer in the methods, languages, and tools that will allow the cost-effective development of software products with sophisticated functionality and high quality.
About University of Calgary: UCalgary is Canada’s entrepreneurial university, located in Canada’s most enterprising city. It is a top research university and one of the highest-ranked universities of its age. Founded in 1966, its 35,000 students experience an innovative learning environment, made rich by research, hands-on experiences and entrepreneurial thinking. It is Canada’s leader in the creation of start-ups. For more information, visit ucalgary.ca.
About International Computer Science Institute: ICSI is an independent non-profit research institute affiliated with the University of California, Berkeley. With a focus on scientific excellence and social impact, ICSI transcends disciplinary boundaries in computing and data sciences and collaborates internationally and across sectors to inspire breakthroughs.
Citation:
Aniketh Girish, Tianrui Hu, Vijay Prakash, Daniel J. Dubois, Srdjan Matic, Danny Yuxing Huang, Serge Egelman, Joel Reardon, Juan Tapiador, David Choffnes, and Narseo Vallina-Rodriguez. 2023. In the Room Where It Happens: Characterizing Local Communication and Threats in Smart Homes. In Proceedings of the 2023 ACM on Internet Measurement Conference (IMC '23). Association for Computing Machinery, New York, NY, USA, 437–456. https://doi.org/10.1145/3618257.3624830
Studying the online deepfake community
In the evolving landscape of digital manipulation and misinformation, deepfake technology has emerged as a dual-use technology. While the technology has diverse applications in art, science, and industry, its potential for malicious use in areas such as disinformation, identity fraud, and harassment has raised concerns about its dangerous implications. Consequently, a number of deepfake creation communities, including the pioneering r/deepfakes on Reddit, have faced deplatforming measures to mitigate risks.
A noteworthy development unfolded in February 2018, just over a week after the removal of r/deepfakes, as MrDeepFakes (MDF) made its entrance into the online realm. Functioning as a privately owned platform, MDF positioned itself as a community hub, boasting to be the largest online space dedicated to deepfake creation and discussion. Notably, this purported communal role sharply contrasts with the platform's primary function — serving as a host for nonconsensual deepfake pornography.
Researchers at NYU Tandon led by Rachel Greenstadt, Professor of Computer Science and Engineering and a member of the NYU Center for Cybersecurity, undertook an exploration of these two key deepfake communities utilizing a mixed methods approach, combining quantitative and qualitative analysis. The study aimed to uncover patterns of utilization by community members, the prevailing opinions of deepfake creators regarding the technology and its societal perception, and attitudes toward deepfakes as potential vectors of disinformation.
Their analysis, presented in a paper written by lead author and Ph.D. candidate Brian Timmerman, revealed a nuanced understanding of the community dynamics on these boards. Within both MDF and r/deepfakes, the predominant discussions lean towards technical intricacies, with many members expressing a commitment to lawful and ethical practices. However, the primary content produced or requested within these forums were nonconsensual and pornographic deepfakes. Adding to the complexity are facesets that raise concerns, hinting at potential mis- and disinformation implications with politicians, business leaders, religious figures, and news anchors comprising 22.3% of all faceset listings.
In addition to Greenstadt and Timmerman, the research team includes Pulak Mehta, Progga Deb, Kevin Gallagher, Brendan Dolan-Gavitt, and Siddharth Garg.
Timmerman, B., Mehta, P., Deb, P., Gallagher, K., Dolan-Gavitt, B., Garg, S., Greenstadt, R. (2023). Studying the online Deepfake Community. Journal of Online Trust and Safety, 2(1). https://doi.org/10.54501/jots.v2i1.126
Efficient object manipulation planning with Monte Carlo Tree Search
This paper was a finalist for the Best Paper Award on Mobile Manipulation at IROS 2023. It is one of 5 finalists out of 2760 submitted papers at one of the largest robotics conferences in the world.
In the field of robotics, the intricate dance of planning how machines touch and maneuver objects is a linchpin for granting them the autonomy to execute intricate tasks. However, this endeavor remains a challenge. The pursuit of identifying dynamically viable sequences of contacts between robotic manipulators and objects often unravels into a web of formidable combinatorial and nonlinear complications.
Consider the ostensibly simple act of reorienting an object nestled on a table using a two-fingered robotic hand. The strategic planning of contacts necessitates a thoughtful consideration of interaction forces. For instance, a cube succumbs to rotation through forces applied from its sides. In stark contrast, when faced with a slender plate, the robotic fingers must exert a downward pressure — initiating a frictional interplay — to achieve a comparable outcome.
The crux of the challenge lies in the nuanced orchestration of interaction forces and judicious contact switches. As the robotic fingers approach their kinematic limits, a process of breaking and re-establishing contacts becomes imperative to propel the object into further rotation. These twin challenges, encapsulated in interaction forces and contact switches, underscore the complexity inherent in planning the manipulation of objects.
Over the past decade, trajectory optimization has emerged as the favored approach for orchestrating multi-contact motion planning. This methodology holds sway due to its capacity to construct efficient formulations for navigating the intricate terrain of interaction forces. Yet, a lingering problem persists—the effective integration of planning for contact modes remains elusive, primarily attributed to its discrete nature, injecting a disruptive discontinuity into the dynamics at the crucial juncture of contact switches.
Now, a team of NYU researchers led by Ludovic Righetti, Associate Professor of Electrical and Computer Engineering Department and Mechanical and Aerospace Engineering, as well as a member of the Center for Urban Science + Progress, have developed a strategy for planning object manipulation, leveraging the power of Monte Carlo Tree Search (MCTS) to discern optimal contact sequences. Complementing this, an adept trajectory optimization algorithm, rooted in Alternating Direction Method of Multipliers (ADMM), evaluates the dynamic feasibility of potential contact sequences. The algorithm was previously developed by Righetti’s group, The Machines in Motion Laboratory.
The team, including Ph.D. students Huaijiang Zhu and Avadesh Meduri, made a key innovation in expediting MCTS involving the development of a goal-conditioned policy-value network, guiding the search toward promising nodes. Additionally, manipulation-specific heuristics prove instrumental in markedly shrinking the search space.
The efficacy of the approach is underscored through a series of meticulous object manipulation experiments conducted both in a physics simulator and on tangible hardware. The methodology exhibits a favorable scalability for protracted manipulation sequences, a testament to the learned policy-value network.
This advancement substantially elevates the planning success rate, marking a significant stride in the realm of object manipulation planning.
Zhu, H., Meduri, A., & Righetti, L. (2023, March 19). Efficient object manipulation planning with Monte Carlo Tree Search. arXiv.org. https://arxiv.org/abs/2206.09023
A large-scale analytical residential parcel delivery model evaluating greenhouse gas emissions, COVID-19 impact, and cargo bikes
The e-commerce industry, which has seen remarkable growth over the past decade, experienced an even more accelerated surge in the wake of the COVID-19 pandemic. This exponential rise in online shopping has triggered a corresponding boom in the parcel delivery sector. However, a glaring gap exists in our understanding of the extensive social and environmental repercussions of this burgeoning industry.
To bridge this knowledge void, researchers at NYU Tandon led by Joseph Chow, Institute Associate Professor of Civil and Urban Engineering and Deputy Director of the C2SMARTER University Transportation Center, have proposed a comprehensive model to scrutinize the multifaceted impacts stemming from the parcel delivery surge. The model's architecture incorporates a parcel generation process, ingeniously converting publicly available data into precise figures detailing parcel volumes and delivery destinations. Additionally, a sophisticated continuous approximation model has been meticulously calibrated to gauge the lengths of parcel service routes.
The veracity of this model was subjected to rigorous examination through a real-world case study, employing a trove of data from the labyrinthine streets of New York City. Impressively, the parcel generation process demonstrated an impressive degree of fidelity to the actual data. Even more striking were the high R2 values, consistently hovering at 98% or greater, characterizing the model's ability to approximate reality. Validation of the model's output was further solidified by comparing it against the tangible UPS truck journeys.
Applying this model to the year 2021, it emerged that residential parcel deliveries in NYC constituted 0.05% of the total daily vehicle-kilometers traveled (VKT), equivalent to a staggering 14.4 metric tons of carbon emissions per day. The COVID-19 pandemic substantially contributed to a surge in parcel deliveries, culminating in an alarming annual greenhouse gas (GHG) emissions figure of 1064.3 metric tons of carbon equivalent (MTCE) within the city's boundaries. To put this in perspective, this is sufficient to power the homes of 532 standard US households for an entire year.
A ray of hope emerges in the form of NYC's existing bike lane infrastructure, which has the capacity to seamlessly replace 17% of parcel deliveries with eco-friendly cargo bikes, thereby precipitating an 11% reduction in VKT. By strategically augmenting this infrastructure with 3 kilometers of bike lanes connecting Amazon facilities, the cargo bike substitution benefit skyrockets from 5% to an impressive 30% reduction in VKT. The prospect becomes even more promising with the construction of an additional 28 kilometers of bike lanes citywide, potentially pushing parcel delivery substitution via cargo bikes from 17% to a remarkable 34%, concurrently saving an extra 2.3 MTCE per day.
Notably, the prioritization of cargo bike deployments holds the potential to disproportionately benefit lower-income neighborhoods, including but not limited to Harlem, Sunset Park, and Bushwick, by substantially curtailing GHG emissions in these communities.
Hai Yang, Hector Landes, Joseph Y.J. Chow, "A large-scale analytical residential parcel delivery model evaluating greenhouse gas emissions, COVID-19 impact, and cargo bikes," International Journal of Transportation Science and Technology, 2023, ISSN 2046-0430.
Microscopy image segmentation via point and shape regularized data synthesis
In contemporary deep learning-based methods for segmenting microscopic images, there's a heavy reliance on extensive training data that requires detailed annotations. This process is both expensive and labor-intensive. An alternative approach involves using simpler annotations, such as marking the center points of objects. While not as detailed, these point annotations still provide valuable information for image analysis.
In this study, researchers from NYU Tandon and University Hospital Bonn in Germany assume that only point annotations are available for training and present a novel method for segmenting microscopic images using artificially generated training data. Their framework consists of three main stages:
1. Pseudo Dense Mask Generation: This step takes the point annotations and creates synthetic, detailed masks that are constrained by shape information.
2. Realistic Image Generation: An advanced generative model, trained in a unique way, transforms these synthetic masks into highly realistic microscopic images while maintaining consistency in object appearance.
3. Specialized Model Training: The synthetic masks and generated images are combined to create a dataset used to train a specialized model for image segmentation.
The research was led by Guido Gerig, Institute Professor of Computer Science and Engineering and Biomedical Engineering, alongside PhD students Shijie Li and Mengwei Ren, as well as Thomas Ach at University Hospital Bonn. The three NYU Tandon researchers are also members of the Visualization and Data Analytics (VIDA) Research Center.
The researchers tested their method on a publicly available dataset and found that their approach produced more diverse and realistic images compared to conventional methods, all while maintaining a strong connection between the input annotations and the generated images. Importantly, when compared to models trained using other methods, their models, trained on synthetic data, outperformed them significantly. Moreover, their framework achieved results on par with models trained using labor-intensive, highly detailed annotations.
This research highlights the potential of using simplified annotations and synthetic data to streamline the process of segmenting microscopic images, potentially reducing the need for extensive manual annotation efforts. The research, in collaboration with the Ophthalmology department at University Hospital Bonn, is a first step in a collaboration to finally process three dimensional retinal cell images of the human eye from subjects diagnosed for age-related macular degeneration (AMD), a leading cause of vision loss in older adults.
The code for this method is publicly available for further exploration and implementation.
“Microscopy Image Segmentation via Point and Shape Regularized Data Synthesis.” S Li, M Ren, T Ach, G Gerig. arXiv preprint, arXiv:2308.09835, 2023.
NYU Tandon researchers unveil tool to help developers create augmented reality task assistants
Augmented reality (AR) technology has long fascinated both the scientific community and the general public, remaining a staple of modern science fiction for decades.
In the pursuit of advanced AR assistants – ones that can guide people through intricate surgeries or everyday food preparation, for example – a research team from NYU Tandon School of Engineering has introduced Augmented Reality Guidance and User-Modeling System, or ARGUS.
An interactive visual analytics tool, ARGUS is engineered to support the development of intelligent AR assistants that can run on devices like Microsoft HoloLens 2 or MagicLeap. It enables developers to collect and analyze data, model how people perform tasks, and find and fix problems in the AR assistants they are building.
Claudio Silva, NYU Tandon Institute Professor of Computer Science and Engineering and Professor of Data Science at the NYU Center for Data Science, leads the research team that will present its paper on ARGUS at IEEE VIS 2023 on October 26, 2023, in Melbourne Australia. The paper received Honorable Mention in that event’s Best Paper Awards.
“Imagine you’re developing an AR AI assistant to help home cooks prepare meals,” said Silva. “Using ARGUS, a developer can monitor a cook working with the ingredients, so they can assess how well the AI is performing in understanding the environment and user actions. Also, how the system is providing relevant instructions and feedback to the user. It is meant to be used by developers of such AR systems.”
ARGUS works in two modes: online and offline.
The online mode is for real-time monitoring and debugging while an AR system is in use. It lets developers see what the AR system sees and how it's interpreting the environment and user actions. They can also adjust settings and record data for later analysis.
The offline mode is for analyzing historical data generated by the AR system. It provides tools to explore and visualize this data, helping developers understand how the system behaved in the past.
ARGUS’ offline mode comprises three key components: the Data Manager, which helps users organize and filter AR session data; the Spatial View, providing a 3D visualization of spatial interactions in the AR environment; and the Temporal View, which focuses on the temporal progression of actions and objects during AR sessions. These components collectively facilitate comprehensive data analysis and debugging.
“ARGUS is unique in its ability to provide comprehensive real-time monitoring and retrospective analysis of complex multimodal data in the development of systems,” said Silva. “Its integration of spatial and temporal visualization tools sets it apart as a solution for improving intelligent assistive AR systems, offering capabilities not found together in other tools.”
ARGUS is open source and available on GitHub under VIDA-NYU. The work is supported by the DARPA Perceptually-enabled Task Guidance (PTG) program.
ARGUS: Visualization of AI-Assisted Task Guidance in AR
Postintervention monitoring of peripheral arterial disease wound healing using dynamic vascular optical spectroscopy
Peripheral arterial disease (PAD) is a vascular disease that is caused by clogging of the arteries due to plaque. The lower legs and feet are often impacted, and symptoms include pain, numbness, difficulty walking, and non-healing wounds.
For many patients, wounds continue not to heal even after they have undergone a surgical revascularization procedure designed to unclog the affected arteries, necessitating another intervention. Determining whether wounds will heal must be done as soon as possible after the first intervention to reduce the duration of a patient’s pain and increase the likelihood of a good outcome.
Currently, the most common methods for monitoring PAD progression (and related wound healing) are the ankle-brachial index (ABI) and ultrasound imaging. The ABI uses the ratio of systolic blood pressure measurements from arteries in the lower extremities to the systolic blood pressure measurement from the brachial artery in the arm. PAD patients generally have pressure ratios that are below a certain threshold (often 0.9). The ABI has low accuracy when monitoring vasculature in diabetic patients and ultrasound imaging has low accuracy when monitoring smaller arteries such as those in the feet. Unfortunately, PAD is often comorbid with diabetes and the affected arteries are commonly in the feet.
To address the existing limitations of the current technology, a team from NYU Tandon School of Engineering’s department of Biomedical Engineering, including lead author Nisha Maheshwari from Andreas Hielscher's Clinical Biophotonics Laboratory, developed optical imaging technology to assist physicians with monitoring the healing of lower limb ulcers after a surgical intervention.
Dynamic vascular optical spectroscopy (DVOS) is an optical imaging technology that uses light in the red and near-infrared ranges to determine characteristics of blood flow through arteries. In the paper “Postintervention monitoring of peripheral arterial disease wound healing using dynamic vascular optical spectroscopy,” published in the Journal of Biomedical Optics, the team used their DVOS system to monitor a set of 14 patients with PAD that underwent a surgical revascularization procedure. Of these patients, five needed a second intervention due to the persistence of non-healing wounds.
The team was able to correctly categorize the long-term healing and non-healing of wounds in 93% of this patient population within only one month after each patient’s initial intervention. The method outperformed the gold standard methods of ultrasound and ABI. These findings suggest that the DVOS may be able to assist physicians in improving patient outcomes and reducing long-term pain by determining wound outcome earlier than existing technology can.
The authors would like to thank the patients who volunteered for this study for their time and participation. This work was supported in part by the National Heart, Lung, and Blood Institute (Grant No. NHLBI-1R01-HL115336); Wallace H. Coulter Foundation; Society of Vascular Surgery; Columbia University Fu Foundation School of Engineering and Applied Science; and New York University Tandon School of Engineering.
"Maheshwari N, Marone A, Altoé M, Kim SHK, Bajakian DR, Hielscher AH. Postintervention monitoring of peripheral arterial disease wound healing using dynamic vascular optical spectroscopy. J Biomed Opt. 2022 Dec;27(12):125002. doi: 10.1117/1.JBO.27.12.125002. Epub 2022 Dec 24. PMID: 36582192; PMCID: PMC9789744."
NYU Tandon School of Engineering researchers develop algorithm for safer self-driving cars
In a promising development for self-driving car technology, a research team at NYU Tandon School of Engineering has unveiled an algorithm — known as Neurosymbolic Meta-Reinforcement Lookahead Learning (NUMERLA) — that could address the long-standing challenge of adapting to unpredictable real-world scenarios while maintaining safety.
The research was conducted by Quanyan Zhu, NYU Tandon associate professor of electrical and computer engineering, and his Ph.D. candidate Haozhe Lei.
Artificial intelligence and machine learning have helped self-driving cars operate in increasingly intricate scenarios, allowing them to process vast amounts of data from sensors, make sense of complex environments, and navigate city streets while adhering to traffic rules.
As they venture beyond controlled environments into the chaos of real-world traffic, however, such vehicles’ performance can falter, potentially leading to accidents.
NUMERLA aims to bridge the gap between safety and adaptability. The algorithm achieves this by continuously updating safety constraints in real-time, ensuring that self-driving cars can navigate unfamiliar scenarios while maintaining safety as the top priority.
The NUMERLA framework operates as follows: When a self-driving car encounters an evolving environment, it uses observations to adjust its “belief” about the current situation. Based on this belief, it makes predictions about its future performance within a specified timeframe. It then searches for appropriate safety constraints and updates its knowledge base accordingly.
The car's policy is adjusted using lookahead optimization with safety constraints, resulting in a suboptimal but empirically safe online control strategy.
One of the key innovations of NUMERLA lies in its lookahead symbolic constraints. By making conjectures about its future mode and incorporating symbolic safety constraints, the self-driving car can adapt to new situations on the fly while still prioritizing safety.
The researchers tested NUMERLA in a computer platform that simulates urban environments – specifically to ascertain its ability to accommodate jaywalkers — and it outperformed other algorithms in those scenarios.
Lei, Haozhe & Zhu, Quanyan. (2023). Neurosymbolic Meta-Reinforcement Lookahead Learning Achieves Safe Self-Driving in Non-Stationary Environments.