Seeing Through a Robot’s Eyes: Augmented Reality Helps Humans Predict Machine Behavior

As robots increasingly move out of factories and into workplaces, hospitals, warehouses and public spaces, a simple challenge becomes increasingly important: helping people understand what those machines are about to do.

A new study by researchers at Bowling Green State University (BGSU) and NYU Tandon suggests that augmented reality (AR) may offer a surprisingly effective solution. By overlaying a robot's goals, planned routes and safety zones onto the real world through a smartphone, the researchers found that people became significantly better at anticipating robot behavior and identifying potential hazards.

The work, published in the journal Empathic Computing, addresses a growing concern in human-robot interaction: transparency. While robots are becoming increasingly autonomous, their decision-making processes often remain opaque to nearby humans. That uncertainty can create confusion, reduce trust and, in some situations, compromise safety.

"One of the biggest challenges in human-robot collaboration is helping people understand what a robot intends to do before it acts," says co-author Vikram Kapila, Professor of Mechanical and Aerospace Engineering at NYU Tandon. "When users can see a robot's planned path, destination and safety boundaries, they are better able to anticipate its actions and make informed decisions about their own movements."

To tackle the problem, the researchers developed a smartphone-based AR application that communicates a mobile robot's intentions in real time. Using a standard Android phone equipped with Google's ARCore software, the system overlays digital information directly onto the user's view of the physical environment.

The application provides three types of visual information. One mode displays the robot's destination as a virtual location pushpin. Another reveals the route the robot plans to follow. A third shows a digital twin of the robot itself moving through the environment, complete with a visual buffer zone indicating areas where collisions or interference could occur.

Unlike many previous AR systems, which rely on specialized headsets or projection hardware, the new approach works on an ordinary smartphone. That simplicity could make the technology easier to deploy in workplaces where workers and autonomous machines routinely share space.

To test whether the system actually improved understanding, the researchers recruited 58 participants with varying levels of experience in robotics and augmented reality. Participants viewed a series of AR scenarios showing robot navigation tasks and then answered questions designed to measure what human-factors researchers call situational awareness — the ability to perceive, understand and predict events unfolding in an environment.

The evaluation was based on the Situational Awareness Global Assessment Technique, or SAGAT, a widely used framework that measures three levels of awareness: perception of relevant information, comprehension of its meaning and projection of future events.

Participants were asked to identify robot goals, recognize obstacles, determine whether objects would interfere with the robot's movement and predict which areas of the environment would remain safe for human occupancy.

The results were striking. Across all tasks, participants achieved an average situational-awareness score of 86.5 percent. They were particularly successful at recognizing obstacles and identifying safe zones where they could avoid interfering with the robot's operation.

Just as important, participants reported feeling more confident about working alongside robots. More than 96 percent said the AR interface improved their understanding of robot intentions and increased their confidence in predicting robot behavior.

"The findings demonstrate that even a lightweight, smartphone-based AR system can substantially improve people's awareness of a robot's goals and movements," Kapila says. "That increased understanding is an important foundation for building trust, safety and effective collaboration between humans and autonomous systems," said paper’s lead author Sonia Chacko, Assistant Professor at BGSU, who received her doctoral degree from NYU Tandon.  

The study arrives at a moment when robots are becoming more common in settings that were once the exclusive domain of humans. Warehouses increasingly rely on autonomous mobile robots to move goods. Hospitals are experimenting with robotic delivery systems. Service robots are beginning to appear in airports, hotels and retail environments. In such settings, the ability to quickly understand what a robot is planning to do may be as important as the robot's ability to understand human behavior.

Kapila also recently developed a mixed-reality system that allows people to communicate force instructions to a robot using a tablet. Instead of programming the robot through complex controls, users place a virtual arrow on the tablet screen over a real-world object. The arrow's position tells the robot where to apply force, its orientation indicates the direction of the force, and its length specifies how much force should be used. The robot then carries out the task and provides visual feedback through a moving virtual indicator that shows whether the desired force has been achieved.

The findings reported in the journal Machines, authored by Christian Lourido, Kishan Reddy Raghunath and Kapila, suggest that mixed reality can make human-robot collaboration more intuitive by allowing people to communicate complex physical intentions visually rather than through specialized programming or expensive haptic equipment. Such technology could eventually be useful in manufacturing, healthcare, and other environments where humans and robots must work together safely and precisely.

The findings suggest that making robotic intentions visible may help bridge one of the most persistent gaps in human-machine collaboration. As autonomous systems become more common in everyday life, a digital window into a robot's plans could make working alongside them feel far more comfortable.