New Faculty Fellow Jing Zhang is Teaching AI to Make Sense of the Physical World

When Jing Zhang talks about artificial intelligence, she does not limit herself to the problems typically associated with robotics or computer vision. Her research has taken her from reconstructing ancient artifacts to building digital versions of New York City that can help robots learn to navigate the real world.

Although these projects may appear very different, they are connected by a common question: how can AI make sense of a physical world that is incomplete, changing, or only partially observed?

That breadth of interests is part of what Zhang brings to NYU Tandon, where she is joining the faculty as a Faculty Fellow in Mechanical and Aerospace Engineering. Her work sits at the intersection of artificial intelligence, computer vision, robotics and simulation, with a particular interest in using AI to help machines understand and interact with complex environments.

Zhang earned her PhD from Wuhan University in China in June 2023 before coming to NYU as a postdoctoral researcher. During her time at NYU, she worked across the Department of Anthropology and Mechanical and Aerospace Engineering, pursuing two seemingly different research directions that share a common thread: using AI to make sense of incomplete information about the physical world. In 2025, she was named an MIT EECS Rising Star.

In one project, Zhang worked with an anthropology professor to develop AI methods for reconstructing fragmented archaeological objects. Ancient pottery, eggshells, bones and other artifacts are often recovered in pieces, leaving researchers to determine how the fragments fit together.

Rather than relying solely on the geometry of the individual pieces, Zhang's approach uses generative AI to reason about the object as a whole.

“We don't want to rely only on the local geometry to fit the fragments together,” she explained. “We also want to imagine the whole shape. For example, these fragments may have once formed a bowl.”

The approach combines perception with what Zhang describes as imagination: The AI generates a possible complete object while simultaneously determining how the individual fragments might fit into it. The same basic problem also has applications beyond archaeology, including medical and industrial cases and other problems involving partial observations.

Her other major research direction is robotics. Working with Professor Chen Feng, Zhang has helped develop algorithms for robot navigation and simulation through Digitize NYC, an NSF-funded project that uses New York City as a kind of living laboratory.

The challenge is straightforward: Testing robots in the physical world can be expensive and sometimes risky. A poorly performing navigation algorithm can cause a robot to fall or collide with something, potentially damaging an expensive machine.

“We want to train and evaluate robots in the digital world,” Zhang said.

Digitize NYC begins by scanning real environments across New York, including spaces that connect indoor and outdoor settings. Zhang and her collaborators then use technologies including 3D Gaussian splatting and neural rendering and generative models to turn those scans into richer, geometrically grounded simulations. A single scan, for example, can be transformed to represent different times of day or seasons, while simulated people, dogs and wheelchairs can be added to create more realistic environments.

The goal is a “real-to-sim-to-real” process. Robots can learn from real-world observations, be trained and tested in diverse digital environments, and then transfer what they learn back to the physical world.

A key advantage of the team's approach is that the simulations combine high-quality visual information with accurate three-dimensional geometry. That allows researchers to create diverse visual conditions without sacrificing the reliable spatial information obtained from the original scans.

At Tandon, Zhang sees opportunities to extend that work through collaborations across the university's robotics community. She is particularly interested in world models, AI systems that can represent not only the current state of an environment, but also how it may change over time. Such models could help robots imagine possible future states, continuously update their understanding of changing environments and make better decisions.

Zhang will also bring her enthusiasm for teaching to Tandon. This fall, she plans to teach Mathematics for Robotics to graduate students. She previously taught courses in robot learning and robot perception, experiences that have reinforced her belief that teaching works in both directions.

“When students ask you a new question, it can push you to see the problem from a different perspective,” she said. “I like the interaction with the students.

For Zhang, that combination of research, teaching and collaboration is part of the appeal of becoming a faculty member. Whether the problem involves an ancient broken bowl or a robot navigating a crowded New York sidewalk, her work is ultimately focused on helping AI understand, imagine and interact with the physical world.