NYU Researchers Are Building AI to Piece Together Broken Archaeological Finds

A team of NYU engineers and archaeologists is working on open-source AI tools that piece together broken stone tools, pottery and bones and fill in what's missing, so that fragments sitting unmatched in collections can become evidence about how people in the past lived.

The project, "Fragmented Legacy: Turning Broken 3D Materials into Archaeological Interpretations," includes partners at the University of Tübingen and is supported by a three-year grant from Schmidt Sciences. It is one of 19 projects selected for this year's Humanities and AI Virtual Institute (HAVI) awards.

The problem it targets is basic to archaeology. Stone tools, pottery, animal bones and human skeletal remains are often excavated broken, and putting the fragments back together, a process called refitting, is slow enough that most collections are never systematically refitted. Reported success rates run from about 1 percent for ceramics to between 4 and 26 percent for bones.

Chen Feng, Institute Associate Professor at NYU Tandon School of Engineering and founding co-director of the NYU Center for Robotics and Embodied Intelligence (CREO), leads the project with Jing Zhang, a CREO Faculty Fellow.

They are joined by Radu Iovita and Scott Williams, associate professor and professor, respectively, in NYU’s Department of Anthropology, and Raiko Krauß, Professor of Central and Southeast European Prehistory at the University of Tübingen.

Collaborators include Leiden University, Arizona State University, and the Max Planck Institute for Evolutionary Anthropology.

"My lab teaches machines to understand 3D shapes in the real world, where things are messy and incomplete," said Feng. "Broken archaeological material is one of the hardest versions of that problem. The pieces are worn, many are missing, and there's no picture on the box. Our job is to build AI that can reason through that uncertainty, and our archaeology colleagues tell us whether the answers make sense."

Refitting matters because a single match between two fragments can carry a lot of information.

If a stone flake found in one part of a site fits back onto a core found in another, researchers learn how people moved through that space, how tools were made step by step, or whether the site's layers were disturbed after the fact. Animal bones that refit can show how an animal was butchered and how meat was shared, and human fossil refits can reveal important anatomical features.

Without refits, many of those questions go unanswered.

The team will start by building a collection of at least 100,000 real and digitally simulated broken objects, pairing 3D models with photographs, descriptions and catalog data, with simulated breaks that follow how stone, clay and bone actually fracture.

The researchers will then develop AI models that work out where each fragment belongs while generating a plausible version of the complete object, and that reason across shape, images and text so archaeological knowledge can rule out unlikely reconstructions.

Archaeologists will stay in the loop throughout, comparing competing reconstructions and teaching the system which ones hold up.

“Refitting is fun, but we rarely have time to do it right. In museums, one can often only look at one drawer at a time, so it’s impossible to lay out an entire collection at once,” said Iovita. “Refitting objects between sites to show direct relationships between them has only ever been done twice. My dream is to use AI to do this on a large scale – it will revolutionize the field!”

The project grew out of a collaboration between Feng’s AI4CE Lab and Iovita’s lab that began about four years ago with LUWA, which used AI to classify microscopic wear on stone tools. Zhang, then a postdoctoral researcher working in both labs, connected the two groups.

In 2024, Zhang asked Iovita about archaeology’s most pressing technical problem, and Iovita pointed to refitting. That led to GARF, which reassembles broken objects, and CRAG, which also fills in missing pieces, and laid the groundwork for the HAVI award.

“Archaeologists are like Sherlock Holmes, piecing together clues to understand the past,” said Zhang. “We want Fragmented Legacy to be their Watson. It can match fragments and suggest possible reconstructions, so researchers have more time to interpret the evidence and ask new questions.”

All datasets, model weights and benchmarks from the project will be released under permissive open-source licenses, so university researchers and cultural resource management (CRM) firms, which carry out much of the archaeological work required ahead of construction in the U.S. and abroad, can use them directly. The team expects the methods to be useful beyond archaeology, in other scientific fields and in practical repair and restoration.

The award follows Schmidt Sciences' August announcement that NYU Tandon’s Dharik Mallapragada is part of one of the projects funded through its new Decarbonization and Energy Virtual Institute, which will develop faster computer models of how heavy industry can cut carbon emissions.