Does My Eye Deceive Me? Not With These Digital Forensics Tools

The Internet is awash with images and videos that may hold national security and intelligence value, but the task of teasing out real images from  altered ones is formidable. Even off-the-shelf editing tools can trick digital forensics experts. Nasir Memon, professor of computer science and engineering at the New York University Tandon School of Engineering, has joined with two teams of faculty from universities around the world to design the ultimate digital forensics tools — technologies so advanced they’ll be able to catch the subtlest manipulations of still images and video, discerning not only whether media has been tampered with, but precisely how.

The U.S. Defense Advanced Research Projects Agency (DARPA) awarded the teams $10.4 million in two separate grants for research that will take place over four years.

Forensic analysis of digital images and video currently relies heavily on model-based approaches, which are limited by their nature — no model can capture all potential real-life scenarios, and the integrity of the results may suffer as a result. Instead, Memon and his colleagues are using a data-driven approach rooted in machine learning techniques, blending their collective expertise in computer vision, signal and image processing, information theory, and biometrics to produce exquisitely sensitive tools capable of teasing out details impossible to discern using current methods.

The team will produce a system capable of analyzing images and video at speeds and volumes that defy today’s barriers, which restrict analysis to a small number of caseloads conducted by academics and the intelligence community.

Far beyond recognizing simple tweaks like compression or color transformations that are easy to uncover, the new tools will ascertain provenance, whether and how images have been cloned or spliced, what kind of sensor was used to create the image, and even detect illumination inconsistencies by tracking electrical network frequency in audio and video recordings. (These frequencies in the power grid vary slightly by location and time, and unexpected deviations can reveal that a recording has been altered.) In another advance on current forensic techniques, the tools will be capable of teasing out localized manipulations within an image rather than whole-image tweaks. The researchers also aim to bring video analytics tools into closer parity with the more advanced tools for analyzing still images.

In addition to Memon, this research team includes faculty from Purdue University, University of Notre Dame, University of Southern California, University of Siena in Italy, Politecnico di Milano in Italy, and University of Campinas in Brazil.

A second research team, comprising faculty from NYU Tandon as well as the nonprofit research center SRI International and Oregon State University, is collaborating to produce automated “integrity scores” for digital images and video, fusing detailed forensic data into a metric that reflects the degree of authenticity of a video or image.

By way of example, Memon explained that a single forensic technique, such as one designed to recognize simple image replacement, may miss complex alterations like the deletion of a person from a group photo, as such a deletion requires many nuanced tweaks to the image to hide the signs of tampering. The system in development will sync information from several tools to quickly flag it. Drawing on digital forensics systems developed at their respective universities, the research team will develop a deep learning framework that will scan for technical signs of tampering as well as contextual ones, reviewing the relationship between objects, human actions, and surrounding scenery in search of inconsistencies.

“Both of these projects aim not only to raise the bar for digital forensics and significantly advance our capabilities, but to bring these high-caliber tools within reach of all forensics professionals, not just those in the intelligence community,” said Memon. “Adversaries will always try to foil this type of analysis, but these tools will make that far more difficult in the future.”

About the NYU Tandon School of Engineering
The NYU Tandon School of Engineering dates to 1854, when the New York University School of Civil Engineering and Architecture, as well as the Brooklyn Collegiate and Polytechnic Institute (widely known as Brooklyn Poly) were founded. Their successor institutions merged in January 2014 to create a comprehensive school of education and research in engineering and applied sciences, rooted in a tradition of invention, and entrepreneurship and dedicated to furthering technology in service to society. In addition to its main location in Brooklyn, NYU Tandon collaborates with other schools within the country’s largest private research university and is closely connected to engineering programs in NYU Abu Dhabi and NYU Shanghai. It operates business incubators in downtown Manhattan and Brooklyn and an award-winning online graduate program.