DARPA Contract to Fund Exploration of Hard-to-Find Information on the Web

Juliana Freire, a professor in the Department of Computer Science and Engineering of NYU Polytechnic School of Engineering, has been awarded $3.6 million by the U.S. Defense Advanced Research Projects Agency (DARPA) to develop methods to locate and explore hard-to-find information both on the web surface and on the deep web, the parts of the web not indexed by standard commercial search engines.

The contract is part of DARPA’s Memex program, a three-year research effort to develop software that will enable domain-specific indexing of open, public web content and domain-specific search capabilities. With Memex, DARPA initially seeks to address a key Defense Department mission: fighting human trafficking. Human trafficking is a factor in many types of military, law enforcement and intelligence investigations and has a significant web presence to attract customers.

Freire and colleagues, including Ari Juels from Cornell Tech and Torsten Suel, a professor in the NYU School of Engineering Department of Computer Science and Engineering, are working on the design scalable techniques to address the shortcomings of traditional search engines for specific information needs. Their research aims to make focused crawlers more usable and able to efficiently handle a wide range of search tasks, including adversarial crawling, which is needed for tasks that involve tracking criminal activities. Another important goal of their research is to enable the integration of crawling with search and data analysis in a transparent and reproducible fashion.

The deep web has long been a focus for Freire; in 2009, she created DeepPeep.org, a National Science Foundation-supported search engine specializing in deep-web sites.  DeepPeep was featured in the front page of the New York Times technology section.

In addition to her faculty position at the School of Engineering, Freire is the director of graduate studies for the NYU Center for Data Science and an associated faculty member of the Center for Urban Science and Progress (CUSP). 

U.S. Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)