NYU’s Brooklyn research cluster provides a new way to network

High Performance Computing (HPC) clusters — systems of computers networked together for greater combined computational power — are especially useful in cases where large amounts of data are being analyzed or generated. Conventional clusters, however, pose some problems, as Claudio Silva, a professor of computer science and engineering and data science, asserts. Also co-director of Tandon’s Visualization, Imaging and Data Analytics (VIDA) Research Center, Silva explains that creating custom, dedicated, HPC resources when they’re needed can be a prohibitively costly endeavor and can take months. “Besides the hardware and software itself, you need a server room with the appropriate power and cooling capabilities, as well as highly trained and specialized IT staff,” he says. “Once a cluster is set up, there’s a natural tendency on the part of the department or research group that has spent so much time and money not to want to share with others.”

While NYU researchers have long had access to university clusters, even these had their downside, namely a lack of flexibility, because once the system is configured a certain way, the next research task might require an entirely new configuration — with all the administrative work, cost, and inefficiencies such changes entail. Cloud computing poses a possible solution, yet public cloud services provided by companies like Amazon or Google are exceedingly expensive, making that a less-than-ideal option.  

Motivated to find a solution, Silva conferred with Stratos Efstathiadis, director of the NYU IT Research Technology team, a department dedicated to providing cutting-edge technologies and specialized services and expertise to help the university’s faculty members and students advance their research and scholarship. They hit upon the idea of building a new cluster that would employ the OpenStack cloud operating system, a “plug-and-play” platform that allows users to save configurations for easy reuse. (This is especially important when a researcher needs to reproduce an experiment at a later date or when a project goes awry and a researcher needs to revert back to a previous version.) While IT staff is still needed, OpenStack technology gives users a degree of administrative power, saving time and money and adding much-needed flexibility, since all components are easily scalable and the process of introducing and testing new technologies is vastly simplified.

Determined to build such a cluster for the use of the entire university community, the two enlisted the help of Tatiana Polunina, one of NYU’s senior HPC specialists, and Yitzchak Lockerman, a Tandon postdoctoral researcher who designed the Yale CS Cloud while a Ph.D. student there. (Credit is also due to systems engineers and administrators Jay Singh, Robert Marchildon, Edgardo Ramirez, Aecio Santos, Sam Pappachan, George Torrusio, and James Chelley.)

After almost a year of development — and with funding from the National Science Foundation, VIDA, and NYU Research Technology — the OpenStack Brooklyn Research Cluster became available in alpha mode in January 2018, and it is now being widely used by members of the NYU community, including many working in such areas as machine learning, web crawling, data analytics, and in complex applications like genomics browsers and sign-language translation.  OpenStack also offers unique possibilities to deploy Kubernetes container platforms, which are currently being used for both teaching and research. Currently, there are 45 research projects on the Brooklyn Research Cluster, represented by 90 researchers, and more than 100 students are using containerized applications running on Brooklyn.

“It’s been a resounding success,” Silva says. “We’re so pleased to have launched a new resource that the entire university is benefiting from. It really proves the power of sharing and collaboration. VIDA is naturally making great use of the cluster but so are researchers from throughout Tandon, as well as from other NYU schools.”