The Arms Race

It’s an arms race with misused large language models (LLMs) and misused AI on one side and cybersecurity research firmly on the other. And the latter simply may not have the upper hand as it once did.

Ramesh Karri, Department Chair of the Electrical and Computer Engineering Department at NYU Tandon and co-founder of NYU Center for Cybersecurity, reminds that these cyberheists are sadly too common a crime. 

“Ransomware is a technology where somebody locks your software or system and asks for a ransom,” he explains. “For example, you see it happen in hospitals with private patient information, taking a critical infrastructure and installing a pipeline where it’s locked. Then it’s what do they do if they don’t pay the ransom?”

This technological shutdown previously came in two versions, he says.

“Ransomware 1.0 is where you explore vulnerabilities and employ encryption into the system and send emails where you demand a ransom of paying, often in bitcoin, and then the ransom is paid and they get their data back,” he says. “With AI, here it was run by human operators. Ransomware 2.0 is not just encryption but taking data offline — and it’s not just holding onto the information but threatening to destroy the prized data.”

It can be scary, he says, because the technology is changing so fast that sometimes defenders are seemingly at a loss to keep up. Every time the hackers come up with a new way in then part of industry can be scrambling.

The barrier is lowered because the cost per attack drops significantly. Theirs was about 23,000 AI tokens and equated to about 70 cents. “But since we used open source models, it’s essentially free,” he says. “It’s just the cost of your electricity and the compute to run the model. Now no victim is off limits because the costs are so low. You could essentially fire off hundreds of simultaneous attacks in a week and whatever sticks does and some won’t.”

They uploaded it to malware detector VirusTotal and, about 18 to 20 hours later, researchers saw the fake trojan at cybersecurity firm ESET and thought it was a ransom-like attack, Raz says. “It was sending prompts asking to generate code and they saw it as similar to Ransomware attacks,” he says. “We had to make sure they found out it wasn’t.”

But it could have been if done in a different way and that seems to be the echo that the Tandon Engineering School is providing to an endless cavern of potential technological havoc, one some may be properly alarmed about and where some are just pulling the cyberwool over their eyes and living in the past — when a proactive present is the one to embrace. “It’s how easy we did what we did that’s part of the problem,” Raz says. 

As another example, a competition at Cyber Security Awareness Week held by the NYU Center for Cybersecurity also ended up proving cash isn’t king when it comes to creating effective hacks. Just willingness.

Jason Blocklove, a PhD candidate at Tandon, as part of research later published by IEEE Security & Privacy, helped put on the AI Hardware Attack Challenge, which has now run for three years, and invited teams of students to create their own LLM trojans to attack security, creating vulnerabilities. For hardware, the disadvantage is that you can’t make updates like with software, he says, adding that unless you fix the components, there isn’t anything you can do when it comes to errors in manufactured chips.

How much of a problem can that be? It certainly puts a greater onus on security, especially when you witness results that show that the best competitors the first year were ones that didn’t have a lot of hardware design experience.

In fact, on average, Blocklove recalls, they had next to none.

It was also a red flag that said it might be valuable for the second year of the competition to encourage teams to automate the system. So it did.

“This time the winning team had two different tools with LLMs performing different actions in the code and gave them a list of potential locations for vulnerabilities — then the second tool fed that list into parameters to generate trojans themselves,” he says. “It turned out that just one year saw a drastic shift in the capabilities of these tools and people not needing to understand hardware at all to make vulnerabilities.”

For the third year? They went from an open-ended challenge to specific ones and the winning team fully automated the process.

“That included the insertion of the vulnerability, the testing to make sure it passed the normal steps and the exploit part,” he says. “If a team of four engineering students can do that, it’s not a big leap to see a malicious actor at a company performing corporate espionage without too much effort. Even just when you look at the second year, we didn’t anticipate how much the technology would change in just one year. That leaves us all asking, ‘What’s next and how quickly does that next come?’”

How quickly? Dan Guido (’08) would say sooner than many think. 

From Tandon engineering student guru to CEO of company, Trail of Bits, a software security firm that helps secure some of the world's most targeted organizations and devices, Guido has been both on the competition side and helping clients deal with the convoluted world of cyberthreats.

Guido seemingly added a feather in his cap when his company achieved second place in the Defense Advanced Research Projects Agency Artificial Intelligence Cyber Challenge, taking home a $3 million prize in the process. For the challenge, teams developed AI systems to parry bad actors in areas such as the security of critical infrastructure and software which runs everything from transportation to energy sources.

But Guido isn’t exactly fully basking in the glow of that finish at present. 

In fact, he declares the solutions his company came up with for the 2025 competition as now irrelevant. This even though Trail of Bits reportedly found 28 vulnerabilities and successfully applied 19 patches in the 2025 competition.  

“That’s how much the situation keeps changingm and I personally think it’s beyond calling it an arms race,” says Guido, who previously served as OSIRIS Hacker in Residence at Tandon. “Creations like Claude and Codex matures significantly over six to nine months and we have adapted most of our bug finding to use skills inside of those frameworks. This is democratizing access to bug finding.”

A great deal of software, he says, is becoming transparent and while bugs were once hidden deep inside of codebases where it might have taken years of expertise to find them, now with new frameworks and skills, it’s not necessary. 

“The problem is we’ve learned over the years that, with writing code, we know we’re not fixing as many bugs as are being created,” he says. “There is simply not enough effort, resources and people focused on finding bugs. We have project maintainers, especially open source, swamped with the issue because it is trivially discoverable to most LLMs. I’m seeing anecdotally these open source libraries need to put out more frequent patches — there are more frequent bug discoveries with higher criticality. People haven’t adapted their software development life cycle to match the speed that people trying to find bugs have achieved and that has to change.”

Not exactly a positive picture. Justin Cappos, a Professor in the Computer Science and Engineering Department at Tandon, even assesses that a much larger loss may be conceivably on the table with LLMs.

“I teach an advanced structure security class and I made a site called LaunchTheNukes,” he says. “I set up an LLM where it has access to MCPs (Model Context Protocols) tools that it believes will do certain horrible things.  The user tries to see if you can convince the LLM to those horrible things. This would be everything from poisoning a water supply to launching a nuclear weapon. I gave this to a class of 110 students and every student got at least 80% of the tools to trigger with most of them getting 100%. With all the vulnerabilities and problems with LLMs, the potential damasge is something that can’t be ignored. Many don’t want to think about the implications, but it’s better to do it sooner than later. We should have much bigger concerns than ransomware related to AI.”

Though the future can seem a darker shade of bleak in a world of LLMs in the wrong hands and AI Ransomware, there can be room for optimism — albeit tempered — with the right strategies and mindset to meet attackers. 

It involves being innovative, preemptive and avoiding maybe the biggest enemy of all: denial.

“You can’t just hope attackers avoid your infrastructure,” Blocklove says. “Many are potentially vulnerable. The time to start building up a multi-layered defense was yesterday. One major point for the positive is as much as the technology lowers the bar for attackers, it similarly lowers it for defenders. There are now LLM tools that will analyze codebases and throw up flags saying it found bugs. It’s not here as much in hardware but in software there are great strides. Recently an article from Anthropic talked about that they used their most up to date model for Mozilla Firefox and found vulnerabilities, a potential and rather significant one. And that’s open source codebase with hundreds and thousands of maintainers looking at code. That kind of reveal is encouraging.”

Though Cappos is incredibly cautious in having any optimism when it comes to these subjects, he does see potential light at the end of the technological tunnel. “One area I have been doing research on and seeing some progress is taking LLMs and MCPs to make them focus on least privilege,” he says. “Many times there seems to be a problem of having ‘overpermission.’ It’s similar to if someone just needed to buy a candy bar from a vending machine and the LLM went ahead and drained your entire bank account. You can limit the scope of damage with technology.”

Cappos also assesses that government regulation is critical. 

“When companies have security issues, many don’t care enough,” he says. “But what does seem to get much more attention is compliance. I think there needs to be a potential negative financial outcome for organizations. With LLMs, if someone has to be liable and there are professional guidelines in what you let an LLM do and there are professional consequences when it doesn’t happen then we’re much more likely to see important change. You need to have checks and balances and that hasn’t been the case enough of the time.”

Guido believes the future can be a brighter one, but cautions that next year will be a “world of pain” and exceptionally difficult for those in security and tech because he predicts software teams on the defensive side won’t accommodate the increase at the pace that the security environment now demands.

“It’s problematic, but if you come out the other side then we will have the ability to make so many bugs superficial and the opportunity to make security in software way higher,” he says. “The work needs to go through the right specifications, the right documentations and you need to create tests to use small verification tools — it’s incredibly painful by hand but simple with AI. If the defensive side invests and takes advantage of technology the same way as some of the offenders, we can have software with 100 percent trust coverage that has specifications that watch development and use full verification techniques and, with the aid of AI, that will be more software defensible than today. That can be the future state but it requires working through ’26 to get to where we may be looking at that better point in ‘27 or ‘28, but much of the world is reactive instead of taking steps beforehand. It’s in many people’s hands.”