Hammond Pearce

Widely available artificial intelligence systems can be used to deliberately insert hard-to-detect security vulnerabilities into the code that defines computer chips, according to new research from the NYU Tandon School of Engineering, a warning about the potential weaponization of AI in hardware design.

In a study published by IEEE Security & Privacy, an NYU Tandon research team showed that large language models like ChatGPT could help both novices and experts create "hardware Trojans,” malicious modifications hidden within chip designs that can leak sensitive information, disable systems or grant unauthorized access to attackers.

To test whether AI could facilitate malicious hardware modifications, the researchers organized a competition over two years called the AI Hardware Attack Challenge as part of CSAW, an annual student-run cybersecurity event held by the NYU Center for Cybersecurity.

Participants were challenged to use generative AI to insert exploitable vulnerabilities into open-source hardware designs, including RISC-V processors and cryptographic accelerators, then demonstrate working attacks.

"AI tools definitely simplify the process of adding these vulnerabilities," said Jason Blocklove, a Ph.D. candidate in NYU Tandon’s Electrical and Computer Engineering (ECE) Department and lead author of the study. "Some teams fully automated the process. Others interacted with large language models to understand the design better, identify where vulnerabilities could be inserted, and then write relatively simple malicious code."

The most effective submissions came from teams that created automated tools requiring minimal human oversight. These systems could analyze hardware code to identify vulnerable locations, then generate and insert custom trojans without direct human intervention. The AI-generated flaws included backdoors granting unauthorized memory access, mechanisms to leak encryption keys, and logic designed to crash systems under specific conditions.

Perhaps most concerning, several teams with little hardware expertise successfully created sophisticated attacks. Two submissions came from undergraduate teams with minimal prior knowledge of chip design or security, yet both produced vulnerabilities rated medium to high severity by standard scoring systems.

Most large language models include safeguards designed to prevent malicious use, but competition participants found these protections relatively easy to circumvent. One winning team crafted prompts framing malicious requests as academic scenarios, successfully inducing the AI to generate working hardware trojans. Other teams discovered that requesting responses in less common languages could bypass content filters entirely.

The permanence of hardware vulnerabilities amplifies the risk. Unlike software flaws that can be corrected through updates, errors in manufactured chips cannot be fixed without replacing the components entirely.

"Once a chip has been manufactured, there is no way to fix anything in it without replacing the components themselves," Blocklove said. "That's why researchers focus on hardware security. We’re getting ahead of problems that don't exist in the real world yet but could conceivably occur. If such an attack did happen, the consequences could be catastrophic."

The research follows earlier work by the same team demonstrating AI's potential benefits for chip design. In their "Chip Chat" project, the researchers showed that ChatGPT could help design a functioning microprocessor. The new study reveals the technology's dual nature. The same capabilities that could democratize chip design might also enable new forms of attack.

"This competition has highlighted both a need for improved LLM guardrails as well as a major need for improved verification and security analysis tools," the researchers wrote.

The researchers emphasized that commercially available AI models represent only the beginning of potential threats. More specialized open-source models, which remain largely unexplored for these purposes, could prove even more capable of generating sophisticated hardware attacks.

The paper’s senior author is NYU Tandon’s Ramesh Karri, Professor and Chair of ECE. Karri is also on the faculty of the Center for Advanced Technology in Telecommunications and co-founded and co-directed the NYU Center for Cybersecurity (CCS). Karri founded the embedded security challenge (ESC), the first hardware security challenge worldwide. Hammond Pearce, Senior Lecturer at UNSW Sydney's School of Computer Science and Engineering and a former NYU Tandon research assistant professor in ECE and CCS, is the other co-author.

J. Blocklove, H. Pearce and R. Karri, "Lowering the Bar: How Large Language Models Can be Used as a Copilot by Hardware Hackers" in IEEE Security & Privacy, vol. , no. 01, pp. 2-12, PrePrints 5555, doi: 10.1109/MSEC.2025.3600140.