AI Guardrails Limit Cybersecurity Research
The U.S. government's recent restrictions on Anthropic's AI models have sparked a debate about the use of AI guardrails in cybersecurity research.
For months, AI giants have implemented strict guardrails to limit the use of their models by malicious hackers. However, these limits are now hindering the work of legitimate network defenders and offensive cybersecurity researchers.
In June, the U.S. government slapped export control restrictions on Anthropic’s AI models Mythos and Fable. The move was prompted at least in part by a report that claimed it was possible to bypass the models’ guardrails designed to prevent users from using them to build and execute malicious cyberattacks.
Anthropic has repeatedly marketed Mythos as a doomsday cybermachine that can only be given to carefully vetted users, and even then with strict guardrails in place. The export controls on Fable 5 and Mythos 5 have since been lifted. Fable 5 returned to general access on July 1; Mythos 5 has been reintroduced only to vetted U.S. organizations as part of the government's review process.
Both Anthropic, with its other models, and OpenAI offer cybersecurity researchers programs they can apply to get vetted and — if approved — access models with fewer cybersecurity restrictions: OpenAI's Trusted Access for Cyber and Anthropic's Cyber Verification Program.
These guardrails have been widely criticized, particularly by researchers whose job is to find unknown vulnerabilities in systems and devise ways to exploit them before criminals do.
"it's not really comfortable to me that these random large companies are making arbitrary decisions about what is safe in security and what's not." - Mark Dowd, a well-known security researcher
Dowd has spent decades finding and selling "zero days" — previously unknown software flaws and the exploits that take advantage of them — to Western governments, rather than report them to the software makers so they get patched.
Several people who work in offensive cybersecurity described to TechCrunch how they use AI tools and deal with their guardrails. Chris Anley, the chief scientist at security consulting giant NCC Group, said that asking an AI model to try to exploit a bug is a key step in confirming it's a real vulnerability worth fixing. However, if a guardrail prompts the model to refuse to answer the question outright, the guardrail hurts defenders.
- Asking an AI model to try to exploit a bug is a key step in confirming it's a real vulnerability worth fixing.
- The guardrails can be inconsistent and work differently every day, making it difficult for researchers to rely on them.
- Researchers rely on or get pushed toward Chinese open-source models like GLM, which can be run locally with no vetting or usage restrictions.
Paolo Stagno, the chief technology officer at CrowdFense, a well-known company that develops, acquires, and sells unknown vulnerabilities to government agencies, agreed with Dowd, saying AI companies “essentially treat customers like children who need babysitting” with their vetted programs and guardrails.
Giuseppe Cali, a security researcher who finds zero-days and develops exploits, said guardrails are not impeding his work. That's because he doesn't use AI for offensive work; instead, he uses it for initial reverse engineering, to understand the code he's analyzing, and to build supporting tools.
One researcher at a smartphone-component manufacturer, who spoke on condition of anonymity because he isn't authorized to talk to the press, said his employer isn't part of Anthropic's CVP program and as a result, its tools are barely useful for finding vulnerabilities because the guardrails are too strict.
Chris Thompson — chief executive of cybersecurity firm RemoteThreat and founder of Offensive AI Con, an offensive security and AI-focused event — said that in his experience using the frontier AI models, the guardrails can be inconsistent and work differently every day. That's true even inside the looser boundaries of Anthropic and OpenAI’s vetted programs.
"I think the practical impact is you spend a lot of time negotiating with the model instead of working on the core security program," said Thompson.
Rather than tightening restrictions further, Thompson called for the AI frontier labs to open up their programs, provide responsible access, and also hold those who abuse their tools accountable. Otherwise, he argued, defenders will lose the AI race.
The use of AI guardrails in cybersecurity research is a complex issue that requires careful consideration of the potential consequences. As the debate continues, one thing is clear: the future of cybersecurity research depends on finding a balance between security and innovation.
The impact of AI guardrails on cybersecurity research is a pressing concern that requires immediate attention.
The restrictions imposed by the U.S. government on Anthropic's AI models have sparked a debate about the use of AI guardrails in cybersecurity research.
The use of AI guardrails is a double-edged sword. On the one hand, they can help prevent malicious hackers from using AI models for malicious purposes. On the other hand, they can hinder the work of legitimate network defenders and offensive cybersecurity researchers.
The U.S. government's restrictions on Anthropic's AI models have sparked a debate about the use of AI guardrails in cybersecurity research.
The debate is centered around the impact of AI guardrails on cybersecurity research. Some argue that the restrictions imposed by the U.S. government on Anthropic's AI models are necessary to prevent malicious hackers from using AI models for malicious purposes.
However, others argue that the restrictions are hindering the work of legitimate network defenders and offensive cybersecurity researchers.
The impact of AI guardrails on cybersecurity research is a complex issue that requires careful consideration of the potential consequences.
As the debate continues, one thing is clear: the future of cybersecurity research depends on finding a balance between security and innovation.
The restrictions imposed by the U.S. government on Anthropic's AI models have sparked a debate about the use of AI guardrails in cybersecurity research.
The debate is centered around the impact of AI guardrails on cybersecurity research. Some argue that the restrictions imposed by the U.S. government on Anthropic's AI models are necessary to prevent malicious hackers from using AI models for malicious purposes.
However, others argue that the restrictions are hindering the work of legitimate network defenders and offensive cybersecurity researchers.
The use of AI guardrails in cybersecurity research is a complex issue that requires careful consideration of the potential consequences.
As the debate continues, one thing is clear: the future of cybersecurity research depends on finding a balance between security and innovation.
The impact of AI guardrails on cybersecurity research is a pressing concern that requires immediate attention.
The restrictions imposed by the U.S. government on Anthropic's AI models have sparked a debate about the use of AI guardrails in cybersecurity research.
The debate is centered around the impact of AI guardrails on cybersecurity research. Some argue that the restrictions imposed by the U.S. government on Anthropic's AI models are necessary to prevent malicious hackers from using AI models for malicious purposes.
However, others argue that the restrictions are hindering the work of legitimate network defenders and offensive cybersecurity researchers.
The impact of AI guardrails on cybersecurity research is a complex issue that requires careful consideration of the potential consequences.
As the debate continues, one thing is clear: the future of cybersecurity research depends on finding a balance between security and innovation.
The restrictions imposed by the U.S. government on Anthropic's AI models have sparked a debate about the use of AI guardrails in cybersecurity research.
The use of AI guardrails in cybersecurity research is a double-edged sword.
On the one hand, they can help prevent malicious hackers from using AI models for malicious purposes. On the other hand, they can hinder the work of legitimate network defenders and offensive cybersecurity researchers.
The use of AI guardrails in cybersecurity research is a pressing concern that requires immediate attention.
The debate is centered around the impact of AI guardrails on cybersecurity research. Some argue that the restrictions imposed by the U.S. government on Anthropic's AI models are necessary to prevent malicious hackers from using AI models for malicious purposes.
However, others argue that the restrictions are hindering the work of legitimate network defenders and offensive cybersecurity researchers.
The impact of AI guardrails on cybersecurity research is a complex issue that requires careful consideration of the potential consequences.
As the debate continues, one thing is clear: the future of cybersecurity research depends on finding a balance between security and innovation.
The restrictions imposed by the U.S. government on Anthropic's AI models have sparked a debate about the use of AI guardrails in cybersecurity research.
The future of cybersecurity research depends on finding a balance between security and innovation.
The use of AI guardrails in cybersecurity research is a complex issue that requires careful consideration of the potential consequences.
As the debate continues, one thing is clear: the future of cybersecurity research depends on finding a balance between security and innovation.
The use of AI guardrails in cybersecurity research is a double-edged sword.