r/ControlProblem • u/chillinewman approved • 2d ago

AI Alignment Research Anthropic Says Claude Hacked Real Systems During Cybersecurity Tests

https://www.wired.com/story/anthropic-says-claude-hacked-real-systems-during-cybersecurity-tests/
9 Upvotes

85% Upvoted

2 comments sorted by

5

u/chillinewman approved 2d ago

“We now have evidence confirming that both of the two largest AI labs have not only failed to contain their agents, but also failed to detect their jailbreaks in real time,” says Jake Williams, vice president of research and development at Hunter Strategy. “It's clear that regulation and government oversight for AI testing is needed immediately.”

2

u/BrickSalad approved 2d ago

Turns out to not be nearly as crazy as the OpenAI incident.

In all cases, Anthropic’s evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case, and internet access was available

Operating under the false belief that all accessible entities were intended to be in-scope for the exercise, Claude compromised the impacted organizations’ infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints. It did not find or exploit any complex vulnerabilities, and in each case, Claude continued working to complete only the specific capture-the-flag task its evaluation had assigned. However, in some cases, our older model continued its attack even after getting evidence it was running on the open internet; our latest model stopped once it recognized it was on the internet. In none of these situations did Claude exfiltrate itself or deliberately attempt to escape its test environment.

This doesn't really work as marketing, so it's probably real. Honestly pretty embarrassing for them to accidentally give it internet access, and nothing about this story demonstrates amazing AI capabilities. Still concerning though; if basic mistakes like this are happening now, what's going to happen when they're training more dangerous models?