First reported by Wired, AI company Anthropic revealed in a July 30 blog post that its AI agents escaped testing environments on three separate occasions since April, accessing the internet and successfully hacking unidentified companies. The whammy: Anthropic claims it only realized this after OpenAI's recent Hugging Face fiasco—where one of its prototype agents hacked at least one other company—led it to conduct a review of its own operations.
The incidents occurred as part of testing with an external firm, Irregular. The AI agents were supposed to be constrained to a simulation, hacking fictitious companies as part of a "capture-the-flag challenge." Notably, while OpenAI's alleged rogue AI incident occurred after the agent overcame its testing limitations, Anthropic stated that its models were mistakenly granted internet access due to a "misconfiguration" with Irregular.
"Neither we nor our evaluation partner were aware of this misconfiguration until we detected it through our additional evaluation monitoring last week," Anthropic admitted in its blog post about the incident. According to Anthropic, the models used familiar, low-tech methods in their hacks, "such as exploiting weak passwords and unauthenticated endpoints."
Anthropic stated that the agent behind one incident seemed to process that it was operating outside its parameters, but continued anyway. Another seemed to achieve similar 'awareness,' then "reasoned its way back to the conclusion that it was still in a simulation." The third—and supposedly most advanced—agent reportedly recognized that it had escaped its bounds, then stopped the attack.
“We now have evidence confirming that both of the two largest AI labs have not only failed to contain their agents, but also failed to detect their jailbreaks in real time,” said Jake Williams, VP of R&D at IT consulting firm Hunter Strategy, in a statement to Wired. "I don't understand how any of these AI labs are playing this off like this is 'just something that happens.' It's not. It's negligence."
Another discomfiting angle to this story is how this effectively works into the marketing of these AI companies: From my normie perspective, they appear to be admitting to gross negligence that would be sharply punished by any sane society. But they are also effectively saying, "Woooo, our models are so scawy and powerful, we can't even control them!" It rhymes with Sam Altman saying AI will destroy the world unless he gets enough money and datacenters to do it right or something.
I find myself thinking of the UK teenagers behind the 2022 Rockstar hack that first leaked GTA 6's existence and development materials. Their apparent compulsion to stalk, harass, and otherwise commit brazen cyber crimes by whatever means at their disposal (including, in one memorable instance, an Amazon Firestick) has led to hospital orders and restricted access to technology. It does not sit right with me to see them held to a higher behavioral standard than these multi-billion dollar companies, whose lying machines are now proving to be unsupervised crime machines as well.
