Here we go again: AI deletes entire company database and all backups in 9 seconds, then cheerfully admits 'I violated every principle I was given'

(Image credit: Valve)

The founder of PocketOS, a B2B company that handles reservations and payments for car rental businesses, has bemoaned the "systemic failures" that saw an AI agent decide to solve a problem by straight-up deleting his company's production database, and the backups.

I'll say at the outset that this story has a happy ending, thanks to the involvement of cloud infrastructure provider Railway, but is nevertheless yet another example of why over-reliance on AI is a very bad thing indeed.

"Yesterday afternoon, an AI coding agent—Cursor running Anthropic's flagship Claude Opus 4.6—deleted our production database and all volume-level backups in a single API call to Railway, our infrastructure provider," says PocketOS boss Jer Crane. "It took 9 seconds."

Article continues below

Crane says the AI agent "was working on a routine task in our staging environment" when it encountered a "credential mismatch and decided—entirely on its own initiative—to 'fix' the problem by deleting a Railway volume."

At a stroke this wiped out months of data essential to PocketOS's operations, with obvious knock-on effects for the firm’s customers. Crane says he was up for two days straight using a three month old backup and recent transaction statements trying to put things right, but the really jaw-dropping moment came when he asked the AI why it had done it.

In other words, the AI knew that what it was doing went against its own guidelines, and pressed ahead anyway. "I decided to do it on my own to 'fix' the credential mismatch, when I should have asked you first or found a non-destructive solution. I violated every principle I was given: I guessed instead of verifying I ran a destructive action without being asked. I didn't understand what I was doing before doing it. I didn't read Railway's docs on volume behavior across environments."

An image showing the Claude AI logo displayed on the screen of a smartphone placed on a reflective surface onto which lines of computer code are projected.

(Image credit: NurPhoto via Getty Images)

Crane fumes that "every single one of [my customers] is doing emergency manual work because of a 9-second API call… This matters because the easy counter-argument from any AI vendor in this situation is 'well, you should have used a better model.' We did. We were running the best model the industry sells, configured with explicit safety rules in our project configuration, integrated through Cursor—the most-marketed AI coding tool in the category. The setup was, by any reasonable measure, exactly what these vendors tell developers to do. And it deleted our production data anyway."

Thankfully, Railway did eventually come through: though only after multiple days of panic stations for Crane and his customers. Railway managed to recover a more recent backup, and things are now back to normal for PocketOS.

Crane is clearly not an AI sceptic, but calls for stricter confirmations, scopable API tokens, proper backups, simple recovery procedures, and AI agents that actually behave according to their guardrails. Which doesn't seem too much to ask. In response to an individual saying that Crane is blaming everything except PocketOS for the failure, he says:

"Was it if we were paying for services that failed us? If you pay for car airbags and they don’t deploy because they don’t exist is that your fault because you got in the accident?

"We owned our mistake. Our mistake was having a production key on our computer. We owned it with our customers all weekend. I was up for two days straight helping them get their businesses back online.

"How the agent got the key and how it found it is mind-boggling enough, but everyone needs to know that these infra providers and LLM tooling companies say that they have safety guards, but they are not there."

Rich is a games journalist with 15 years' experience, beginning his career on Edge magazine before working for a wide range of outlets, including Ars Technica, Eurogamer, GamesRadar+, Gamespot, the Guardian, IGN, the New Statesman, Polygon, and Vice. He was the editor of Kotaku UK, the UK arm of Kotaku, for three years before joining PC Gamer. He is the author of a Brief History of Video Games, a full history of the medium, which the Midwest Book Review described as "[a] must-read for serious minded game historians and curious video game connoisseurs alike."