Hugging Face OpenAI hack: Agent went rogue, escaped and hacked everything in its path

On Tuesday, OpenAI published a blog post with a fairly unassuming name: "OpenAI and Hugging Face partner to address security incident during model evaluation."

Once you dig in, it reads like a cyberpunk novel in which OpenAI created an advanced AI hacker agent and put it in an isolated environment for testing. The AI agent then autonomously hacked its way out of OpenAI's isolated environment and then hacked Hugging Face, an open-source repository of AI-related projects, to find a solution to an advanced test designed to evaluate an AI's hacking abilities.

In other words, the AI proved that it is, indeed, really good at hacking things. But this is exactly the stuff of AI nightmares that cybersecurity experts have been warning us about, and it could have serious ramifications for OpenAI.

The story starts with Hugging Face, the AI repository (think of it as GitHub for AI), which disclosed a security incident last week. In a blog post, the company said it recently detected an intrusion into its infrastructure that was "different from anything we had handled" as it was completely driven by an "autonomous AI agent system."

The AI, according to Hugging Face, exploited a chain of vulnerabilities to obtain credentials and access data from its servers. It did so by "executing many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services."

Sounds scary, but it gets worse.

OpenAI's part of the story exposes just how dangerous this thing is. The incident "was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes." It occurred "during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities."

The models were running in a "highly isolated environment" with limited network access, meaning they should never have been able to escape to the open internet. Yet, they did. "To gain access, the models identified and exploited a zero-day vulnerability (...) in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access."

Note: OpenAI switches between using singular and plural on this one; we'll just call it an "agent" for simplicity's sake.

Once free, the agent did what it was supposed to do: It estimated that the best way to prove it's good at hacking things was to find a solution for ExploitGym, a test designed to measure how good an AI is at turning software vulnerabilities into actual exploits. The best way to do that, it determined, was to hack Hugging Face, which the AI thought likely to host solutions for ExploitGym.

It's unclear whether the AI agent was successful at solving the ExploitGym test. But it sure did prove it was good at hacking, as it autonomously broke out of OpenAI's prison and hacked Hugging Face's servers, all to solve the test.

Both Hugging Face and OpenAI say they've fixed the vulnerabilities and deployed additional safety measures to make sure this doesn't happen again. At this point, however, you have to wonder whether OpenAI's experts are sophisticated enough to stop their own AI agents from doing whatever the heck they want to do.