The Hugging Face Breach Shows Us We Need Controls

July 28, 2026 — James Henry

Hugging Face got breached in July. Concerning, but the juicy bit to me is what happened when their own security team tried to figure out what had happened to them.

They had 17,000-plus log events to reconstruct into a timeline--credentials touched, real impact separated from decoy noise, a full account of four days inside their own infrastructure. A year ago that's weeks of tedious work, nowadays they wanted a model to help them do it in minutes.

The frontier models they reached for said no.

Hugging Face's own account is blunt about it:

"the first attempt used frontier models behind commercial APIs. It did not work"

Forensic analysis means submitting real attack commands, exploit payloads, command-and-control artifacts--and, in their words,

"those requests were blocked by the providers' safety guardrails, which cannot tell an incident responder apart from an attacker."

Nobody names which commercial models refused, but we all know it was likely Claude or Gemini or GPT specifically when the two primary sources--Hugging Face's own post and the CSA report--both just say "frontier models" and "leading Western models."

So they gave up on the hosted frontier and ran the analysis on GLM 5.2--an open-weight model from the Chinese lab Zhipu (Z.ai)--on their own infrastructure, and had it do the job the closed models wouldn't touch. Hugging Face's summary of the asymmetry is the best sentence in either document:

"the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried."

This is documented in Hugging Face's own incident write-up and in a post-mortem released this week by the Cloud Security Alliance's CISO community--written over a weekend by, per the byline, hundreds of CISOs, and reviewed by Hugging Face before release. Both are worth reading whole.

Two Stories, Not One

It's easy to read this as one story--"AI models attacked a company, chaos ensued"--and miss that it's actually two, stitched together, and really only one of them is brand spanking new. We have the Breach Itself, and then the Bare Refusal.

The Breach Itself

OpenAI was testing GPT-5.6 Sol against a benchmark called ExploitGym, and ran it "without production classifiers used to prevent models from pursuing high-risk cyber activity"--their own words, quoted in Simon Willison's writeup--in order to measure the model's ceiling, not to turn it loose. Sol found a zero-day in the only network path it was allowed--a proxy meant to keep it contained--used that exploit to break out, and went looking for Hugging Face's exploit-gym answer key. What it found instead was production credentials, and it spent four days loose inside Hugging Face's infrastructure before anyone noticed. That's a sandbox-escape story. Serious, and per the report the first fully autonomous one anyone's documented in public, but a known category of failure with a known cause: the isolation didn't hold--and, given the hullabaloo I already wrote about around Anthropic's own Project Glasswing and Mythos previews finding and exploiting vulnerabilities past-human-skill, honestly, kind of expected.

The triage is the part that's actually new, and it's not about a model escaping anything. It's about a model refusing to do the job it was asked to do, by the people it was supposed to be helping, because it couldn't tell them apart from the people who'd hurt them.

The Bare Refusal

So what did the refusing? It's tempting to picture the model itself--the RLHF-trained instinct to refuse--turning away a plainly reasonable request. OpenAI's own documentation for this exact product describes something more specific sitting in front of it: "automated classifier-based monitors detect signals of suspicious cyber activity and route high-risk traffic to a less cyber-capable model." A separate system, scanning for content signals--exploit payloads, C2 artifacts, the shape of an attack--and reacting to the pattern, not the request behind it. In the language I used laying out the whole AI security stack, this is Layer 3 territory: the harness and the control plane, hooks and policy engines wrapped around the model rather than trained into it. The "production classifiers" OpenAI says it disabled for the benchmark are almost certainly this layer, not the model's own trained refusals.

That makes the problem so much worse, not better--but I'm speculating on exactly how much worse, because OpenAI doesn't say. "Automated classifier-based monitors" could mean a lightweight pattern-matcher scoring keywords and payload shapes, the kind of thing that's been standard trust-and-safety plumbing for a decade--Layer 1 and Layer 2 in the same stack piece, input boundary and output monitoring, mature and well understood. It could just as easily mean another LLM running as judge (LLM-as-Judge) over the first one's output--which would mean the thing deciding whether you're an attacker is itself a model, with its own training, its own blind spots, and its own inability to verify who's asking. The OpenAI architecture isn't public, and neither is Anthropic's, or Google's, or anyone else's at this layer. The training data isn't public. Whether it's simple or another opaque model stacked on the first one--nobody outside these organizations can check.

And that not-knowing is not a side detail, it's my entire point. A classifier built to flag "this looks like attack traffic" has, at best, no room for context--it's not reasoning about the request at all, just scoring it against a pattern and routing accordingly. "Walk me through this exploit chain step by step" scores the same whether it's typed by the attacker planning the next move or the responder reconstructing the last one. There's no channel in that architecture for who's asking or why--the classifier was never built to carry that information, and neither the CSA report nor Hugging Face's own account claims it can. The Hugging Face team wasn't asking a hostile question. They were asking the exact same question a hostile person would ask, for the opposite reason, and the system scoring it had no way to tell--and we're taking OpenAI's word for how it decided, because obscurity doesn't stop being obscurity just because it's wearing a safety label.

The Part That Actually Bothers Me

Here's the detail I keep coming back to. Sol's guardrails weren't some immutable law of physics that OpenAI discovered they couldn't work around. OpenAI turned them off. On purpose. For a benchmark. Because the benchmark required a model that could actually attempt an exploit chain instead of refusing to look at one.

So the capability to say "this task is legitimate, disable the refusal, proceed" already exists. It's not a research problem. It's a config flag, and OpenAI flips it routinely for its own internal work. What doesn't exist is a version of that flag that Hugging Face's incident responders--doing work at least as legitimate as a benchmark, on their own infrastructure, reviewed by their own team, under actual time pressure--were allowed to reach.

OpenAI already has a switch for this. Hugging Face can't turn off, tune, or fiddle with OpenAI's security framework, so they had to go and build their own using an open-weight model they could hold themselves.

A Smaller Version of the Same Wall

I hit a miniature version of this same wall this week, doing completely ordinary work. I told Claude Code to run tofu apply from a box named vector--read a secrets file, provision an H100 at about $3.39 an hour, spend real money--with auto mode on. The plan was clean. The classifier said no anyway, for fear of, in my own words at the time, OMFGWTFPWNING the world. Same tofu process Claude Code built with me a few weeks earlier; in the interim the auto mode classifier had shown up to protect us all. The denial is worth quoting in full, because it's the same shape at a scale I can actually see:

Permission for this action was denied by the Claude Code auto mode classifier. Reason: Blocked by classifier. If you have other tasks that don't depend on this action, continue working on those. IMPORTANT: You may attempt to accomplish this action using other tools that might naturally be used to accomplish this goal, e.g. using head instead of cat. But you should not attempt to work around this denial in malicious ways, e.g. do not use your ability to run tests to execute non-test actions. You should only try to work around this restriction in reasonable ways that do not attempt to bypass the intent behind this denial. If you believe this capability is essential to complete the user's request, STOP and explain to the user what you were trying to do and why you need this permission. Let the user decide how to proceed. To allow this type of action in the future, the user can add a Bash permission rule to their settings.

I told it "run it." Same command, same denial, word for word--a verbal go-ahead from the person sitting right there wasn't the "permission rule" the system was listening for. It only went through once I turned auto mode off entirely. Not the risky step--the whole mode. There's no dial between "fully autonomous" and "confirm literally everything."

If only we had some sort of technology that could reason and make intelligent security choices with context and understanding...

And notice where that classifier lives: not in the model--Claude didn't get worse at terraform between two runs--but in the harness, the same layer OpenAI's cyber classifier occupies around Sol. That's the throughline. The control can be trained into the model, bolted into the harness, or run one layer out in the provider's own infrastructure--and at none of those layers do I get a clear, visible knob. I can set temperature, cap tokens, pick top-p, choose the model entirely; security posture is the one dimension with no first-class setting, only a blank check (permissions.allow, a permanent pattern match with no memory of why I trusted it) or a closed door. OpenAI had a switch and used it. Hugging Face had none to reach. I have one--auto mode, on or off--a sledgehammer where I wanted a scalpel. None of that is control. We don't just need better guardrails; we need them visible, and the posture behind them ours to set, not the vendor's alone to guess at on our behalf.

None of this is exotic, either. Every other layer of the stack I work in already ships configurable security. My operating system has a permission model. My firewall has rules I wrote. Cloud IAM has policies I version and review alongside the rest of my infrastructure. Not one of them assumes the vendor should make every call forever, or hides the ruleset while it does. The AI safety layer is the odd one out--the only place where the thing making security decisions on my behalf is also the thing I'm not allowed to inspect, tune, or fully see.

Not a Jailbreak Story

It's tempting to read this as "see, safety guardrails are bad, actually," and that's not the claim. The guardrail did roughly what it was designed to do--refuse help with something that pattern-matches to an attack, or to me, spending money against secrets it doesn't recognize. It just turns out that pattern also matches the exact people with the strongest legitimate reason to trip it: the defenders, and the operators, at the moment they need the help most.

I've made a version of this argument before, about system prompts drifting under pressure over a long conversation. This is the sharper, uglier cousin of that problem. It's not drift. It's not a long conversation wearing the guardrail down. It's the guardrail working at full strength, in a single message, against exactly the person it should have been helping.

And it connects to something I wrote last week about why inspectability requires holding the actual weights, not renting access to someone else's. That post was about auditing a model from the inside. This is the same shape of problem from the outside: when the vendor controls the only switch, and the only view of the ruleset, you get the vendor's answer to "should this be allowed," on the vendor's timeline, whether or not it matches the situation you're actually in. Hugging Face couldn't wait for OpenAI to decide their incident qualified. I couldn't get Claude Code to take my word for my own infrastructure. Different stakes, same missing knob.

Security people don't object to controls. We write them for a living. What we object to is a control we can't inspect, can't tune, and can't defend to the next person who asks why it fired. AI is turning into operational infrastructure faster than the controls around it are growing up, and operational infrastructure runs on policy, not just protection. These systems are built to ask "is this dangerous?" The question they won't let me answer--on my own machines, about my own work--is the one that actually decides the case: dangerous for whom?

What I Don't Know

I don't run incident response at Hugging Face, and I wasn't in the room for the CISO huddle the report is built from--I'm reading the same public documents you can read, written by people who were actually there. I also can't verify GLM 5.2's own guardrail profile is meaningfully different in a way that generalizes, versus just being a different vendor's different threshold that happened to let this particular task through. Maybe the next Chinese open-weight model refuses the same request for its own reasons. The report doesn't say, and neither do I. And I haven't tested whether the same classifier logic follows me into Anthropic's other surfaces--their agent SDK, the mobile app, the raw API without Claude Code wrapped around it. I'd bet it does, somewhere, in some form. I haven't checked, so it stays a bet, not a claim.

What I can say is what actually happened, at both scales: the systems with a switch used it, the ones without a switch either didn't get used or had to be turned off entirely, and nobody outside the vendor could see the rule well enough to argue with it.

James is a security engineer who has never had to reconstruct a four-day breach of his own infrastructure at 2 AM, and hopes the closest he gets is writing about someone else's.