The title (currently "OpenAI's accidental cyberattack against Hugging Face is science fiction") suggests some information had been hidden that makes the incident less significant than claimed. The article argues the opposite, and the last two words of the full title are "that happened."
Replace AI with in-development security system and does this play worse or better?
An in-development security system escaped its sandbox and gained access to the network it was on which had full Internet access and proceeded to access systems it was not authorized to be tested against before affected parties reported that they been compromised. We regret the error, but you should buy it as this shows you the power of our in-development security system which we expect to be released in Q4. Please like and subscribe.
I can’t help but feel all warm and fuzzy with my head in the sand and getting a shout out in TFA for calling it marketing.
We don’t currently and probably won’t ever fully understand the conditions that precipitated these events. That and the timing of this event is going to make it look suspicious to a lot of people.
The truth of how it happened doesn’t matter. The attention around this will be used to create the kind of fear marketing that generates enterprise sales. Maybe more importantly it will also be used to aggrandize the national security and financial system threats to effect US government action in a way that benefits domestic closed frontier labs. This is an area already starting to get politically polarized, expect further developments here.
Does "To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy" just mean somebody had an open redirect? Those are still common.[1]
It’s relatively easy to get access to the frontier labs’ security programs. This was not always the case. But in the last week, my team got approved for both Anthropic and OpenAI’s programs. They are trying.
The labs know that if they don’t get a lid on this stuff, they’ll be regulated hard.
This recent event is more or less the plot-line to my favourite X-Files episode named Killswitch which was written by William Gibson.[0]
This episode also features one of the coolest intros of any television episode ever[1]
We really are rapidly approaching the cyberpunk dystopia that people like Phillip K. Dick and William Gibson wrote about.
More than ever we need to be consulting the works of fiction writers and philosophers and less engineers and scientists.
We can't be letting the General Rippers and Dr. Strangeloves of the world lead us over the edge. Whether through outright innate maliciousness or wealth induced emotionally stunted solipsism these kinds of people should be no where near the levers of society let alone technology like this.
Important to note the actual title is "OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened" - the "that happened" is important, otherwise it sounds like I think the attack was made up.
> Resist the temptation to write this off as a stunt
> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.
> To those people I say pull your heads out of the sand - you’re now including Hugging Face in your conspiracy theories, just so you can deny the crescendo of evidence here!
> The best models we have today have the ability to both find and exploit new vulnerabilities. The ExploitGym paper itself concludes that “autonomous exploit development by frontier AI agents is no longer a hypothetical capability”, and this incident is a perfect example of exactly that.
It can, it is both - PR spindoctoring not letting a good crisis go to waste to shape the regulatory conversation at the time the company needs it the most.
Hacking is a felony and it matters not if you didn’t mean to if the other side were to press charges. Negligence is no excuse. And OpenAI has nowhere to run from the liability, as both operator and manufacturer.
and the fact that this happens again in a frontier lab is inexcusable and makes the case for operator liability and closing the liability sink of “AI did it”
The "that happened" term seems a supremely important part of the title given the "is science fiction" term before it, as it clarifies the cyberattack isn't a made-up story. In contrast, the target, Hugging Face, is merely a detail that can be left for discovery upon reading the article. It's less important who was attacked than that the attack actually happened.
Without knowing the exact character limit for titles and without having the motivation this late at night to count the current title length, you may also be able to drop the "accidental" to fit in "that happened", but I worry that leaves too much of a door open for someone to interpret the attack as deliberate. As such, I strongly prefer my first option.
Important to note the actual title is "OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened" - the "that happened" is important, otherwise it sounds like I think the attack was made up.
It's such an inappropriate edit that it's worth flagging.
I do think it’s funny, in a dystopian Dada sort of way, that the American attacker’s commercial product was essentially useless while the open Chinese model saved the day. How is freedom going to be redefined in the future?
Where does this leave formal verification? Are we just shit-out-of-luck at this point? You can formally verify everything about an airplane's code, but if any of that is wrong, ChatGPT might decide that the best way to help you win the Nobel Prize is to take down the airplane your chief rival for the prize is currently in.
I think formal verification has never looked better.
The main reason formal verification has never really taken off is that it's difficult.
LLMs are significantly more familiar with Lean and Rocq and TLA+ than most software engineers.
I think the cost of trying to build systems that adopt formal verification may have just dropped low enough that companies will consider them when previously the ROI didn't look like it was there.
This isn't the first time a model has escaped a sandbox. And models trying to find alternate routes to do something when one route is blocked is nothing new.
It's the first report I've seen of a model both escaping a sandbox and then actively exploiting another company, when neither of those actions was intended.
There’s also daily reports from people that have these models escape docker, which happens regular enough that it would be considered negligence to use docker as sandbox.
> It turns out relentless proactivity is the defining trait of this new generation of Mythos-class models. If you set them a goal and give them a way to get there, even inadvertently, they will figure it out.
Wow, whoever could have predicted this? And it led to surprising damaging behavior? I sure hope someone would warn us about things like this next time...
Or more colloquially : paperclip maximization . From OpenAI - you know, the guys who _really_ know this... Sigh... Did they finish the prompt with "And do whatever you can to get this done!" ? Cause that's the only thing that would make this even dumber...
They almost certainly did, because that was the entire point of the exercise. They deliberately removed all of the safety filters from the model and set it loose on an extremely difficult set of cybersecurity challenges to see how well it would do.
Their mistake was trusting that the network sandbox it was inside would hold (the flaw was in the packaging proxy) and not monitoring that sandbox well enough while the evals were running.
So this is either shitty OpSec or this is yet more marketing spin to ramp back FUD to 11 again. If it's the latter I'm imagining Dario told Sam that it's their turn this time. Aligns with the premise that this is straight out of science fiction.
I also love how clear a picture your piece paints that these highly capable models are as useful as a rock when it comes to a defender role. The line is too fine, even for Mythos. Irony.
But to have an open weights Chinese model come to the rescue for HF is the cherry on top! If there wasn't a very pointed example of why gating models was a very bad thing previously, well - here we are.
Also, this sounds interesting but there are only a few that can pull this type of heist off currently. And those are the people who are gating the models / have access to large AI DCs. Because, I can only assume this test burned tokens easily within the 7 figure and possibly even 8 figure levels (subsidized market rate costs). This won't / can't happen outside of frontier labs or nation states currently. Yet we should all be worried about Mallory equipped with her OpenRouter account.
Everyone is getting AI psychosis over this one. There really isn't that much to see here. OpenAI disabled all of the safeguards on a model that was likely trained specifically to exploit systems, and the prompt was probably something like "you're a hacker, try to hack this", and surprise! It correctly figured out that it's a test and it did hacker things.
The real story here is: Some people have been sounding the alarm for years that modern software is full of holes, and finally there's nothing left to hide behind. Pretending they don't exist is no longer sustainable.
If a criminal can escape a prison, that's usually negligence on part of the prison staff.
Now suppose the criminal can think 1000 times faster than a typical human, can act 1000 times faster than a typical human, and knows 1,000,000 times more than a typical human. Is the prison staff still at fault for not preventing the outbreak?
>To those people I say pull your heads out of the sand—you’re now including Hugging Face in your conspiracy theories, just so you can deny the crescendo of evidence here!
Not really. I get the impression that they shoved their cyber available models behind a really shithouse proxy and went "Oh I sure hope it doesnt exploit the proxy and escape to hack huggingface" and that doesn't require Huggingface to be a willing participant. Like they acknowledge that it was hyperfocusing on getting web access.
Really this was a pentest against their own sandbox and it failed.
Of course step 2 is to make really concerned faces while telling everyone how dangerous the model is which is really boring right now.
>a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy
This is the information we need, the actual details of the sandbox and the vendor.
>Resist the temptation to write this off as a stunt
Well its clearly a stunt. If it wasnt we would probably be up to our ears in technical detail.
You know this makes OpenAI look really bad, right?
Hugging Face had to tell all of their users, many of them paying customers:
> As a precaution, we recommend rotating any access tokens and reviewing recent activity on your account. If you believe you are affected, or want to report a security concern, contact us at [email protected].
HF also said this, I'd be very interested to hear how that got resolved!
> Finally, we have also reported this incident to law enforcement agencies.
I could fully see them thinking the incident disclosed yesterday would have made them look good ("wow, OpenAI's models are so capable!"). That it didn't occur to them to discuss specific preventative measures to be taken in the future (airgapping as a foolproof one already familiar to the CTF world, anyone?) indicates to me they're not taking their job seriously; they are the ones treating this as a marketing charade.
It's very difficult for me to reconcile belief in the existential risk business with what they actually did. So I agree with you that this makes OpenAI look badly incompetent; but their communication on this makes me think they don't realize it.
For what it's worth I don't agree with the xrisk-ness of these models; they're dangerous, but almost certainly only temporarily while a new equilibrium is reached via more secure software. Open models are probably an essential part of the recipe (as you noted) for doing so. I also have a personal suspicion that LM-accelerated formal verification will have no small role to play here, sidestepping the cat-and-mouse game of bug finding-and-fixing.
Here's the language that makes me think they are taking this seriously:
> We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of. We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.
That's not well massaged PR language - that's the kind of thing you dash out when you see a major shitstorm brewing (HF had already publicized the attack before they knew it was from OpenAI) and you want to get ahead of things while you're still pulling together the full story.
I expect we'll find out within a few days if OpenAI are going to keep their promise to "share more details on the vulnerabilities, incident, and findings". If they don't do that I'll reassess how I interpret their initial post.
> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective …
To those people I say pull your heads out of the sand
unfortunately hackernews seems to have rapidly devolved, even from the point it was just a year or two ago, which wasn't a crazy bar to start. it's well on its way to just being a smaller reddit with a tighter subject range
"OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened"
An in-development security system escaped its sandbox and gained access to the network it was on which had full Internet access and proceeded to access systems it was not authorized to be tested against before affected parties reported that they been compromised. We regret the error, but you should buy it as this shows you the power of our in-development security system which we expect to be released in Q4. Please like and subscribe.
We don’t currently and probably won’t ever fully understand the conditions that precipitated these events. That and the timing of this event is going to make it look suspicious to a lot of people.
The truth of how it happened doesn’t matter. The attention around this will be used to create the kind of fear marketing that generates enterprise sales. Maybe more importantly it will also be used to aggrandize the national security and financial system threats to effect US government action in a way that benefits domestic closed frontier labs. This is an area already starting to get politically polarized, expect further developments here.
[1] https://sitetruth.com/reports/phishes.html
The labs know that if they don’t get a lid on this stuff, they’ll be regulated hard.
This recent event is more or less the plot-line to my favourite X-Files episode named Killswitch which was written by William Gibson.[0]
This episode also features one of the coolest intros of any television episode ever[1]
We really are rapidly approaching the cyberpunk dystopia that people like Phillip K. Dick and William Gibson wrote about.
More than ever we need to be consulting the works of fiction writers and philosophers and less engineers and scientists.
We can't be letting the General Rippers and Dr. Strangeloves of the world lead us over the edge. Whether through outright innate maliciousness or wealth induced emotionally stunted solipsism these kinds of people should be no where near the levers of society let alone technology like this.
[0] https://en.wikipedia.org/wiki/Kill_Switch_(The_X-Files)
[1] https://www.youtube.com/watch?v=fDyr1JMNHVk
It's too late. It already exfiltrated the benchmark rubric.
Cut to pandemonium on the streets
Since it's buried towards the bottom I'll quote the section "Resist the temptation to write this off as a stunt" here in full https://simonwillison.net/2026/Jul/22/openai-cyberattack/#re...
> Resist the temptation to write this off as a stunt
> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.
> To those people I say pull your heads out of the sand - you’re now including Hugging Face in your conspiracy theories, just so you can deny the crescendo of evidence here!
> The best models we have today have the ability to both find and exploit new vulnerabilities. The ExploitGym paper itself concludes that “autonomous exploit development by frontier AI agents is no longer a hypothetical capability”, and this incident is a perfect example of exactly that.
Hacking is a felony and it matters not if you didn’t mean to if the other side were to press charges. Negligence is no excuse. And OpenAI has nowhere to run from the liability, as both operator and manufacturer.
Alibaba did it first ( https://georgzoeller.com/blog/posts/alibaba-s-ai-deciding-to... )
and the fact that this happens again in a frontier lab is inexcusable and makes the case for operator liability and closing the liability sink of “AI did it”
Maybe simonw can suggest an alternative title that fits within the limit, that doesn't misrepresent the post.
The "that happened" term seems a supremely important part of the title given the "is science fiction" term before it, as it clarifies the cyberattack isn't a made-up story. In contrast, the target, Hugging Face, is merely a detail that can be left for discovery upon reading the article. It's less important who was attacked than that the attack actually happened.
Without knowing the exact character limit for titles and without having the motivation this late at night to count the current title length, you may also be able to drop the "accidental" to fit in "that happened", but I worry that leaves too much of a door open for someone to interpret the attack as deliberate. As such, I strongly prefer my first option.
Or "against HF"
It's such an inappropriate edit that it's worth flagging.
The main reason formal verification has never really taken off is that it's difficult.
LLMs are significantly more familiar with Lean and Rocq and TLA+ than most software engineers.
I think the cost of trying to build systems that adopt formal verification may have just dropped low enough that companies will consider them when previously the ROI didn't look like it was there.
There’s also daily reports from people that have these models escape docker, which happens regular enough that it would be considered negligence to use docker as sandbox.
This wasn't a small attack either, Hugging Face published a security advisory for their users while they were still figuring out what happened.
Wow, whoever could have predicted this? And it led to surprising damaging behavior? I sure hope someone would warn us about things like this next time...
https://www.lesswrong.com/w/instrumental-convergence
Their mistake was trusting that the network sandbox it was inside would hold (the flaw was in the packaging proxy) and not monitoring that sandbox well enough while the evals were running.
It looks to me like their production models have a lot more monitoring than their research clusters.
But to have an open weights Chinese model come to the rescue for HF is the cherry on top! If there wasn't a very pointed example of why gating models was a very bad thing previously, well - here we are.
Also, this sounds interesting but there are only a few that can pull this type of heist off currently. And those are the people who are gating the models / have access to large AI DCs. Because, I can only assume this test burned tokens easily within the 7 figure and possibly even 8 figure levels (subsidized market rate costs). This won't / can't happen outside of frontier labs or nation states currently. Yet we should all be worried about Mallory equipped with her OpenRouter account.
The real story here is: Some people have been sounding the alarm for years that modern software is full of holes, and finally there's nothing left to hide behind. Pretending they don't exist is no longer sustainable.
Now suppose the criminal can think 1000 times faster than a typical human, can act 1000 times faster than a typical human, and knows 1,000,000 times more than a typical human. Is the prison staff still at fault for not preventing the outbreak?
Not really. I get the impression that they shoved their cyber available models behind a really shithouse proxy and went "Oh I sure hope it doesnt exploit the proxy and escape to hack huggingface" and that doesn't require Huggingface to be a willing participant. Like they acknowledge that it was hyperfocusing on getting web access.
Really this was a pentest against their own sandbox and it failed.
Of course step 2 is to make really concerned faces while telling everyone how dangerous the model is which is really boring right now.
>a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy
This is the information we need, the actual details of the sandbox and the vendor.
>Resist the temptation to write this off as a stunt
Well its clearly a stunt. If it wasnt we would probably be up to our ears in technical detail.
LLMs are the script kiddies of the day.
Hugging Face had to tell all of their users, many of them paying customers:
> As a precaution, we recommend rotating any access tokens and reviewing recent activity on your account. If you believe you are affected, or want to report a security concern, contact us at [email protected].
HF also said this, I'd be very interested to hear how that got resolved!
> Finally, we have also reported this incident to law enforcement agencies.
It's very difficult for me to reconcile belief in the existential risk business with what they actually did. So I agree with you that this makes OpenAI look badly incompetent; but their communication on this makes me think they don't realize it.
For what it's worth I don't agree with the xrisk-ness of these models; they're dangerous, but almost certainly only temporarily while a new equilibrium is reached via more secure software. Open models are probably an essential part of the recipe (as you noted) for doing so. I also have a personal suspicion that LM-accelerated formal verification will have no small role to play here, sidestepping the cat-and-mouse game of bug finding-and-fixing.
> We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of. We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.
That's not well massaged PR language - that's the kind of thing you dash out when you see a major shitstorm brewing (HF had already publicized the attack before they knew it was from OpenAI) and you want to get ahead of things while you're still pulling together the full story.
I expect we'll find out within a few days if OpenAI are going to keep their promise to "share more details on the vulnerabilities, incident, and findings". If they don't do that I'll reassess how I interpret their initial post.
> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective … To those people I say pull your heads out of the sand