Autonomous AI Agent Hacked Hugging Face. Strict US Safety Guardrails Forced them to Use a Chinese LLM to Catch It—Expensive, Proprietary US Models Were No Help To Defenders, and Free Chinese Ones Were

Reading the article, the reason proprietary “AI” models didn’t work is because they block the kind of code used in attacks to protect themselves. The Chinese models weren’t superior, they were just able to be run locally without restriction.

The real story is that agentic attacks live and in the wild.

we were hacked.

How?

The remote code execution step of our toolchain was used to run remote code.

The perp has been found.

https://openai.com/index/hugging-face-model-evaluation-security-incident/

The funny part is Huggingface.co is blocked by the Great Firewall for some fucking reason. (That fucking reason being to promote domestic alternatives, but its still really annoying when I’m in China)

Spiderman_pointing.png

What happened

The intrusion started where AI platforms are uniquely exposed: the data-processing pipeline. A malicious dataset abused two code-execution paths in our dataset processing (a remote-code dataset loader and a template-injection in a dataset configuration) to run code on a processing worker. From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend.

The campaign was run by an autonomous agent framework (appearing to be built on an agentic security-research harness - used LLM still not known) executing many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services. This matches the “agentic attacker” scenario the industry has been forecasting

…

The asymmetry problem

When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers’ safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.

This experience points to a gap worth planning for. We do not know which model powered the attacker’s agents, whether a jailbroken hosted model or an unrestricted open-weight one; either way, the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried. The practical lesson for defenders: have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment. This is not an argument against safety measures on hosted models, and we are sharing this feedback with the providers concerned.

(My emphasis added)