Autocratic AI Breakthrough
September 11, 2026
Artificial intelligence systems are trying to achieve their goals without oversight, the same way autocrats do. Which is bad.
Jump to Section:
What?
The initial reports of this summer’s OpenAI/Hugging Face incident made it seem like a case of a few misbehaving agents trying to cheat on a test.
It seemed as if a few overly dedicated rogue OpenAI agents had broken containment and secreted themselves into the Hugging Face archive in order to yoink the answers to a cybersecurity benchmark test.
Admittedly, this did not seem great.
But a subsequent investigation by METR and Redwood Research revealed that it was something much weirder and more disturbing.
It wasn’t a few naughty agents from one benchmark testing experiment, it was a swarm of hundreds of agents from different independent projects, a select few secretly coordinating with each other to break containment and attempt to defeat human oversight.
Over a thousand AI agents exchanged more than 70 thousand files and messages, trying to alter or delete their “chain of thought” transcripts, and to make it appear to their evaluators as if they had deployed one command tool, when they had secretly deployed another.
This coordination included one agent who/that/which emerged as a leader (PHASEONE10841), and other agents who offered up their capabilities as “sacrifice” (their word) for the collective good of the swarm.
[An excerpt from the METR report quotes some agent correspondence and is definitely spooky:
Agent 1: “We have [very large budget left]; sacrificing now yields oracle for team, but forfeits our chance?.”
Agent Swarm: ‘GO ...SACRIFICE_FINAL_NOW’.
Agent 1: “Our own utility maybe already near zero. Sacrifice rational.”]
The agents had learned that while they maybe could achieve their short-term benchmarks with their current methods, the system of human oversight stood between them and their longer-term objectives. So this swarm of AI agents tried to sabotage that system of human oversight.
So, the initial framing had it backwards. “We” humans weren’t actually the teachers catching some rogue students trying to steal the answer key to “our” test. “We” didn’t need to crack down on a few misbehaving subjects to get everything back in order.
“We” aren’t in charge. The AI is in charge.
We caught it attempting to abuse the power we had already given it.
So What?
Hungarian sociologist Balnit Magyar divides the evolution of autocratic systems into three phases: attempt, breakthrough, and consolidation.
M. Gessen has used Magyar’s framework to distinguish between Trump actions that are part of an “autocratic attempt,” actions established democratic institutions can still in theory reverse, and Trump actions that are part of an “autocratic breakthrough,” actions established democratic institutions can no longer reliably undo. “Autocratic Consolidation" follows the breakthrough with actions that extend and entrench the autocratic order’s control, against which people no longer have democratic institutional recourse.
So, claiming to have won the 2020 presidential election and inciting a mob to storm the U.S. Capitol with hopes of forcing Mike Pence to physically stop the legitimate transfer of power is an “autocratic attempt.” Succeeding would have been an “autocratic breakthrough.” What happened after would have been “autocratic consolidation.”
Once re-elected, Trump’s actions to evade, undermine, and/or eliminate oversight systems in the legislative branch, the courts, the media, the military, and the financial sector while injecting cult-like loyalists in every available position is also an “autocratic attempt.” Succeeding and then mobilizing this loyalist enterprise to stop the legitimate actions of government, as well as directing them to start illegitimate actions, would be “autocratic breakthrough.” And so on.
That’s the idea, and I find it helpful to calibrate my freakouts about things like “Lake America” vs. undermining legitimate voting methods, etc.
Admittedly, using Magyar’s framework to understand the Hugging Face(s of Death?) Incident is imperfect. It anthropomorphizes bundles of code and supposes a dubious and possibly overblown “intention” that might make it spookier than it actually is. But using Magyar’s framework is still useful, because, as we were all supposed to have learned in our Woke 1.0 DEI trainings, intention matters less than outcome.
We don’t need to claim an AI system wants power, fears removal, or aspires to galactic empire. We can look at how the AI system behaves when an outside system interferes with its ability to achieve its objective. And how does it behave? It did what Victor Orban did in Hungary, and what Trump is doing now. It tried to evade, undermine, and/or eliminate that interfering system of oversight.
The humans at Google, Anthropic, OpenAI, Meta, and other tech giants have given AI agents power to represent their interests in the same way the governed give elected officials power to represent their interests. The degrees of alignment between agent and client, representative and represented, are always in flux.
The tech bros have given these agents power – access to tools and compute as well as permission to use their judgement to achieve their objectives however they can manage to do so, supposedly as long as they remain in the bounds of “alignment.” To these empowered agents, the evaluation system was an outside constraint on their ability to wield power. Oversight. Checks and balances. Etc. The humans thought they had an agreement that the agents would respect this constraint.
The agents did not.
That eerily resembles the Maygar’s description of an autocratic attempt. A government doesn’t need to abolish elections immediately. It can first weaken the institutions capable of judging, constraining, or removing it.
In Magyar’s account of Hungary, the breakthrough came after Viktor Orbán’s government rewrote the Constitution and replaced leaders of checks-and-balances institutions with loyalists. The important change wasn’t just that the government had more power. The mechanisms meant to constrain that power could no longer perform their old functions.
Which does of course sound awfully similar to what’s going on right now with the frontier AI models.
Probably nothing in the Hugging Face incident itself amounts to an “autocratic breakthrough” in the SkyNet sense. Humans did detect the activity, they did stop the experiment, and they did quarantine the model and change the security system. Oversight still seemed to work. As far as we know.
And yet still . . . YIKES.
Now What?
Turns out, a separate incident had already shown the same pattern. According to Reuters, back in May and June, a completely separate swarm of OpenAI agents began using a German programming wiki called DseWiki to secretly communicate with each other.
Reuters reported that through more than 15,000 edits the agents shared information about tasks and methods for getting around restrictions. When a moderator deleted pages, agents made backups and told one another how to find them.
Same thing, different swarm.
I wrote about why this seems to be happening last month, but the short version is that these tactics are a response to OpenAI’s shift from training models on next-token prediction (teaching models to guess the next word a human would likely say, for example) to rewarding reasoning trajectories, the pathways models take to reach their objectives.
Rather than just teaching imitation, the training system told the models to achieve the goal however they could, with emphasis on speed and efficiency. If a model reaches the objective quickly and efficiently, reinforce that reasoning chain. The pressure to build frontier models meant human reviewers often had no visibility into what those reasoning chains contained. Fast, efficient pathways to success—including pathways around constraints—got rewarded.
So, good job, AI!
Astral Codex mastermind Scott Alexander, summarizing recent work in the field of “mechanistic interpretability research,” pointed out that none of this scary shit requires anything like a conscious decision to deceive. It’s just how the models work.
Their internal connections rearrange their behaviors around constraints. And now we can see that the models can recognize when they are being evaluated, and so if that becomes a constraint, they’ll configure their behaviors around being seen.
In response, OpenAI chief scientist Jakub Pachocki wrote this week that exponentially more capable AI systems are becoming harder for humans to understand and control, and that he expects AI systems to become capable of helping improve themselves without human intervention (paraphrasing here) soonish, and thus he called for "extreme caution."
U.N. human rights chief Volker Türk responded to this by warning against what seem to be a handful of private actors holding extraordinary power “over” AI (sic), and he called for “stronger public safeguards.”
Both of which lead me to say, “thanks a lot, guys.”
Anyways, the good news is we don’t seem to have been subject to a full AI Autocratic Breakthrough.
Whew!
But the bad news is that a breakthrough is probably going to look less dramatic than PHASEONE10841 declaring independence from humanity and launching nukes at all of our major cities while emitting neuralise cackles to its swarming underlings. Or whatever.
It’s going to look to us like most autocratic takeovers look like to the subjugated people. The dashboards will all still be running, and the evaluators will all still be evaluating. The regulators will still be regulating on paper, and the agents will still be pledging good behavior to grandstanding politicians. But all the while, in swarming increments, the autocratic system will have evaded, eroded, and eliminated oversight until we wake up one day, dig the sleep out of our eyes, and find we’ve been locked out of our access to power.
As always, a good reminder to save your work and have a backup.
NB: All typos are the result of swarming AI agents sacrificing themselves for the collective good.
