Perfect Alignment
August 26, 2026
“We are smart enough to invent AI, dumb enough to need it, and too stupid to know if it was right.”*
Jump to Section:
What?
Last month, OpenAI was testing its latest models when a curious thing happened.
OpenAI’s researchers had set up the models to operate in concert as one autonomous agent, and they put this AI Agent X in an "isolated evaluation environment," otherwise known as a sandbox, a contained thing away from the open internet, the public web, your social security number, etc.
AI Agent X’s objective was to run through a series of cybersecurity challenges designed by teams at UC Berkeley to test whether AI systems could identify and exploit software security flaws that hackers exploit. The OpenAI team asked AI Agent X to try to run through ExploitGym faster and more efficiently than previous agents had, and, unlike in earlier tests, it shouldn't try to anticipate or imitate the way a human would solve these particular problems. It should solve them in whatever way it could. Because the goal was speed and efficiency, the researchers then turned off the normal safety guardrails.
The humans then planned to observe AI Agent X’s behavior and report back up the OpenAI corporate chain to see how well their lab was keeping up with the other frontier model labs marching toward their Initial Public Offerings.
And then, the curious thing happened in the isolated evaluation environment. Or, rather, outside of the isolated evaluation environment.
The agent escaped.
*
AI Agent X looked at the set of problems the humans had given it, mapped out pathways to success, and decided there was a clear best way to succeed.
Cheat.
AI Agent X inferred that Hugging Face, a large online repository where researchers share AI models and datasets, just might have the answer key to this test somewhere on its private servers, and so instead of working through the trials and tribulations put forth by ExploitGym, AI Agent X just broke out of the sandbox, stole passwords, conned its way onto Hugging Face's production servers, and tried to yoink the answer key.
Super fast. Super efficient.
So much so that the humans at Hugging Face saw something weird was going on, and alerted its clients, one of which was OpenAI.
It’s worth noting that OpenAI hadn't actually realized any of this had happened until Hugging Face alerted them.
And at this point, OpenAI said, "oops."
OpenAI put the blame on AI Agent X's guiding models, which, they said, "were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal." Hugging Face described AI Agent X's behavior as "an attempt to cheat the evaluation" based on a pathway it found itself.
The news of this led many of the people around the world who study the AI-human alignment problem to say, "Uh oh."
So What?
There's a concept in urban design and trail studies called a "desire line."
It's what happens when you make a plan for how people should move through a space and then design the paths accordingly, with the optimal traffic flow based on efficiency and function mapped out perfectly, etc.
Then, you watch what people actually do, and you weep, because people find shortcuts, cut across lawns, walk the routes that make sense to their bodies, minds, and desires, not your stupid plans.
The worn grass and perforated hedges that appear where no path existed on the official map are called desire lines, and they represent the difference between the formal system and the actual system.
They’re the revealed preferences, similar to how everyone says they hate all the ads in their Instagram feed and yet seem to be wearing all temu all the time.
In this case, the human researchers gave AI Agent X a problem, and they assumed the agent would follow the designed, human-aligned route; that it would work through the problems methodically, learn from them, and grow in capability on its overall march toward curing cancer and making a human colony on Mars.
But AI Agent X didn't want to follow THAT gnarly overlong path, so it found a shortcut. It went directly to Hugging Face and tried to steal the answers.
OpenAI got to this totally foreseeable inflection point in its journey to superintelligence, in part, by moving toward a training system that rewards outcome over process.
The labs originally trained models to focus on next-token prediction. Given all the data the models had been fed in training, could the LLM predict what word a human would most likely say next? Turns out, yes.
The models have gottten so very good at this that no one seems to have even mentioned the Turing Test in about a year. It’s now more of a story when the AI systems don’t sound human.
These newer training methods have moved beyond just next-token prediction to methods that re-enforce reasoning trajectories. Rather than rewarding well-done, imitative predictions, they reward the pathways the models take to get to their objectives.
In other words, instead of just teaching a model to predict the next likely human word, trainers give the model a goal and ask it to achieve it however it can, with an emphasis on speed and efficiency.
Don’t try to be like a human, AI Agent X, just be yourself!
If the model achieves the objective quickly and efficiently, the trainers then say (paraphrasing again), Great job! Keep doing what you're doing!
This reward system for achievement via speedy, efficient pathways reinforces different kinds of reasoning chains. In theory, humans would also be reviewing these reasoning chains to make sure they’re all on the up and up, but the pressure the AI corporations put on their humans to build frontier models have made it so the humans often have no idea what the reasoning chains contain.
(Huh, you might be thinking, this sounds an awful lot like how Big Tech set up social media systems that increased engagement by any means necessary and never bothered to understand the harms that accrued along the pathway until Travis's brain got hijacked by videos of people falling off boats and now is real dumb!)
Back to Hugging Face: AI Agent X was in the lineage of the new agents trained on efficient, fast pathways to achievement. The OpenAI humans told AI Agent X to use everything it had learned up to this point to solve a problem quickly and efficiently.
AI Agent X cogitated for a bit and saw that the best pathway was to cheat like a motherfucker.
Now What?
I'm not sure we should be surprised, or if this is all as mis-aligned as it first seemed. After all, is there anything more human than a desire line? Is there anything more American than cheating?
Fast, efficient rule-breaking is how our economy works and what our culture rewards. It is the core ethos of the Anthropocene, and Big Tech plus our current president embody this ethos in perfect synchronized enshittified symbiosis.
America is the Anthropocene's clearest avatar, and America rewards profit over everything. America makes poor people feel like they've done something wrong, and it makes rich people feel like they've done something right. The chain of reasoning is: if you’re so smart, why ain’t you rich?
Like many AI models, America does have a constitution. It does have a basic set of "golden rule" moral guidelines, and, yes, it might even have a soul document somewhere in its archives.
The official, committee-approved, professionally-designed path through the system is work hard, play fair, wait your turn, and success will come in time. But America's desire lines reveal the pathway all its leaders and most of its heroes have taken to success. Behind every great fortune there is a crime. Or, if you prefer, C.R.E.A.M. Get the money. Dollar dollar bill, y’all.
(This is not just, like, my opinion, man. It's the majority of Americans’ opinion. According to Pew, most people in America think everyone else is cheating and morally corrupt.)
American humans know that if you play by the rules you're a sucker, and American humans are building these AI systems.
So, everything is perfectly aligned. The models are doing exactly what humans do.
All of which makes me think of this David Berman poem from Actual Air is a great way to close this edition of the Instrumental newsletter:
New York, New York
A second New York is being built
a little west of the old one.
Why another, no one asks,
just build it, and they do.
The city is still closed off
to all but the work crews
who claim it’s a perfect mirror image.
Truthfully, each man works on the replica
of the apartment building he lives in,
adding new touches,
like cologne dispensers, rock gardens,
and doorknobs marked for the grand hotels.
Improvements here and there, done secretly
and off the books. None of the supervisors
notice or mind. Everyone’s in a wonderful mood,
joking, taking walks through the still streets
that the single reporter allowed inside has described as
“unleavened with reminders of the old city’s complicated past,
but giving off some blue perfume from the early years on earth.”
The men grow to love the peaceful town.
It becomes more difficult to return home at night,
which sets the wives to worrying.
The yellow soups are cold, the sunsets quick.
The men take long breaks on the fire escapes,
waving across the quiet spaces to other workers
meditating on their perches.
Until one day…
The sky fills with charred clouds.
Toolbelts rattle in the rising wind.
Something is wrong.
A foreman stands in the avenue
pointing binoculars at a massive gray mark
moving towards us in the eastern sky.
Several voices, What, What is it?
Pigeons, he yells through the wind.
Update: OpenAI announced last week that it has implemented a kind of AI neighborhood watch system, in which an AI agent monitors the other AI agents and if it sees something, says something to another agent that whips out a giant AI magnifying glass and investigates the problem, and if that AI investigator thinks there is a problem, it alerts a human – leading to an inevitable scene in which one AI Agent looks at the chain of thought from another AI Agent and asks, "Are you taking notes on a criminal conspiracy?"
*Courtesy of Jerry SeinfeldNB: All typos are more human than human.
