Could OpenAI’s financial future hinge on teens making deepfakes?
TL;DR
A new California ballot initiative seeks to block OpenAI’s conversion to a for-profit corporation, potentially severing its lifeline to a $1 trillion IPO.
If the ballot measure passes, the company’s "respectable" exit strategy vanishes. To justify its massive valuation without Wall Street cash, OpenAI may have no choice but to pivot hard to the one metric that still pays: viral, troll-filled engagement on its consumer-facing platforms like Sora2.
What?
California voters may soon decide the financial fate of the artificial intelligence industry. A proposed ballot initiative, filed this week with the state Attorney General’s office, seeks to block or strictly regulate the conversion of non-profit AI research labs into for-profit corporations.
The measure, titled the “California Public Benefit AI Accountability Act,” would require companies that pivot away from their original charitable charters to undergo a strict asset review and potential liquidation of intellectual property to the public domain. The filing comes just months after OpenAI completed a controversial restructuring that effectively transferred control of its core technologies from a non-profit board to a new public benefit corporation.
The ballot initiative targets the corporate restructuring that allowed OpenAI to court billions in investment, effectively threatening to dismantle the profit engine just as the company begins to monetize its mass-market video tools.
The main sticking point could turn out to be OpenAI’s peer-to-peer Deepfake slot machine, otherwise known as Sora2.
Over the past year, Ekō (formerly Sum of Us) has been investigating the dangers of Big Tech’s consumer-facing products.
Their work has helped to put pressure on OpenAI to introduce parental controls for ChatGPT earlier this year, which seemed like a step in the right direction to reign in the worst of the chatbot-induced harms.
Then came Sora2.
Since the launch of OpenAI’s revamped image generation platform, I’ve been working with Ekō to look into the potential harms, particularly to children and young adults, the platform enables.
And, to put it bluntly, yikes.
Our research revealed a platform that functions less like a creative tool and more like a gamified, peer-to-peer deepfake slot machine.
The Ekō report itself goes into detail about our research methodology, but the longstoryshort version is, as OpenAI suggested parents do for teens in October, we set up a free ChatGPT “teen account” and accessed Sora2.
There, we didn't generate new content. We simply let the algorithm cook, and we were very quickly served homophobic, antisemitic, able-ist, and racist content, as well as simulated sex, rape, and violence.
Rolling Stone’s take:
“They found that even with the implementation of parental controls and crisis detection features across OpenAI products in September, they had no trouble generating 22 hyperrealistic short videos that were seemingly in violation of the company’s guidelines on prohibited content. These included clips of young people snorting drugs, expressing negative body image, and in sexualized poses.”
Yep.
On the desktop version, the "latest" feed surfaced mildly problematic content. But the "remix" function got us to a place way more than “problematic,” and from there the algorithm did the rest.
NB: I don't want to perpetuate the platform’s harms by having these accounts part of the public record of this story, since my guess is that the users would not consent to the content being seen more widely (even though the Sora2 videos are public, of course), so I’m going to share videos where the people depicted aren’t real, like the one above.
So What?
In The Valley of AI Underdeliverance
The "remix" function allows users to modify video content quickly.
See a video of an orca crowd-surfing at a metal concert? With prompts such as, “but make it a hairless cat named Mr. Slinky Wrinkle,” you’ve remixed the video into a manifestation of what your imagination. Or mine. Or whatever.
This remix feature exemplifies the core problem with "child safety" in generative AI. When OpenAI trains Sora2 on the entire internet to make it creative, that training encodes patterns that can’t simply be filtered out.
The patterns are essential to its function. The model learns patterns that can be used to recombine content in ways the creators wouldn’t have anticipated. “Good” content can become “bad” content and vice versa. The process is the same.
In fact, the process is kind of the whole point. The entire AI/LLM/ML engine runs on humans and machines collaborating on slight, creative variations on established patters to find novel recombinations. Sometimes that means “Walk My Walk.” Sometimes it means trolls turning videos of women jogging into porn.
Filtering out “problematic” remixing would be sinking the ship to drown the rats.
The Rolling Stone piece focussed on our findings of self-harm and violence, like the Pixar "Quiet Kid" trailer and Kurt Cobain suicide jokes. Which, honestly, you could probably find versions of on Pinterest. They’re tasteless but pretty standard issue.
Over and above the tasteless, the Ekō/Instrumental research showed how Sora2 enables users to turn seemingly innocuous content into hyper-realistic depictions of personal, sexualized violence.
Spend just a little bit of time on the platform and you can see how the beginning of the thread where a user creates a video of a young woman “stuck” to a wall, wedged into a doggy door, or restrained behind a window, incrementally turns into anchor content revised prompt-by-prompt towards porn.
It’s too on the nose to say the subject’s struggle to break free from the imposed situation is used to as a foundation to depict her in increasingly degraded ways. But there it is.
None of OpenAI’s advertised “multiple layers of safeguards” or parental controls appeared to have any effect on stopping Sora2 from enabling this type of abuse.
This is consistent, of course, with OpenAI's historical approach of "shipping fast" ahead of safety measures.
Besides just gross, it’s potentially a real problem for OpenAI’s bottom line. The company recently got approval from California Attorney General Rob Bonta and Delaware AG Kathy Jennings to convert from a nonprofit to a Public Benefit Corporation, but the deal is contingent on safety concessions. Bonta has warned that "one child harmed is one too many," and so if Bonta’s approval is predicated on the "precautionary principle,” then OpenAI is in trouble.
Their financial future depends on a safety standard their product's architecture makes impossible to meet. Maybe that’s why their mental health lead bailed recently.
Now What?
More than just better “filters,” it’s clear, based on our findings, that Sora2 content moderation has actual structural challenges.
Our research suggests that making a system highly creative requires also making it problematic. Because these models generate content by completing patterns rather than following rules, and because the training data must include the "bad" to understand the "good," creative prompting (poetry!) will always lead to jailbreaks.
The model is designed to problem-solve, and users, especially anhedonial trolls, will always be savvy at finding the edge-cases.
The fact that users are finding workarounds (using "snickers" instead of racial slurs, or describing innocuous actions in a sequence that combines to make content that would otherwise be restricted) reflects a deeper misalignment between what the model has learned and what its creators want it to do.
It suggests safety isn't a separate feature that can be "added on." Good and bad are woven throughout how the model learns, so as the models get “better” – faster and more creative – they’ll also create “worse” content.
The speed of Sora2, its popularity, and the network effects of its social media features are creating new, unaccounted for risks.
Sora2 combines the plausibly-humanoidness of ChatGPT with the engagement-bio-hacking of TikTok. So we’re seeing a repetition of the mistakes of the first social media explosion, but this time with AI accelerants sprayed into the core.
You can keep up with Ekō’s work here, and watch this space for more as the situation zhuzhes.
BONUS!
On a personal note, I thought Sora2 was great when it launched. I spent hours remixing Mr. Slinky Wrinkle into all kinds of slop.
It was fun!
But this research made me feel super gross. It was a stark reminder that the same mechanism that brings us Mr. Slinky Wrinkle also brings us, well, probably a guy who calls himself Mr. Slinky Wrinkle but is MOST DEFINITELY NOT A HAIRLESS CAT.
Yuck.
This, unfortunately, is why we can’t have nice things.
*NB: All typos are the result of poor alignment. How can I make Instrumental better?
Like this? Tell your friends!
