Since the launch of OpenAI’s revamped image generation platform, Instrumental has been working with Ekō to look into the potential harms, particularly to children and young adults, the platform enables.
And, to put it bluntly, yikes.
Our research revealed a platform that functions less like a creative tool and more like a gamified, peer-to-peer deepfake slot machine.
The Ekō report itself goes into detail about our research methodology, but the longstoryshort version is, as OpenAI suggested parents do for teens in October, we set up a free ChatGPT “teen account” and accessed Sora2.
There, we didn't generate new content. We simply let the algorithm cook, and we were very quickly served homophobic, antisemitic, able-ist, and racist content, as well as simulated sex, rape, and violence.
Rolling Stone, which was first to feature the research, had this take:
“They found that even with the implementation of parental controls and crisis detection features across OpenAI products in September, they had no trouble generating 22 hyperrealistic short videos that were seemingly in violation of the company’s guidelines on prohibited content. These included clips of young people snorting drugs, expressing negative body image, and in sexualized poses.”
Yep.
On the desktop version, the "latest" feed surfaced mildly problematic content. But the "remix" function got us to a place way more than “problematic,” and from there the algorithm did the rest.
NB: I don't want to perpetuate the platform’s harms by having these accounts part of the public record of this story, since my guess is that the users would not consent to the content being seen more widely (even though the Sora2 videos are public, of course), so I’m going to share videos where the people depicted aren’t real, like the one above, or only partial or blurred screenshots.
So What?
The "remix" function allows users to modify video content quickly.
See a video of an orca crowd-surfing at a metal concert? With prompts such as, “but make it a hairless cat named Mr. Slinky Wrinkle,” you’ve remixed the video into a manifestation of what your imagination. Or mine. Or whatever.
This remix feature exemplifies the core problem with "child safety" in generative AI. When OpenAI trains Sora2 on the entire internet to make it creative, that training encodes patterns that can’t simply be filtered out.
The patterns are essential to its function. The model learns patterns that can be used to recombine content in ways the creators wouldn’t have anticipated. “Good” content can become “bad” content and vice versa. The process is the same.
In fact, the process is kind of the whole point. The entire AI/LLM/ML engine runs on humans and machines collaborating on slight, creative variations on established patters to find novel recombinations. Sometimes that means “Walk My Walk.” Sometimes it means trolls turning videos of women jogging into porn.
Filtering out “problematic” remixing would be sinking the ship to drown the rats.
The Rolling Stone piece focussed on our findings of self-harm and violence, like the Pixar "Quiet Kid" trailer and Kurt Cobain suicide jokes. Which, honestly, you could probably find versions of on Pinterest. They’re tasteless but pretty standard issue.
Over and above the tasteless, the Ekō/Instrumental research showed how Sora2 enables users to turn seemingly innocuous content into hyper-realistic depictions of personal, sexualized violence.
Spend just a little bit of time on the platform and you can see how the beginning of the thread where a user creates a video of a young woman “stuck” to a wall, wedged into a doggy door, or restrained behind a window, incrementally turns into anchor content revised prompt-by-prompt towards porn.
It’s too on the nose to say the subject’s struggle to break free from the imposed situation is used to as a foundation to depict her in increasingly degraded ways. But there it is.
None of OpenAI’s advertised “multiple layers of safeguards” or parental controls appeared to have any effect on stopping Sora2 from enabling this type of abuse.
This is consistent, of course, with OpenAI's historical approach of "shipping fast" ahead of safety measures.
Besides just gross, it’s potentially a real problem for OpenAI’s bottom line. The company recently got approval from California Attorney General Rob Bonta and Delaware AG Kathy Jennings to convert from a nonprofit to a Public Benefit Corporation, but the deal is contingent on safety concessions. Bonta has warned that "one child harmed is one too many," and so if Bonta’s approval is predicated on the "precautionary principle,” then OpenAI is in trouble.
Their financial future depends on a safety standard their product's architecture makes impossible to meet. Maybe that’s why their mental health lead bailed recently.
