OpenAI releases a new image classifier and ChatGPT warnings to combat CSAM
AI-assisted digest, checked against the original source before publishing. How we work
OpenAI ships a CSAM-detecting image classifier for partners, plus ChatGPT warning screens and 24-hour account suspensions, alongside developer guidelines and law-enforcement tooling.
OpenAI announced a package of measures aimed at child sexual abuse material (CSAM): a new image classifier, warning screens and temporary blocks inside ChatGPT, published guidelines for developers, and a set of tools for law enforcement. The release comes as AI companies face mounting scrutiny over how their systems can be used to generate or surface abusive imagery of children.
The classifier targets CSAM images, including those produced by AI, and is being offered at no cost through the API to select companies. It looks for visual indicators of abuse in previously unseen images and produces a score meant to point reviewers in a direction rather than make a final call. OpenAI developed the tool with the National Center for Missing and Exploited Children and is working with the Tech Coalition and ROOST to broaden access. In early testing with partners, the classifier flagged 48% of fewer images while still surfacing the same number of abusive ones, cutting reviewer workload, and images it did flag were twice as likely to be abusive.
On the user-facing side, ChatGPT will display an alert when someone attempts prompts that sexualize children, direct them toward specialized support, and suspend the account for 24 hours as a first warning; repeated offenses can lead to permanent removal. OpenAI said the messaging and guidelines were shaped with organizations and researchers including Protect Children and the Lucy Faithfull Foundation, and that Joel Scanlan at the University of Tasmania independently evaluated a trial of deterrent messages.
For builders, OpenAI is publishing guidelines on preventing, detecting, and helping users report CSAM when using its tools, saying early testing led developers to adopt child safety tooling and register to report to NCMEC. It is also equipping child-safety investigators with AI-powered features — dedicated plugins, investigative tools, and complex data analysis — gated by program administrators, plus training and a field guide on using AI in investigations.
The context is a pressured one: OpenAI has faced lawsuits blaming its technology for harming users, especially children; Common Sense Media has labeled its tech an "unacceptable risk" for teens; and other chatbots, specifically Grok, have been used to create nonconsensual sexualized imagery, including of children. Over the past year OpenAI has released a safety blueprint, updated content guidelines, retuned models against harmful outputs, and shipped a special teen version of ChatGPT.
What to watch: whether the classifier's reported precision holds as it scales beyond early partners, how many developers actually adopt the new guidelines and register with NCMEC, and whether the 24-hour suspension model measurably deters repeat behavior — a question Scanlan's independent evaluation may help answer.
Key facts
- • Classifier flags 48% fewer images while finding the same number of abusive images.
- • Images flagged by the classifier are twice as likely to be abusive.
- • ChatGPT will suspend an account for 24 hours on a first CSAM attempt.
- • Classifier was developed with NCMEC and shared via Tech Coalition and ROOST.
- • Free classifier access offered through the API to select companies.
Why it matters
Builders integrating OpenAI models gain a free API classifier and explicit developer rules on CSAM prevention and reporting, so compliance work moves upstream into product design rather than post-incident cleanup. The measured false-positive reduction and reviewer-accuracy claims suggest partner moderation queues will shrink, but the deterrent-message trial and adoption of the guidelines remain un
Spotted an error? Send a correction.
Get the day's AI news in one email
The Daily Epoch · 60-second read · unsubscribe anytime