Child safety and generative AI: bridging gaps across sectors

120 representatives from tech companies, child protection organizations, government, academia, and law enforcement came together in Washington, D.C. to examine emerging AI-related child safety risks, identify gaps in existing safeguards, and explore practical opportunities for collaboration.

dsc 0514

No single organization can see the whole picture of how AI is being used to harm children, or where safeguards might make the greatest difference. Tech companies, child protection organizations, researchers, law enforcement, and public authorities each bring different perspectives, information, and capabilities.

Bringing those perspectives together was the purpose of From emerging risk to coordinated response: child safety and generative AI, a multi-stakeholder forum convened by the Tech Coalition in Washington, D.C., on October 21, 2026.

The forum brought together 120 participants representing approximately 70 companies, organizations, and agencies from across North America, Europe, and the Asia Pacific. Attendees included representatives from tech and financial services companies, policymakers, regulators, law enforcement, child protection and survivor advocacy organizations, and experts in AI safety testing and tech policy.

Discussions examined how to test AI systems safely for child safety risks, identify intervention opportunities as AI systems become more autonomous, and strengthen responses when people disclose abuse or other concerning behavior through AI services.

Bringing different perspectives together

Generative AI is changing both the scale and nature of the challenges involved in  preventing, detecting, and responding to online child sexual exploitation and abuse (OCSEA). Developers are introducing new capabilities and integrating AI into a growing range of products and services. Meanwhile, offenders are adapting how they use these technologies to create and distribute abusive material and facilitate other forms of exploitation. 

These risks span platforms, sectors, and borders, creating new questions about how harms are identified, where responsibilities sit, and what effective safeguards and interventions look like. 

Addressing these challenges requires expertise from across the child safety community. Companies developing and deploying AI systems need to understand how their products might be misused and how risks may evolve as capabilities change. At the same time, hotlines, investigators, researchers, and organizations supporting children and survivors bring distinct expertise and responsibilities in preventing and responding to abuse, making coordination across sectors essential.

This forum brought participants together for a cross-sector discussion of the threat landscape, industry presentations on prevention, detection, and response, and three collaborative workstreams on:

  • AI red teaming and adversarial testing for child safety
  • The implications of increasingly autonomous, or agentic, AI systems
  • User safety and intervention pathways for AI chatbots

Across these discussions, participants explored how different organizations encounter risk, what safeguards are already being developed or used, and where collaboration could strengthen responses.

Improving the information available to child safety responders

American-Institute-of-Architects-DC-Event-Photojournalism-photography

When companies report suspected child sexual exploitation and abuse, the information accompanying those reports can help hotlines and law enforcement assess the material and determine the appropriate response.

AI-generated or manipulated content can complicate how responders interpret and prioritize reports. Participants discussed what responders need to know about material’s origin, including whether AI was used to generate or alter it, and what context reporting companies can provide. Across the discussions, participants emphasized the need for clear, actionable reports as AI systems become more complex.  

One area for practical follow-up is concerning the information that companies include in reports to the National Center for Missing & Exploited Children (NCMEC) CyberTipline.

As a next step, the Tech Coalition will convene members to examine how file annotations could be incorporated into tools used for CyberTipline reporting. Such annotations could help convey relevant information about AI-generated or manipulated material, subject to technical feasibility and applicable reporting requirements.

We will also update a resource in our Pathways program for non-member companies with guidance on file annotations and associated tooling considerations.

Understanding responsibilities across the AI ecosystem

AI-related child safety risks do not arise at a single point in the technology ecosystem. Different organizations develop models, build applications, deploy services, provide infrastructure, and integrate AI capabilities into products used by the public.

Each may have different information about how a system is being used and different opportunities to prevent or respond to harm. These questions become more complex as AI systems gain the ability to plan and carry out tasks with greater autonomy. Agentic AI may introduce new ways for systems to encounter, enable, or facilitate harmful activity, but the risks and appropriate safeguards depend on what the system can do and how it is deployed.

Participants explored where developers and service providers could introduce safeguards across an agent’s lifecycle. Examples included restricting access to information and systems, requiring confirmation before potentially risky actions, and detecting harmful activity before outputs are delivered. Across these discussions, participants raised a recurring challenge: individual organizations may see only part of an agent’s activity.

A practical next step is to clarify who can identify particular risks, what information they need from others, and which safeguards they can reasonably implement within their own services.

The Tech Coalition will therefore develop two publicly available resources:

  • Understanding the AI agent lifecycle: opportunities for safeguards and intervention, examining the stages of an AI agent’s development and use, and where child safety safeguards, monitoring, or intervention may be appropriate.
  • A map of child safety roles and capabilities across the AI ecosystem, explaining the visibility and potential child safety safeguards of different actors involved in developing, providing, and using AI systems.

These resources are intended to help practitioners move from general concerns about emerging technology toward a clearer understanding of where action is possible.

We are also testing the potential value of sharing prompt-based signals through Lantern, our cross-platform signal-sharing program, including their potential to provide insights into emerging trends in AI-facilitated abuse. The utility of these signals, their limitations, and appropriate safeguards for sharing them remain important considerations. The Tech Coalition will share high-level findings from this pilot in the 2026 Lantern Transparency report.

Strengthening responses when users disclose abuse

AI chatbots and other conversational systems may receive disclosures from people experiencing abuse, individuals concerned about a child’s safety, or users describing behavior that could put children at risk.

Participants examined risks specific to conversational AI, including emotionally responsive chatbots and systems designed for sustained interactions with users. Discussions emphasized the need for context-sensitive safeguards that account for how risks evolve over the course of a conversation, rather than assessing individual messages in isolation.

These discussions raised important operational questions: How should an AI system respond to potentially harmful interactions? When should a company escalate an interaction for human review? What reporting obligations apply, and how can users receive accurate information about available help?

Effective responses require more than appropriate chatbot language. Companies also need operational processes for assessing concerns, managing escalation, and connecting people with suitable support, while navigating legal, privacy, and safeguarding requirements.

The Tech Coalition previewed its now published guide, Responding to disclosures of child sexual exploitation and abuse through AI systems during the chatbot workstream. Commissioned by Anthropic, Google, and OpenAI, the guide provides practical guidance for companies addressing these challenges.

Following the forum, the Tech Coalition will undertake further work and consultation to explore how companies can test and apply the guidance across different contexts, and where it could be strengthened:

  • Next month, the Tech Coalition will convene stakeholders at an invitation-only gathering in Singapore to examine how the guide’s principles apply in the Asia-Pacific regional context, and where additional guidance may be needed.
  • The Tech Coalition also intends to engage globally with policymakers, hotlines, law enforcement, and organizations supporting children and survivors, exploring how company response procedures can connect more effectively with wider safeguarding and reporting systems.
  • Through the Tech Coalition Safe Online Research Fund , we will also support independent research building on learnings from this work. One priority for the Tech Coalition would be to further explore questions such as how AI systems communicate reporting obligations, actions taken, and the limits of confidentiality to people who disclose abuse. 

Testing AI systems safely and sharing what works

American-Institute-of-Architects-DC-Event-Photojournalism-photography

AI red teaming involves adversarial testing of systems to identify weaknesses, potential misuse, and opportunities to strengthen safeguards, as our recent explainer sets out. In the child safety context, this work must account for particular legal, technical, and ethical constraints.

Testing may require expertise from outside the companies developing AI systems, including people who understand patterns of exploitation, investigative processes, and the experiences of children and survivors.

During the forum, participants explored these challenges through a practical scenario, considering who should contribute to testing, how different forms of expertise can be brought together, and what information can be shared safely.

A central challenge surfaced was ensuring that testing findings translate into practical safeguard improvements. Participants emphasized that findings need to reach engineering, product, and moderation teams early enough to inform development and operational responses. Beyond identifying and blocking harmful outputs, teams should use what they learn to strengthen detection, intervention, and reporting processes.

Sharing findings from adversarial testing also requires care. Information that helps one organization improve safeguards could create opportunities for misuse if disclosed without appropriate protections. Participants discussed the need for secure mechanisms to share relevant findings, while balancing transparency, accountability, and the risks of exposing vulnerabilities.

These discussions underscored the importance of continued cross-sector collaboration to address barriers to rigorous, lawful testing and ensure that lessons learned can strengthen safeguards across the AI ecosystem.

From discussion to continued collaboration

The forum brought together people with different roles in preventing and responding to child sexual exploitation and abuse. Together, they explored where safeguards are needed, who can act, and how they can coordinate across sectors to address gaps and practical barriers.

Not every challenge can be resolved in a single meeting. The pace of AI development, the movement of harmful activity across services, and differences between legal and operational environments make continued cooperation important.

The Tech Coalition plans to build on the forum through regional multi-stakeholder gatherings across Asia-Pacific, Europe, and the United States in 2027. These activities will provide opportunities to revisit emerging risks, assess progress on practical safeguards, and draw on regional expertise to identify priorities for further collaboration. 

Acknowledgments

Thank you to Anthropic and OpenAI for co-chairing the forum’s planning group, and Adobe, Block, Bumble, Cantina Labs, Cloudflare, Microsoft, Pinterest, Spotify, Stability AI, and Yahoo for helping shape the event as part of the planning group.

We also thank all speakers, facilitators, and participants for sharing their expertise and contributing openly to the discussions. 

We look forward to taking these commitments forward with our members and stakeholders across sectors, strengthening our collective efforts to prevent and respond to child sexual exploitation and abuse in the age of AI.