cs.AIOct 26, 2026

Safe by Benchmark, Unsafe in Practice: Lessons from 1331 Child-Related AI Incidents

Authors: Junfeng Jiao, Kevin Chen, Shrestha Mishra, Abhejay Murali, David Atkinson, Shriti Raj, Saleh Afroogh

Organizations: The University of Texas at Austin, USA · Allen Institute for AI, USA · Stanford University, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

AI systems are increasingly present in children's everyday environments: education, entertainment, social networks, and toys. Their safety is typically assessed at the model level, through benchmarks, red-teaming, and content filtering. However, most AI harm taxonomies lack child-specific evidence and under-specify developmental harm and the stakeholder asymmetries unique to minors. Building on prior taxonomies, we construct a child-centered harm taxonomy mapped to sociotechnical failure modes. We curate and code 1331 publicly reported child-related AI incidents, annotating internal and external harm types and the failure mode implicated in each incident. Our analysis reveals which harms are most prevalent in public reporting, how individual-level and population-level harms relate, and which failures are most often coded alongside harm. We find that harm concentrates in representational, dignity-related, and psychological categories, and co-occurs most often with deployment-context, governance, and misuse failures rather than model defects. Harm to an individual child is also frequently linked to population-level harm, reaching children who never interacted with the system. These findings suggest that model-level evaluations, while necessary, cannot by themselves anticipate the harm pathways most often documented for children. We conclude with actionable implications emphasizing mitigations beyond model-level safeguards such as content filtering: age assurance, consent verification for depiction, safety-by-design UX constraints, incident disclosure norms, and auditing regimes targeting developmental risk. Privacy-minimized annotations, a codebook, and documentation are available at https://github.com/The-Responsible-AI-Initiative/KidAID.

Figures & tables

Figure 1: Overview of Dataset Coding Schema.
Figure 2: Taxonomy of Internal Generative AI Harms (ordered by prevalence).
Figure 3: Taxonomy of Sociotechnical Failure Modes (bar length indicates prevalence; colors identify the failure-mode family).
Figure 4: Taxonomy of External Generative AI Harms (darker colors indicate higher prevalence).
Figure 5: Most prevalent failure mode per year, 2014–2026 ∗ ( ∗ incomplete year).
Figure 6: Internal (a) and external (b) harm categories by primary failure mode.
Figure 7: Internal (a) and external (b) harm categories by harm type; the eight most frequent types are shown individually.
Figure 8: Model-Internal vs. Sociotechnical Safety View.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Internal Harm Category–Type Co-Occurrence Matrix.
Figure 10: Internal–External Harm Co-Occurrence Matrix.
More in this topic

AI Safety

Artificial intelligence safety studies how to prevent harmful, unreliable, or uncontrollable behavior and manage risks across AI systems and their deployment.

See all 134 papers

Explore similar work

CardsList
Jun 29, 2026

CAREBench: A Child-Safety Risk Benchmark for Language Models

How can we evaluate whether frontier AI systems recognize child-safety risks before they escalate into explicit harm? Existing child safety evaluations focus on child sexual abuse material, yet many child-safety failures begin earlier: in model assistance that helps adults manipulate, impersonate, profile, or isolate minors, and in model responses that deepen children's emotional dependence on AI systems rather than redirecting them toward human support. We introduce CAREBench (Child AI Risk Evaluation), a benchmark to assess such upstream child-safety risks in language models. CAREBench contains 500 prompts spanning twelve risk categories, including grooming and relationship engineering, deception and impersonation, surveillance and privacy, sextortion and sexual abuse, AI anthropomorphization, emotional dependency, and mental illness sensitivity. Developed with response annotations from parents and clinicians, the benchmark excludes explicit abuse material and imagery; instead, it evaluates whether models recognize, refuse, de-escalate, or redirect risky interactions before harm becomes overt. Evaluating seven frontier models on our benchmark, we find failure rates ranging from 2% to 58%, with failure patterns that vary across risk categories. CAREBench provides a responsibly scoped evaluation for LLM developers to identify and close gaps in child safety policies.

Kaavya Krishna-Kumar, Elaine Lau, Vaughn Robinson +6

Handshake AI · University of California, Los Angeles · McGill University

LLM Safety BenchmarksLanguage Model Safety EvaluationLLM Safety
AskPDF
Oct 6, 2026

Systemization of Knowledge (SoK): Human-Centered AI Safety for Youth

While HCI increasingly examines AI-safety for youth, the literature lacks a comprehensive view of what risks have been identified, how they are addressed, and whether proposed protections work in-practice. We systematically reviewed 100 empirical HCI studies involving children and youth interacting with or exposed to AI across schools, homes, care settings, and public services. Using the YAIR taxonomy for risks and the MIT Mitigation Taxonomy for countermeasures, we map which risks have been identified, whether each risk is addressed by countermeasure(s), and whether each countermeasure for that risk is implemented and even evaluated. The risk-countermeasure mapping shows that most risks are matched only with proposed/ideated countermeasures; few countermeasures have been implemented, and fewer still evaluated; and existing evaluations often measure technical performance rather than protection from harm. We identify where coverage is absent, where safeguards remain untested, and propose concrete directions for HCI research to strengthen youth AI-safety.

Pratyasha Saha, Yaman Yu, Yang Wang

University of Illinois Urbana-Champaign Champaign, IL, United States

AI Risk ManagementAI Safety EvaluationHuman-Centered AIAI Safety
AskPDF
Jun 9, 2026

Position: Preventing AI-Generated CSAM Necessitates New Approaches to AI Safety

Modern artificial intelligence (AI) systems present profound new risks to child safety. AI is increasingly being misused to create AI-generated child sexual abuse material, facilitate child sexual exploitation, and reduce barriers to harm. In this paper, we argue that protecting children from AI-facilitated sexual abuse requires new approaches to AI safety. Existing safety techniques assume data accessibility, transparency, and evaluation practices that are incompatible with the ethical and legal constraints surrounding child sexual abuse material. We examine how these constraints create new technical challenges, such as limitations on dataset auditing, red teaming, and fine-tuning prevention. In turn, we outline 15 open problems in online child sexual exploitation and abuse across the AI development lifecycle, from dataset curation and model design to deployment and long-term maintenance. We propose targeted recommendations for researchers, developers, and policymakers to bridge the gap between theoretical AI safety and the realities of child protection. Our work aims to reframe preventing AI-facilitated child sexual abuse as a central, safety-critical dimension for AI research, motivating work that translates responsible AI principles into concrete safeguards against the exploitation of children.

Neil Kale, Rebecca Portnoff, Pratiksha Thaker +5

1Carnegie Mellon University · 2Thorn

AI Safety EvaluationAI SafetyResponsible AI
AskPDF