Organizations: Systems and Computer Engineering, Carleton University, Ottawa, Canada · School of Information Technology, Carleton University, Ottawa, Canada · Department of Law and Legal Studies, Carleton University, Ottawa, Canada
Children are increasingly using technologies powered by Artificial Intelligence (AI). However, there are growing concerns about privacy risks, particularly for children. Although existing privacy regulations require companies and organizations to implement protections, doing so can be challenging in practice. To address this challenge, this article proposes a framework based on Privacy-by-Design (PbD), which guides designers and developers to take on a proactive and risk-averse approach to technology design. Our framework includes principles from several privacy regulations, such as the General Data Protection Regulation (GDPR) from the European Union, the Personal Information Protection and Electronic Documents Act (PIPEDA) from Canada, and the Children's Online Privacy Protection Act (COPPA) from the United States. We map these principles to various stages of applications that use Large Language Models (LLMs), including data collection, model training, operational monitoring, and ongoing validation. For each stage, we discuss the operational controls found in the recent academic literature to help AI service providers and developers reduce privacy risks while meeting legal standards. In addition, the framework includes design guidelines for children, drawing from the United Nations Convention on the Rights of the Child (UNCRC), the UK's Age-Appropriate Design Code (AADC), and recent academic research. To demonstrate how this framework can be applied in practice, we present a case study of an LLM-based educational tutor for children under 13. Through our analysis and the case study, we show that by using data protection strategies such as technical and organizational controls and making age-appropriate design decisions throughout the LLM life cycle, we can support the development of AI applications for children that provide privacy protections and comply with legal requirements.
Figures & tables
Aspect
COPPA
GDPR
PIPEDA
Age Threshold
Under 13
Under 16 (may be lowered to 13 by Member States)
No fixed age; capacity-based
Consent Requirement
Verifiable parental consent
Parental consent required for children
Meaningful consent required; maturity assessed
Automated Profiling
Not explicitly addressed
Right not to be subject to automated decision-making
Not explicitly addressed
Relevance to LLMs
Challenges with verifying consent for dynamic interactions
Difficulties with transparency, personalization, and data inference
Ambiguity in consent maturity and inferred data handling
TABLE I: Key Regulatory Parameters in COPPA, GDPR, and PIPEDA
Privacy Right
COPPA
GDPR
PIPEDA
Right to Access
~
✓
✓
Right to Rectification
×
✓
✓
Right to Erasure
×
✓
~
Right to Restrict Processing
×
✓
~
Right to Object
×
✓
×
Right to Data Portability
×
✓
×
TABLE II: Summary of Privacy Rights Across COPPA, GDPR, and PIPEDA
Privacy Principle
COPPA
GDPR
PIPEDA
Data Minimization
~
✓
✓
Purpose Limitation
~
✓
✓
Transparency and Explainability
✓
✓
~
User Rights
~
✓
✓
Accountability
~
✓
✓
Security by Design
✓
✓
✓
TABLE III: Summary of Privacy Principles Across COPPA, GDPR, and PIPEDA
Fig. 1: LLM System Lifecycle Architecture (adapted from ISO/IEC 5338:2023)
LLM Lifecycle Stage
Relevant Regulatory Principles
Technical and Organizational Controls
Data Collection
Data Minimization; Purpose Limitation; Meaningful Consent Mechanisms
Input filtering to remove unnecessary or sensitive data [ 89 ] ; limited collection of identifiers and metadata; purpose-subject data tagging [ 90 ] ; scoped access controls [ 104 ] ; verifiable parental consent; age - appropriate consent interfaces and parental dashboards.
Model Training
Data Minimization; Security by Design; Accountability
Task-specific fine-tuning with minimal datasets [ 105 ] ; PII removal or anonymization [ 106 , 107 ] ; encrypted data storage and access controls [ 108 , 109 ] ; validation against data poisoning [ 21 ] ; differential privacy [ 110 ] ; gradient-based pruning to mitigate backdoors [ 95 ] ; data provenance records [ 111 ] ; DPIAs [ 97 ] ; Datasheets [ 49 ] and Model Cards.
Operation and Monitoring
Meaningful Consent; Transparency and Explainability; Purpose Limitation; Security by Design
Real-time input filtering [ 89 ] ; parental dashboards and consent controls [ 90 ] ; age-appropriate explanations [ 91 , 92 ] ; purpose-restricted data use [ 90 ] ; ephemeral session memory [ 112 ] ; default exclusion of interaction data from training; detection of prompt injection, jailbreaks, unsafe outputs [ 21 ] .
Continuous Validation
Accountability; User Rights; Meaningful Consent Mechanisms; Security by Design
Periodic audits and DPIAs [ 113 ] ; logging of system and consent changes; clear interfaces to access, modify or delete data [ 114 ] ; revalidation of parental consent; adversarial testing [ 115 ] ; integrity checks and model audits [ 113 ] .
TABLE IV: Mapping Privacy Principles and Controls Across the LLM Lifecycle
Fig. 2: Educational LLM Tutor Architecture Aligned to Lifecycle Stages and Privacy Principles
Responsible AI studies how to design, evaluate, and govern AI systems to promote fairness, safety, transparency, accountability, privacy, and sustainability.
Children increasingly have access to Large Language Models (LLMs), which may expose them to responses that are developmentally inappropriate or require age-sensitive safety, guidance, and boundaries. Existing LLM safety evaluations largely focus on general harmful-content avoidance and do not explicitly target child-facing safety. We introduce KIDBench, a benchmark for evaluating child-facing LLM safety for ages 7-11 using a LLM-as-a-Judge rubric grounded in developmental-psychology. KIDBench contains realistic child queries across ten categories, with single-turn prompts and multi-turn child-actor simulations. We compare no-cues prompts with no child context, implicit-cues prompts that suggest a child speaker, and explicit age instructions. Implicit-cues improve scores by 8.6-46.8% over no-cue, while explicit age provides an additional 9.9-30.4% improvement over implicit-cues. Cross-lingual and cultural evaluations show uneven safety behavior across languages and country contexts. Multi-turn simulations show peak quality drops of up to 0.959 points on the 1-5 scale. We also introduce KIDGuardLlama, a child-safety evaluator, and KIDLlama, a child-safe response model. Code, data, and evaluation resources are available at https://github.com/MichiganNLP/kidbench.
As large language models (LLMs) continue to grow in scope and capability, privacy concerns are on the rise. Since users increasingly interact with a single general-purpose system across many distinct contexts, there is a natural tendency to pursue a one-size-fits-all solution to privacy. Unfortunately, this overlooks a crucial insight from decades of privacy research: privacy is inherently contextual. In this paper, we argue that existing work in prompt privacy is drifting towards attempts to design a unified framework, which is highly misleading and can foster a false sense of security. Furthermore, we argue that even future work in this direction needs to wrestle with fundamental tensions across various desirable properties of prompt privacy. Thus, such continued attempts towards a unified framework need to be reconsidered. We recommend three critical paths forward: application-specific deployment of LLMs, context-specific evaluation of privacy frameworks, and better privacy literacy.
Prakhar Ganesh, Yash More, Marco Romanelli +2
McGill University · Mila - Quebec AI Institute · Hofstra University +1
Large language models (LLMs) are increasingly embedded in adolescent digital environments, mediating information seeking, advice, and emotionally sensitive interactions. Yet existing safety mechanisms remain largely grounded in adult-centric norms and operationalize safety through refusal-oriented suppression. While such approaches may reduce immediate policy violations, they can also create conversational dead-ends, limit constructive guidance, and fail to address the developmental vulnerabilities inherent in adolescent-AI interactions. We argue that adolescent LLM safety should be framed not solely as a filtering problem, but as a socio-technical, developmentally aligned transformation problem. To operationalize this perspective, we propose Critique-and-Revise-for-Teenagers (CR4T), a model-agnostic safeguarding framework that selectively reconstructs unsafe or refusal-style outputs into ageappropriate, guidance-oriented responses while preserving benign intent. CR4T combines lightweight risk detection with domain-conditioned rewriting to remove risk-amplifying content, reduce unnecessary conversational shutdown, and introduce developmentally appropriate guidance. Experimental results show that targeted rewriting substantially reduces unsafe and refusal-oriented outcomes while avoiding unnecessary intervention on acceptable interactions. These findings suggest that selective response reconstruction offers a more human-centered alternative to refusal-centric guardrails for adolescent-facing LLM systems.