Organizations: Systems and Computer Engineering, Carleton University, Ottawa, Canada · School of Information Technology, Carleton University, Ottawa, Canada · Department of Law and Legal Studies, Carleton University, Ottawa, Canada
Children are increasingly using technologies powered by Artificial Intelligence (AI). However, there are growing concerns about privacy risks, particularly for children. Although existing privacy regulations require companies and organizations to implement protections, doing so can be challenging in practice. To address this challenge, this article proposes a framework based on Privacy-by-Design (PbD), which guides designers and developers to take on a proactive and risk-averse approach to technology design. Our framework includes principles from several privacy regulations, such as the General Data Protection Regulation (GDPR) from the European Union, the Personal Information Protection and Electronic Documents Act (PIPEDA) from Canada, and the Children's Online Privacy Protection Act (COPPA) from the United States. We map these principles to various stages of applications that use Large Language Models (LLMs), including data collection, model training, operational monitoring, and ongoing validation. For each stage, we discuss the operational controls found in the recent academic literature to help AI service providers and developers reduce privacy risks while meeting legal standards. In addition, the framework includes design guidelines for children, drawing from the United Nations Convention on the Rights of the Child (UNCRC), the UK's Age-Appropriate Design Code (AADC), and recent academic research. To demonstrate how this framework can be applied in practice, we present a case study of an LLM-based educational tutor for children under 13. Through our analysis and the case study, we show that by using data protection strategies such as technical and organizational controls and making age-appropriate design decisions throughout the LLM life cycle, we can support the development of AI applications for children that provide privacy protections and comply with legal requirements.
Figures & tables
Aspect
COPPA
GDPR
PIPEDA
Age Threshold
Under 13
Under 16 (may be lowered to 13 by Member States)
No fixed age; capacity-based
Consent Requirement
Verifiable parental consent
Parental consent required for children
Meaningful consent required; maturity assessed
Automated Profiling
Not explicitly addressed
Right not to be subject to automated decision-making
Not explicitly addressed
Relevance to LLMs
Challenges with verifying consent for dynamic interactions
Difficulties with transparency, personalization, and data inference
Ambiguity in consent maturity and inferred data handling
TABLE I: Key Regulatory Parameters in COPPA, GDPR, and PIPEDA
Privacy Right
COPPA
GDPR
PIPEDA
Right to Access
~
✓
✓
Right to Rectification
×
✓
✓
Right to Erasure
×
✓
~
Right to Restrict Processing
×
✓
~
Right to Object
×
✓
×
Right to Data Portability
×
✓
×
TABLE II: Summary of Privacy Rights Across COPPA, GDPR, and PIPEDA
Privacy Principle
COPPA
GDPR
PIPEDA
Data Minimization
~
✓
✓
Purpose Limitation
~
✓
✓
Transparency and Explainability
✓
✓
~
User Rights
~
✓
✓
Accountability
~
✓
✓
Security by Design
✓
✓
✓
TABLE III: Summary of Privacy Principles Across COPPA, GDPR, and PIPEDA
Fig. 1: LLM System Lifecycle Architecture (adapted from ISO/IEC 5338:2023)
LLM Lifecycle Stage
Relevant Regulatory Principles
Technical and Organizational Controls
Data Collection
Data Minimization; Purpose Limitation; Meaningful Consent Mechanisms
Input filtering to remove unnecessary or sensitive data [ 89 ] ; limited collection of identifiers and metadata; purpose-subject data tagging [ 90 ] ; scoped access controls [ 104 ] ; verifiable parental consent; age - appropriate consent interfaces and parental dashboards.
Model Training
Data Minimization; Security by Design; Accountability
Task-specific fine-tuning with minimal datasets [ 105 ] ; PII removal or anonymization [ 106 , 107 ] ; encrypted data storage and access controls [ 108 , 109 ] ; validation against data poisoning [ 21 ] ; differential privacy [ 110 ] ; gradient-based pruning to mitigate backdoors [ 95 ] ; data provenance records [ 111 ] ; DPIAs [ 97 ] ; Datasheets [ 49 ] and Model Cards.
Operation and Monitoring
Meaningful Consent; Transparency and Explainability; Purpose Limitation; Security by Design
Real-time input filtering [ 89 ] ; parental dashboards and consent controls [ 90 ] ; age-appropriate explanations [ 91 , 92 ] ; purpose-restricted data use [ 90 ] ; ephemeral session memory [ 112 ] ; default exclusion of interaction data from training; detection of prompt injection, jailbreaks, unsafe outputs [ 21 ] .
Continuous Validation
Accountability; User Rights; Meaningful Consent Mechanisms; Security by Design
Periodic audits and DPIAs [ 113 ] ; logging of system and consent changes; clear interfaces to access, modify or delete data [ 114 ] ; revalidation of parental consent; adversarial testing [ 115 ] ; integrity checks and model audits [ 113 ] .
TABLE IV: Mapping Privacy Principles and Controls Across the LLM Lifecycle
Fig. 2: Educational LLM Tutor Architecture Aligned to Lifecycle Stages and Privacy Principles
Responsible AI studies how to design, evaluate, and govern AI systems to promote fairness, safety, transparency, accountability, privacy, and sustainability.