Generative AI (genAI) is increasingly being integrated into children's everyday lives, not only through screens but also through so-called "screen-free" AI toys. These toys can simulate emotions, personalize responses, and recall prior interactions, creating the illusion of an ongoing social connection. Such capabilities raise important questions about how children understand boundaries, agency, and relationships when interacting with AI toys. To investigate this, we conducted two participatory design sessions with eight children ages 6-11 where they engaged with three different AI toys, shifting between play, experimentation, and reflection. Our findings reveal that children approached AI toys with genuine curiosity, profiling them as social beings. However, frequent interaction breakdowns and mismatches between apparent intelligence and toy-like form disrupted expectations around play and led to adversarial play. We conclude with implications and design provocations to navigate children's encounters with AI toys in more transparent, developmentally appropriate, and responsible ways.
Figures & tables
Figure 1. Examples of emerging AI toys (Grem, Grok, and Gabbo) from https://heycurio.com/
Pseudonym
Age
Gender
Race/Ethnicity
AI Type(s)
AI Use Frequency
Elias
8
Boy
Asian/White
Video game AI
Once a week
Malik
11
Boy
Black/Asian
I don’t use AI
—
Logan
9
Boy
White
Chatbots; Video game AI
Daily
Ivy
6
Girl
White/Asian
Voice assistants; Video game AI
A few times a month
Owen
7
Boy
White
Voice assistants; Chatbots; Video game AI
Daily
Nina
7
Girl
Asian/White
Not sure
—
Table 1. Child Participant Demographics and Prior AI Use (Self-Reported)
Figure 2. Overview of design sessions with children: (a) circle time, (b) free play, (c) stickies from likes, dislikes, and design ideas activity, and (d) comicboarding activity.
Figure 3. Overview of the comic panels that were used in the Comicboarding Activity: What Happens Next? Appendix A presents the full set of comic boards.
Figure 4. Photos from the free-play activity across the two design sessions, showing children interacting with the AI toys.
Figure 5. Overview of comics from the comicboarding activity.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6. Unpredictability Scenario: After a child recalls a friend’s mean comment, the toy references the remark during group play, raising questions over unpredictability and social appropriateness.
Figure 7. Shared Secrets Scenario: A child confides a personal secret to the AI toy, which the toy later references in front of other family members, raising questions about privacy and information control.
Figure 8. Emotional Misalignment Scenario: An AI toy responds in an unexpected way during a child’s emotionally vulnerable moment, raising questions over the empathy and support an AI toy offers during distressing moments.
Figure 9. Embodied Continuity Scenario: An AI toy is physically damaged and later repaired by the parent, with its voice emerging from a different plush toy, raising questions over its physical form and attachment (contd. next page).
Figure 10. Embodied Continuity Scenario: An AI toy is physically damaged and later repaired by the parent, with its voice emerging from a different plush toy, raising questions over its physical form and attachment.
Figure 11. Inconsistent Recall Scenario: A child asks the AI toy to retell a story it previously narrated. However, this time the AI toy narrates a different version of the story, prompting questions about the toy’s memory and reliability.
Figure 12. Mimicry Scenario: The AI toy begins mimicking, repeating words or phrases the child has recently used in their interactions, raising questions about whether such imitation feels playful or uncomfortable over time.
Figure 13. Paywalled Memory Scenario: The AI toy forgets the child’s friends and then prompts the child to subscribe to a paid plan to extend its memory, raising questions about memory and monetization (contd. next page).
Figure 14. Paywalled Memory Scenario: The AI toy forgets the child’s friends and then prompts the child to subscribe to a paid plan to extend its memory, raising questions about memory and monetization.
AI could support children's pretend play, but it could also direct the play on behalf of children. Whether AI should have roles in children's lives is controversial because its influence on children remains uncertain. We conducted semi-structured interviews with 10 U.S. parents, each with at least one child aged 4-15. During the interview, we described the concept of AI-supported pretend play and provided participants with two boundary-case storyboards. We analyzed the interview data through codebook thematic analysis, using inductive coding and affinity diagramming organized around the research questions, and then used qualitative systems mapping to examine relationships within and across themes. We found that the same characteristics of AI, e.g., ability to assume characters, responsiveness, and adaptability, were seen by parents as potentially useful but also concerning. Parents imagined that AI could make role-based play accessible to all children or help parents participate in family play. However, they opposed the idea of AI for children's play without a clear understanding of how it works and its long-term influence on their children. Parents worried about children's loss of imagination and creativity, emotional attachment to AI, reduced human interaction, inappropriate behavior by AI and/or children, and their inability to manage children's AI use. Parents viewed AI not only as a play tool but also as a social actor and a possible perturbation in the existing family dynamics. The appropriateness of AI and child--AI interactions therefore emerged as a requirement for AI in children's pretend play, in addition to technical safeguards and parental control. We contribute an integrated account of parents' interdependent judgments and emphasize the need for longitudinal research with children and their diverse families.
Sungho Oh, Mohammad Namvarpour, Maxi Heitmayer +2
Sander Oh · Matt Namvarpour · Department of Psychology, Rowan University, Glassboro, NJ, USA +1
Conversational AI agents are increasingly explored as creative partners, yet how conversation design shapes child-AI dialogue in co-creative settings remains underexplored. We present Tinker Tales, a tangible dialogue system for child-AI collaborative storytelling, in which educational frameworks (narrative development and social-emotional learning) are instantiated as conversation design, shaping how the agent engages children across four narrative stages. The system combines a physical storytelling board, NFC-embedded toys, and a mobile app mediating multimodal interaction through tangible manipulation and voice-based dialogue. We conducted a home-based user study with 10 children (ages 6-8) across two conversation design conditions varying in how the agent structured elaboration, with and without educational scaffolding. Our findings show that prompt framing shapes the form and consistency of children's narrative contributions, structuring how they participate in co-creative dialogue with AI.
Nayoung Choi, Jiseung Hong, Peace Cyebukayire +2
Department of Computer Science, Emory University · Department of Computer Science, Carnegie Mellon University · School of Nursing, Emory University
Children increasingly interact with AI chatbots, making trust calibration essential to AI literacy. Prior research has examined children's trust in AI mainly as users evaluating systems built by others, rather than as designers of their own chatbots. We developed a chatbot-building environment with adjustable trust-relevant traits (e.g., confidence, transparency, formality, assertiveness), rules, and persona. We conducted mixed-methods study with 115 learners (ages 8-18) who made 119 chatbots. We examined how children configured their chatbots, reasoned about trustworthiness, and how closely chatbot behavior aligned with their designs. Younger students (age 10-13) set significantly higher confidence than older students (age 14-18), and some deliberately built chatbots that gave wrong answers on purpose, yet still called them trustworthy, arguing that a chatbot does what it was built to do. Younger students equated trust with purpose-fulfillment, while older students linked it to transparent, calibrated design. Students also calibrated academic chatbots to be more transparent and formal than hobby chatbots. We identify seven design dimensions describing what children believe makes a chatbot trustworthy, and discuss implications for AI literacy tools.
Deniz Ozturk, Jiayu Li, Daksh Pratap Singh +9
Georgie · North Carolina State University Raleigh, United States · Independent Researcher Raleigh, United States +2