ELLA: Generative AI-Powered Social Robots for Early Language Development at Home

Victor Nikhil Antony; Shiye Cao; Shuning Wang; Chien-Ming Huang

Figure 1. We present ELLA (Early Language Learning Agent), an interactive storytelling robot designed with educators and families to support early language development in children ages 4–6. The robot tells at most four stories per day upon request with a goal of teaching four target words per day determined by parents. The robot incorporates the target words into stories generated using LLMs based on the children’s interests and engages the child through targeted and scaffolded interactions. This figure illustrates one sample story session from our deployment study. The children’s name was replaced with pseudonyms. The target vocabulary word that the robot is trying to teach is “Sympathy” and the theme of this story is the TV show “Fancy Nancy”. Greeting/farewell from different story sessions were presented to showcase different possibilities. This figure presents a flowchart illustrating the structure of an interactive storytelling session with ELLA, the Early Language Learning Agent. The target vocabulary word (sympathy) and the story theme (Fancy Nancy, a television show) are specified at the start of the session. The interaction begins with a Greeting stage, where the robot introduces itself and asks the child whether they would like to hear a story. If the child agrees, the robot proceeds to Storytelling, narrating a story that embeds the target word in context and provides a child-friendly definition. The session then moves through Story Perception, Story Recall, and Target Word Practice stages, during which the robot checks the child’s understanding and encourages reflection on the target word through guided questions. At the end of the session, the robot asks whether the child would like to hear another story. If the child chooses “yes,” the flow returns to storytelling for an additional story. If the child chooses “no,” or if the maximum number of stories has been reached, the interaction ends with a Farewell stage.

Figure 2. To design a social robot to support language development in children ages 4- 6, we engaged in a three-stage design process involving: 1) online interviews; 2) in-home workshops; and 3) in-home deployment study. This figure outlines a three-stage design process used to develop a social robot to support language development in children aged 4–6. Stage 1 consists of online interviews with parents (n=7) and educators (n=5) to understand needs and inform the initial system design. Stage 2 consists of in-home workshops (n=12) organized into multiple iterative phases: Phase 1 (n=3) focuses on improving interaction quality; Phase 2 (n=4) focuses on improving story content; Phase 3 (n=3) focuses on improving interaction questions; and Phase 4 (n=2) focuses on improving onboarding. Stage 3 is an eight-day in-home deployment study (n=10) with four steps: (1) an introduction letter from ELLA; (2) onboarding with a vocabulary test and setup; (3) eight days of interactive storytelling with ELLA; and (4) offboarding with a vocabulary test, interview, and survey.

Pseudo- nyms	Age	Gender	Ethnicity	Behavioral & learning differences	Household size	Household income	50% AMI	80% AMI	Days/week reading (Eng.)	Minutes/day reading (Eng.)	Phases involved
P1-Grace*	4	female	Asian	typical	6	$100k–$ 125k	above	below	5 days	10–20 min	1, 2, 3
P2-Andrew	4	male	Black and White	speech delay, speech therapy, undergoing diagnosis for behavioral & neurological differences	6	$50k–$ 75k	below	below	4 days	10–20 min	1, 2, 3
P3-Sarah	4	female	White	typical	3	$>$ $200k	above	above	7 days	10–20 min	1, 2, 3
P4-Sam	4	male	Asian	speech therapy	4	$100k–$ 125k	above	above	2 days	10–20 min	1
P5-Tom	4	male	White	typical	4	prefer not to answer	above	above	7 days	$>$ 30 min	2
P6-George	4	male	Multiethnic	typical	3	$150k–$ 200k	above	above	7 days	0–10 min	3

Table 1. Family Demographics. (*) indicate siblings. ASD stands for autism spectrum disorder. ADHD stands for Attention-deficit/hyperactivity disorder. 50% of area median income (AMI) is a standard benchmark used by local government programs to define very low-income households, while 80 percent of AMI commonly defines low-income households ( of Housing and Development, ; Program, ; Aid, ) . Age indicates age at the start of the study.

P	Gender	Race	Education	Experience	School Type	Student Age	Student Socio-Economic Status
1	F	Hispanic	Master’s	7 yrs	Public (GE, SE, ESL)	4–6	Mixed
2	F	White	Bachelor’s	20 yrs	Public (GE, SE, ESL)	3–5	Mixed
3	F	White	Master’s	40 yrs	Private (GE)	4–11	High
4	F	Asian	Bachelor’s	3.5 yrs	Private (GE, ESL)	3–4; 14–18	High
5	M	Black	Master’s	7 yrs	Public Charter (GE, SE, ESL)	7–8; 10–14	Mixed

Table 2. Teacher Demographics. GE stands for General Education. SE stands for Special Education. ESL stands for English as a Second Language.

Figure 3. Pipeline for generating personalized and targeted content for interactive storytelling with ELLA. Design evolution ( Phase 1 , Phase 2 , Phase 3 , Phase 4 ) of components indicated by color. This figure presents the ELLA Interaction Content Generation Pipeline, with color-coded components indicating design evolution across Phases 1–4. The pipeline begins with inputs including the child’s name, target vocabulary word, and story theme, which are used to generate a story and associated interactions. The pipeline then branches, with one path converting the story to audio and producing a timestamped transcript, and the other passing the story and target word directly forward. Both paths feed into a high-level analysis stage that identifies interaction elements such as palettes, motifs, cues, and story segments. These elements are used to generate facial and bodily behaviors, which are converted into executable code and stored, along with the story and audio, in a cloud-based story database. Each stage in the diagram includes example artifacts illustrating the information passed between components.

Figure 4. Example story and behaviors. S denotes behaviors generated with the storytelling behavior generation pipeline. The rest of the behaviors are generated using the interaction behavior generation pipeline. Child name is replaced with pseudonym. This figure shows an example storytelling session with ELLA, including both spoken content and the robot’s accompanying behaviors. The child’s name is replaced with a pseudonym. The session begins with a greeting and proceeds through storytelling, during which the robot performs synchronized gestures (e.g., arm, head, and base movements) and changes facial expressions to emphasize the narrative and target vocabulary. The interaction also includes comprehension questions and feedback, and concludes with a farewell and sleep animation. Behaviors marked with S are generated by the storytelling behavior generation pipeline, while the remaining behaviors are generated by the interaction behavior generation pipeline. Each interaction state is paired with a corresponding image of the robot.

Figure 5. ELLA System Diagram. Design evolution ( Phase 1 , Phase 2 , Phase 3 , Phase 4 ) of components indicated by color. This figure presents a system diagram of ELLA, illustrating how speech input, story content, and interaction logic are integrated to generate robot behaviors. Color-coded components indicate design evolution across Phases 1–4. Considered as a whole, the system takes audio input from the child, processes it to determine turn-taking and interaction state, and uses language models to manage storytelling flow, moderation, and dialogue decisions. Story content is retrieved or generated and combined with high-level interaction handling to determine whether to continue storytelling, scaffold responses, or conclude the session. Based on these decisions, the system generates coordinated facial expressions, gestures, and animations, which are executed by the robot. The diagram highlights the end-to-end flow from user speech and story content to robot behaviors, emphasizing modular design and interaction safety.

Figure 6. ELLA Usage over time. This figure shows a bar chart of ELLA usage over time, measured as the number of stories told per day for each child across an eight-day deployment period. The horizontal axis represents the day of deployment, and the vertical axis represents the number of stories per day. Each color of bars corresponds to an individual child, illustrating day-to-day variation in usage. Bars with diagonal hatching indicate weekend days, during which usage patterns differ from weekdays. Across children, usage ranges from zero to four stories per day, reflecting differences in engagement frequency over time. The chart highlights both periods of consistent daily use and days with no usage, providing an overview of how storytelling activity varied across participants and across the deployment duration.

Who Initiated	n	Who Were Present	n	How Child Felt About Storytelling Session	n
Parent prompted	34	Parent	42	Neutral	23
Scheduled routine	19	Sibling	27	Excited	20
Child initiated	16	Child alone	5	Frustrated	7
Sibling encouraged	7	Other caregivers	2	Hesitant	6
Friend initiated	1	Friend	1	Want to continue beyond session	3
				Happy	1

Table 3. Context based on parent diaries.

Figure 7. Child survey results. The blue line marks the median. This figure presents results from a child survey evaluating interactions with ELLA. The survey includes items assessing enjoyment and perceived understanding, such as liking ELLA, finding the stories fun, feeling understood when talking to ELLA, and enjoying talking to ELLA. For each item, responses are summarized using descriptive statistics, with mean scores ranging from 3.9 to 4.2 and standard deviations between 1.1 and 1.32. The blue line indicates the median response for each survey item, providing a visual reference for central tendency across participants.

Pseudonym	Age	Target Words				Example Theme	Days used ELLA	Stories listened to
Grace	4	Imitate	Ordinary	Clumsy	Massive	Rapunzel (disney princess)	5	18
Andrew	4	Imagine	Confident	Permission	Self-control	Tom and Jerry (TV show)	4	13
Sarah	4	Perseverance	Compassion	Gumption	Awestruck	Pony	7	18
George	4	Permission	Consequences	Orbit	Chirp	Ironman	2	4
Jason	5	Imitate	Somersault	Clumsy	Frisky	Science	8	11
Natalie	5	Apartment	Advocate	Justice	Bait	Dance	7	12

Table 4. Target words and themes for each child during deployment. Green highlight represents words that the child learned by the end of the deployment study.

Pseudonym	Age	Stories listened to	Target words used	Target word usage during story sessions	Total target word used	Avg words/ turn
Grace	4	18	3/4	Massive (14), Ordinary (16), Clumsy (13), Imitate (0)	43	3.78
Andrew	4	13	3/4	Permission (3), Self-Control (2), Imagine (1), Confident (0)	6	3.48
Sarah	4	18	4/4	Compassion (2), Awestruck (2), Perseverance (0), Gumption (0)	4	3.48
George	4	4	1/4	Chirp (2), Permission (0), Consequences (0), Orbit (0)	2	2.46
Jason	5	11	2/4	Clumsy (8), Imitate (1), Somersault (0), Frisky (0)	9	4.91
Natalie	5	12	2/4	Advocate (2), Bait (1), Justice (1), Apartment (0)	4	3.02

Table 5. Child Language usage during deployment. Green highlight represents words that the child learned by the end of the deployment study.

Lesson: Inviting Caregiver Partnership Could Enrich Language Development
Design Opportunity 1	Make learning gains visible through caregiver-facing reflective summaries.
Design Opportunity 2	Extend language learning beyond session through robot-caregiver collaboration.
Lesson: Stories Need Freshness to Retain Playfulness Over Instructional Tone
Design Opportunity 3	Support low-effort, ongoing co-authoring to keep content aligned with shifting interests and goals.
Design Opportunity 4	Balance meaningful repetition and novelty through narrative variation.
Lesson: Making Talk Feel Safe Needs Legible, Low-Pressure Interaction

Table 6. Lessons Learned and Design Opportunities for ELLA

Figure 8. Our deployment data highlighted interesting interaction patterns that inspire new design opportunities. We highlight instances of object-based engagement, sibling dynamics, and sharing of personal experiences here. Pseudonyms are used, and the target vocabulary for the given interaction is highlighted in red. This figure presents qualitative examples from the deployment that illustrate recurring interaction patterns with ELLA. The examples highlight three types of engagement: object-based engagement, sharing of personal experiences, and sibling dynamics. In the object-based engagement example, a child temporarily leaves the interaction to retrieve a personal toy related to the story theme and target vocabulary. In the personal experience example, the child connects the story to a real-life adventure and expands on their own experiences during the interaction. In the sibling dynamics example, an older sibling provides whispered assistance while the child responds to the robot’s questions. Pseudonyms are used for all children, and target vocabulary words referenced in the interactions are highlighted in red.

Figure 9. Examples demonstrating additional interaction patterns. We highlight instances of parent prompting, playful rambling, and story requests here. Pseudonyms are used, and the target word for the given interaction is highlighted in red. Overlapping speech is shown in brackets. This figure presents examples from the deployment that illustrate recurring interaction patterns with ELLA. The examples highlight three types of engagement: parent prompting, playful rambling, and story request. In the parent prompting example, the parent helps model for the child how to say their one-word response in a full sentence. In the playful rambling example, the child responds with blah blah blah and unintelligible halfway through their sentence. In the story request example, a child interrupts Ella to ask for a specific story.

ELLA: Generative AI-Powered Social Robots for Early Language Development at Home

Abstract

Figures & tables

Human-Robot Interaction

Co-designing a Social Robot for Newcomer Children's Cultural and Language Learning

Robo-Blocks: Generative Scaffolding in End-User Design and Programming of Social Robots

Designing Social Robots for Inclusive Child Wellbeing Assessment: Insights from Communities Supporting Developmental Language Disorder and Forced Migration

Abstract

Human-Robot Interaction

Explore similar work

Co-designing a Social Robot for Newcomer Children's Cultural and Language Learning

Robo-Blocks: Generative Scaffolding in End-User Design and Programming of Social Robots

Designing Social Robots for Inclusive Child Wellbeing Assessment: Insights from Communities Supporting Developmental Language Disorder and Forced Migration