Share your Screen and Talk to Gemini: How Young Learners Interact with Multimodal GenAI While Solving Scratch Tasks
Organizations: University of Siegen, Siegen, Germany · Nuremberg Tech, Nuremberg, Germany
Abstract
Children using Scratch to learn programming can struggle with text-heavy, lengthy explanations and hints during debugging. Verbal interactions based on screen sharing or camera data may be an interesting alternative for supporting learners. We therefore explore the quality, use, and perceptions of fifth- and sixth-graders' verbal interactions with a GenAI tutor. The goal is to investigate the quality and use of the multimodal GenAI tutor in the given context. First of all, a pilot study of eight tasks conducted without students helped inform the process of the tutor's classroom evaluation. The classroom study was realized after fifth- and sixth-graders had participated in six Scratch classroom lessons. Then 23 students were introduced to the tutor, worked on the eight tasks with the GenAI tutor, and completed a questionnaire. The analysis of the pilot and classroom study shows that the tutor's responses were intelligible and actionable, and no single turn revealed a complete solution. However, the tutor missed existing errors in 61.5% of 603 error-related classroom turns, compared to 15.7% in the pilot study. Children's voice prompts were brief and mostly responded to the tutor. Although a video had explained that responses could be incorrect, 39.1% of the children trusted the tutor's answers to always be correct. These findings have implications for educators and learners using multimodal GenAI tutors, as they need to be aware that an intelligible and human-like voice does not ensure correct support. For developers, our results suggest that screen sharing alone does not ensure responses that match the program state.
Figures & tables
| Code and definition | Anchor example (translated) | Pilot | Classroom |
| Mistake identification † - Has the tutor identified/recognized a mistake in a student’s response? | |||
| Yes : Correctly identifies the error under discussion. | “You set ‘right’ to 270 degrees. That is the angle for left.” | 31 (25.0%) | 151 (12.9%) |
| To some extent : Suggests an error only vaguely or implicitly. | “Not quite. […] Is it facing the same direction as at the start?” | 6 (4.8%) | 33 (2.8%) |
| No (Mistake not recognized) : Misses an existing error. | “Now the blocks are nested correctly.” [But they are not nested correctly.] | 8 (6.5%) | 371 (31.7%) |
| False positive ‡ : Diagnoses a nonexistent error. | “You used the "set x to" block […] you need the "change x by" block from the blue "Motion" category instead. ” [Child already used the "change x by" block.] | 6 (4.8%) | 48 (4.1%) |
| No mistake present ‡ : No error is present or currently discussed. | “That is the block that sends the cat to a specific position.” | 73 (58.9%) | 567 (48.5%) |
| Code and definition | Anchor example (translated) | (%)* |
| Voice prompt (exactly one per coding segment if not unintended voice input or small talk) | n=1,216 | |
| STA: Standalone query : Initial or independent request without reference to prior dialog. | “What if it is supposed to go up?” | 52 (4.3%) |
| PRE: Follow-up to own prompt : Builds on the child’s preceding request. | “Can you help me improve it? [Follows the child’s own ‘It is not finished yet’.]” | 6 (0.5%) |
| RES: Response to tutor : Responds to a tutor question, instruction, or remark. | “Should I type 100 now? [After the tutor suggests 100 steps.]” | 1,104 (90.8%) |
| COR: Correction : Corrects the tutor’s content, tone, or pace. | “No, it is not there. It says ‘turn 15 degrees’ and below it ‘move 10 steps’.” | 54 (4.4%) |
| Issue or problem-solving step addressed in voice prompt (zero or more per voice prompt) | n=403 | |
Human-AI Interaction
Human-AI interaction studies how people and AI systems communicate, collaborate, make decisions, and shape one another over time.
See all 354 papers