ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluation

Rose Niousha; Mihran Miroyan; Abigail O'Neill; Joseph E. Gonzalez; Gireeja Ranade; John DeNero; Narges Norouzi

cs.CYJul 16, 2025

ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluation

Authors: Rose Niousha, Mihran Miroyan, Abigail O'Neill, Joseph E. Gonzalez, Gireeja Ranade, John DeNero, Narges Norouzi

Organizations: University of California, Berkeley

Abstract

Evaluating Artificial Intelligence (AI) tutor feedback before deployment requires anticipating student engagement, typically assessed through real interaction data. We introduce ParaStudent, a fine-tuning framework for simulating novice programming revisions to support AI tutor evaluation. Compared with prompted baselines, ParaStudent's revisions more closely match real student code distributions across functional, stylistic, and semantic metrics. Our best variant achieves AUCs of 0.80 for both feedback relevance and successful uptake when distinguishing streams with real engagement above versus at or below the median, while prompted baselines remain near chance on successful uptake. These findings demonstrate the promise of simulated engagement for pre-deployment feedback triage.

Explore similar work

Simulating Disengaged Students to Evaluate LLM-based Tutors
Sep 14, 2026Xianghui Meng, Jionghao LinTutors Student
StudentSim: Training LLM-based Student Simulators
Sep 1, 2026Ke Yang, Chenglong Wang, Michel Galley +4Tutors User Simulation
The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness
May 7, 2026Rose Niousha, Samantha Boatright Smith, Bita Akram +5Tutors Artificial Intelligence Evaluation

Sep 14, 2026 · cs.LGJ/K move · Enter open · S save

Simulating Disengaged Students to Evaluate LLM-based Tutors

Xianghui Meng, Jionghao Lin

Simulated students generated by computational models provide a practical way to evaluate tutoring strategies and pedagogical approaches used by human and AI tutors. However, such simulations should account for disengaged behaviors, including gaming the system, wheel-spinning, and off-task behavior, because tutors may need different responses for different learner states. We present Disengagement-Aware Student Simulators (DAS2), a reproducible pre-deployment protocol that models five learner-engagement states: engaged, gaming, wheel-spinning, off-task, and mixed, and evaluates AI tutor performance across these states. Using ASSISTments09, two coders independently labeled 100 sampled tutoring sessions based on anonymized interaction-log summaries. They achieved 84% agreement (Cohen's kappa = 0.78), and among agreed cases, human consensus labels matched DAS2 rule-based labels in 81% of cases (kappa = 0.75). Conditioning simulations on intended learner states reduced the correctness-rate gap between simulated and authentic sessions from 0.54 to 0.20 for gaming and from 0.51 to 0.18 for wheel-spinning. Fine-tuned Qwen2.5-7B better matched authentic response-time distributions, while prompt-only GPT-4o generated more distinguishable learner states. Evaluation of five AI tutors from the Claude, Llama, Gemini, Qwen, and GPT families showed that relative rankings remained stable across learner states and interaction lengths, while absolute performance varied, revealing state-specific differences in tutor support. Human validation further showed that automated tutor evaluation does not fully align with human judgment. DAS2 provides a pre-deployment framework for evaluating how AI tutors respond to diverse learner-engagement states before deployment.

Tutors Student User Simulation Low-Latency Systems

Open paper Ask about it PDF

Abstract

Explore similar work

Simulating Disengaged Students to Evaluate LLM-based Tutors

StudentSim: Training LLM-based Student Simulators

The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness