cs.CLOct 11, 2026
SaveDo LLM Tutors Actually Teach? A Process-Centered Synthesis of the Tutoring Loop
Organizations: Indian Institute of Technology Kharagpur · National University of Singapore · TCG CREST · Adobe Research ·
Abstract
Large language models can produce fluent tu- toring dialogue, but whether they actually help students learn remains unclear. We present a process-centered synthesis of LLM tutoring that follows the path from learner-state infer- ence to pedagogical action, tutor training, eval- uation, and student simulation. This view ex- poses a central gap: implementing a tutoring component does not establish that it works or that its outputs meaningfully affect later stages. A structured study-level audit shows that peda- gogical evaluation is common, while evidence becomes much thinner for learner-conditioned intervention, human learning gains, and espe- cially transfer or delayed retention. Human studies nevertheless provide a conditional posi- tive result: LLM tutors can improve learning, but benefits depend on pedagogical structure, learner engagement, and how learning is mea- sured. We argue that progress requires validat- ing the interfaces between tutoring stages, not only improving individual components.
Figures & tables
Figure 1: The tutoring loop: our loop-based taxonomy of LLM-based adaptive tutoring.
| Tutoring-loop component | Studies | Coverage |
|---|---|---|
| Learner-state modeling (L1) | 20/88 | 22.7% |
| Learner-conditioned action (L1 L2) | 14/88 | 15.9% |
| Pedagogical policy (L2) | 62/88 | 70.5% |
| Policy optimization (L3a) | 15/88 | 17.0% |
| Student simulation | 14/88 | 15.9% |
| Pedagogical evaluation (L3b) | 77/88 | 87.5% |
Table 1: Coverage of tutoring-loop components in the coded corpus ( ). Explicit pedagogical policy and pedagogical evaluation are common, but only 9 studies jointly instantiate learner-state modeling, learner-conditioned action, an explicit policy, and a downstream outcome.
Figure 2: Implementation and learner-evidence coverage in the coded corpus ( ). Percentages describe the structured corpus rather than population-level prevalence.
| Level | Strongest evidence established | Highest | level |
|---|---|---|---|
| E1 | Component / response evidence | 12 | 88 (100.0%) |
| E2 | Pedagogical-quality evidence | 49 | 76 (86.4%) |
| E3 | Downstream task/interaction evidence | 17 | 27 (30.7%) |
| E4 | Human learning-gain evidence | 9 | 10 (11.4%) |
| E5 | Transfer or delayed retention | 1 | 1 (1.1%) |
Table 2: Evidence ladder over the coded corpus ( ). “Highest” assigns each paper to the strongest claim it establishes; the last column is cumulative.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: The LLM Tutoring Loop and its research landscape. Panel (a) models tutoring as a closed process in which learner evidence updates a learner-state model, the learner state conditions a pedagogical policy, and observed outcomes feed back into subsequent decisions. Panel (b) organizes the literature into learner-state modeling, tutor policy and pedagogy, training and optimization, and evaluation. Student simulation provides shared infrastructure for training and evaluation, while agentic architectures, grounding, multimodality, and safety cut across stages. Citations are representative rather than exhaustive.
Figure 4: How far does current LLM-tutor evaluation progress toward evidence of learning? Bars report cumulative coverage over the coded corpus. E1 includes component- or response-level evidence; E2 pedagogical-quality evidence; E3 downstream student/task or interaction behavior; E4 direct human learning-gain evidence; and E5 transfer or delayed retention. The ladder ranks the strength of the learner-related claim supported, not overall methodological quality.
| Approach | Representative work | Primary constraint | What it attempts to preserve | Main remaining risk |
|---|---|---|---|---|
| Role-based simulation | Prompted / MathDial-style learners | Role, task, or proficiency prompt | Surface student-like dialogue and task interaction | Model priors dominate learner knowledge and error behavior |
| Profile-conditioned simulation | Liu et al. (2024c) | Explicit personality or learner profile | Variation across simulated learner types | Plausible profile faithful task behavior |
| History-conditioned simulation | Duan et al. (2026) | Observed responses and dialogue history | Cross-turn consistency with a particular learner | History consistency does not imply realistic learning dynamics |
| Psychometrically grounded simulation | Scarlatos et al. (2025b) | Ability and item-difficulty parameters | Calibrated competence–difficulty relationship | Aggregate calibration may hide unrealistic error processes |
| Ability-aware / cognitive-diagnosis grounded | Que et al. (2026) | Estimated learner ability used to select or configure the simulator | Ability-dependent response behavior | Ability matching does not guarantee dialogue or learning fidelity |
| Data- and cognition-grounded simulation | Qi et al. (2026) | Observed behavioral data and cognitively motivated learner dynamics | Error patterns and evolution of learning behavior | Fidelity may remain specific to the data and learning setting |
Table 8: From unconstrained role-play to grounded student simulation. The upper rows describe increasingly constrained simulator designs; the final row is not a separate architecture but the validation step needed to establish whether those constraints yield behavior that is sufficiently similar to real learners for the intended training or evaluation use. More conditioning should therefore not be interpreted as stronger validity without external evidence.
| Validity hazard | Failure | Why it biases tutor evaluation | What should be checked |
|---|---|---|---|
| Fake confusion | The simulator expresses misunderstanding that is not grounded in an actual knowledge deficit, or resolves it unrealistically quickly. | Rewards tutors for handling artificial errors or makes interventions appear more effective than they would be with real learners. | Error-type distribution, misconception persistence, and recovery trajectories against real learners. |
| Competence / answer leakage | The simulator has access to information, reasoning, or target answers that the modeled learner should not possess. | Inflates solve rate and can make weak tutoring policies appear successful. | Information isolation and consistency between declared or calibrated ability and observed responses. |
| Profile leakage | Profile metadata directly determines responses instead of influencing behavior through realistic learner characteristics. | Creates artificially strong personalization signals and overstates the benefit of profile-conditioned tutoring. | Counterfactual profile tests and separation between profile metadata and answer information. |
| Simulator–judge circularity | The same or closely related model family acts as student, constructs the reward, and/or judges the resulting dialogue. | Optimizes the tutor toward shared model-family biases rather than toward real learner behavior. | Cross-model evaluation plus independent human or real-student validation. |
| Behavioral / learning-dynamics mismatch | The simulator produces plausible turns but differs from real students in dialogue acts, error processes, help-seeking, persistence, or response to instruction. | A policy can overfit to interaction patterns or learning rates that do not occur with real learners. | Distributional and trajectory-level comparison with real tutoring data ( Martynova et al., 2025 ; Scarlatos et al., 2026 ) . |
Table 9: Validity hazards in simulated-student training and evaluation. Simulator failures are especially consequential because the same simulated interaction may affect both the tutoring policy being optimized and the benchmark used to evaluate it. Surface plausibility alone cannot rule out these hazards.
| Resource | Type | Domain / source | Provides | Loop relevance |
|---|---|---|---|---|
| MathDial ( Macina et al., 2023 ) | Dataset | Mathematics | Tutor–student dialogues grounded in math reasoning problems with pedagogical tutor-move annotations | L2 / L3a |
| MathTutorBench ( Macina et al., 2025 ) | Benchmark | Mathematics | Open-ended tutoring tasks and learned scoring for pedagogical capability | L3b |
| MRBench ( Maurya et al., 2025 ) | Benchmark | Mathematics | Mistake-remediation responses annotated using a multi-dimensional pedagogical taxonomy | L3b |
| BEA 2025 ( Kochmar et al., 2025 ) | Benchmark | Mathematics | Tutor responses labeled on multiple pedagogical dimensions for automatic assessment | L3b |
| BIPED ( Kwon et al., 2024 ) | Dataset | ESL | Human one-to-one tutoring dialogue with fine-grained tutor and student dialogue acts | L2 |
| ConvoLearn ( Sharma et al., 2026 ) | Dataset | Earth Science | Extended constructivist tutor–student dialogues organized around pedagogical dimensions | L2 / L3a |
Table 10: Representative datasets and benchmarks for LLM tutoring. Resources differ not only in subject domain but also in which part of the tutoring loop they expose. Mathematics remains the dominant benchmark setting, while newer resources add language learning, science, longitudinal learner modeling, and long-horizon agent evaluation. The table is representative rather than exhaustive.
| Component | Implementation establishes | Stronger validation requires |
|---|---|---|
| Learner state (L1) | An explicit estimate of knowledge, misconception, profile, goal, or affect. | Agreement with expert labels, observed responses, learner histories, or other external learner evidence. |
| State action (L1 L2) | The learner state is supplied to or used by the tutoring policy. | Ablation, controlled counterfactual, or real-user evidence showing that conditioning changes the intervention and contributes to performance. |
| Tutor policy (L2) | An explicit pedagogical act, strategy, plan, or help-control mechanism. | Evidence that the selected action is appropriate under the relevant learner condition and, for stronger claims, improves downstream learner behavior. |
| Student simulation | A simulated learner is used for training, rollout, or evaluation. | Behavioral, state, and ideally interventional fidelity against grounded or real-student data for the intended downstream use. |
| Pedagogical evaluator / reward | A rubric, learned scorer, verifier, or LLM judge scores tutoring behavior. | Calibration against independent human judgments and robustness across model families, prompts, or domains. |
| Learner outcome | A downstream task, interaction, or learning measure is reported. | Real learners for claims about human learning; delayed or novel-task assessment for retention/transfer claims. |
Table 11: Implementation is not validation. The evidence needed depends on the claim being made. Richer architectures or more detailed conditioning do not by themselves establish that a learner state, simulator, evaluator, or outcome is valid.
More in this topic
Intelligent Tutoring Systems
Intelligent tutoring systems model learner knowledge and adapt instruction, practice, hints, and feedback to support learning.
See all 149 papers