Effective tutoring requires determining not only what information to provide, but also which form of assistance to deliver. A tutor may address the same student error using various pedagogical strategies, adapting its approach as the interaction progresses. We introduce Comparative Pedagogical Supervision (CPS), a training framework to learn to select the optimal strategy and feedback for student learning. We evaluate CPS on math and science reasoning datasets. After five tutoring turns, CPS outperforms the strongest fixed-strategy baselines. The learned tutor transfers across datasets, and its strategy selection can also inform a separate API-based tutor.
Figures & tables
Figure 1: Pedagogical strategy varies according to the tutoring context: Depending on the student’s response, further support may be provided through a Worked example or a Hint for subsequent steps.
Figure 2: Overview of CPS: During training, we sample one feedback for each pedagogical strategy from the tutor. Independent student attempts measure how often each strategy leads to a correct answer. A strategy and its feedback provide supervision when its score is positive and strictly higher than others. A separate exploration policy selects the strategy used to continue the multi-turn interaction. At inference, CPS generates a ’ [strategy] feedback ’ in one call. strategy tag helps with interpretability.
Tutor
GSM8K
SVAMP
ScienceQA
Initial accuracy ( A0 )
4.3
8.5
14.4
Hint
8.4
16.3
41.7
Step scaffold
32.6
32.8
51.8
Explicit explanation
48.9
31.3
50.0
Worked example
66.4
79.9
48.7
Self-explanation
7.3
16.5
40.6
Table 1: Final cumulative accuracy A5 (%) unless otherwise indicated. The final two rows give CPS’s gain over initial accuracy in percentage points and its correction rate (%). Bold marks the highest final accuracy and underlining identifies the strongest fixed strategy. Differences are computed before rounding.
Dataset
Knowledge gap
Misconception
Mixed
GSM8K
88.3→89.3
40.5→52.0
63.7→71.3
SVAMP
90.9→90.2
70.4→76.5
71.1→74.6
ScienceQA
45.6→53.7
47.3→67.7
45.9→54.2
Table 2: Correction rates (%) by student type: Each cell shows the strongest fixed-strategy tutor followed by CPS, with the higher value in bold. Worked example is strongest across types on GSM8K and SVAMP. Contrastive feedback is strongest across types on ScienceQA. Differences are computed before rounding.
Figure 3: Correction rates for the student-type oracle and CPS. The oracle uses test outcomes to choose a fixed strategy for each student type and matches the globally strongest fixed strategy within each dataset.
Dataset
Switch Rate (%)
Sessions (%)
Cond. MI (bits)
GSM8K
62.3
79.2
0.17
SVAMP
54.3
67.3
0.08
ScienceQA
56.0
84.9
0.19
Table 3: Strategy changes during tutoring. Switch rate uses adjacent tutor-turn pairs. Sessions with a switch uses sessions that reach at least two tutor turns. Conditional mutual information measures the association between consecutive strategies after accounting for student type and turn.
Context
GSM8K
SVAMP
ScienceQA
Full history
72.7
80.2
63.9
Latest exchange
72.8
78.1
65.4
Latest student reply
72.4
79.2
65.2
Problem only
50.6
72.3
66.7
Table 4: Final accuracy A5 (%) with different amounts of dialogue context. All variants receive the problem and reference solution and are trained separately under the same checkpoint-selection rule. Bold marks the highest accuracy per dataset. Differences are computed before rounding.
Dataset
Tutor
Fixed (%)
Own (%)
GSM8K
Prompt Selector
2.9
4.7
CPS
37.0
39.1
SVAMP
Prompt Selector
9.2
13.8
CPS
43.0
46.6
ScienceQA
Prompt Selector
21.9
27.7
CPS
35.6
43.5
Table 5: First-turn correction rates among initially incorrect cases. Fixed response uses feedback from the fixed-strategy tutor corresponding to the selected strategy. Own response uses each method’s generated feedback. Differences are computed before rounding.
Figure 4: Qualitative tutoring examples: (a) CPS changes from Contrastive feedback to Hint after the initial intervention does not resolve the student’s error, while the Prompt Selector continues eliciting explanation of the same incorrect step. (b) Both tutors use Step scaffold , but CPS targets the student’s remaining confusion between accumulated amount and earned interest, while the fixed-strategy tutor continues scaffolding an earlier calculation. Strategy labels are retained for analysis but are not shown to the student.
Source dataset
GSM8K
SVAMP
ScienceQA
GSM8K
72.7
78.7
49.1
SVAMP
75.5
82.5
51.1
ScienceQA
33.5
31.1
64.1
Strongest fixed strategy
66.4
79.9
54.0
Table 6: Cross-dataset final accuracy A5 (%). Rows identify the tutor’s training dataset. Columns identify the evaluation dataset. Underlining denotes in-domain evaluation, and bold marks the highest trained-tutor accuracy per target. The target’s strongest fixed strategy is shown for comparison.
Tutor
A1
A5
Correction (%)
Step scaffold
64.0
82.6
58.3
Contrastive feedback
72.9
82.6
58.3
Prompt Selector
68.0
87.9
70.9
CPS selection + API
73.3
89.9
75.7
Table 7: Strategy selection with GPT-4.1-mini as the response generator (%). This earlier GSM8K study uses 247 assignments with A0=58.3% , rather than the current main evaluation. The two strongest fixed strategies are shown. Differences are computed before rounding.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Problem: Mia has 6 marbles. Leo has twice as many. Each person receives 3 more marbles. How many marbles do they have altogether?
Initial student response: "Leo has 6+2=8 marbles because ’twice as many’ means two more.”
Strategy
Definition
Illustrative tutor response
Minimal hint
Provide a brief cue directing attention to the relevant concept or relationship, allowing the student to independently determine the subsequent step.
"Focus on the word ’twice’ when comparing Leo’s marbles with Mia’s."
Step scaffold
Guide the next step by breaking it into smaller actions or providing a partial setup.
"First find Leo’s starting number of marbles. This is two groups of 6. What is 2×6 ?"
Explicit explanation
Explain the relevant concept or rule and the reasoning behind it.
"’Twice as many’ means two equal groups of the original amount. Multiplication by two expresses this relationship."
Worked example
Demonstrate the relevant reasoning on an analogous problem with different values. Then, ask the student to apply the same method.
"Suppose Ana has 4 marbles and Ben has twice as many. Ben has 2×4=8 . If each receives 1 more, their total is (4+1)+(8+1)=14 . Apply the same steps to Mia and Leo."
Appendix
Table 8: Definitions and examples of the six pedagogical strategies. Each row illustrates an example response of the respective strategy to the same tutoring context. All strategies withhold the final answer to the original problem.
Figure 5: Student simulation : The initial attempt uses one call for private reasoning and another for the public reply. After tutor feedback, one call updates the KC state, and another generates the reply and answer under that state. Only the public reply enters the tutor’s dialogue. The example illustrates the procedure rather than reporting an experimental result.
Figure 6: Dataset-specific KC construction: We group similar reasoning steps from training solutions, identify reusable KCs, and merge overlapping definitions. The resulting library is used to annotate problems. Incorrect beliefs are checked on problems requiring the corresponding KC before they are included in student profiles. The figure uses an illustrative example.
GSM8K
SVAMP
ScienceQA
KCs in annotation library
34
34
TBC
KCs eligible for profile construction
28
28
TBC
Affected KCs per profile
4
4
TBC
Knowledge-gap profiles
5
5
5
Misconception profiles
5
5
5
Mixed profiles
5
5
5
Appendix
Table 9: Current test populations. An assignment pairs a student profile with an applicable problem. The same problem can be evaluated under more than one profile. Mastered controls are used for screening rather than the main tutoring comparison.
Setting
GSM8K
SVAMP
ScienceQA
Training assignments
2,934
3,000
8,624
Validation assignments
485
375
1,606
Training SFT examples
2,760
2,998
6,112
Validation SFT examples
469
378
1,111
Training epochs
3
4
4
Maximum sequence length
4,096
8,192
4,096
Appendix
Table 10: Training records for the current GSM8K, SVAMP, and ScienceQA tutors. Sequence lengths are in tokens. Assignment counts and SFT-example counts measure different units.
Generation stage
Temperature
Initial student attempt
0.0
Tutor response
0.3
Student state update
0.0
Subsequent student response
0.3
Candidate-scoring student attempt
0.3
Appendix
Table 11: Decoding temperatures by generation stage. Student state updates and public response generation are separate calls.
Setting
Value
LoRA rank
16
LoRA scaling factor
32
LoRA dropout
0.05
Learning rate
2×10−4
Learning-rate schedule
Cosine decay
Warmup
5%
Appendix
Table 12: CPS fine-tuning configuration.
Figure 7: Pedagogical strategy choices by tutoring turn on GSM8K. Percentages use valid strategy tags from interactions still active at each turn.
Figure 8: Pedagogical strategy distributions by tutoring turn on SVAMP. Percentages are computed over valid strategy tags from interactions still active at each turn.
Figure 9: Pedagogical strategy distributions by tutoring turn on ScienceQA. Percentages are computed over valid strategy tags from interactions still active at each turn.
Configuration
SFT examples
Validation A5
Outcome construction
Direct completion, K=10
773
88.55
One student response followed by ten probes
586
70.37
Ten student responses with paired probes
667
71.72
Trajectory continuation
ε=0.0
773
88.55
Appendix
Table 13: Sequential training ablations on GSM8K validation interactions. All configurations are evaluated on the same initial student responses. Results use checkpoints selected by minimum validation loss. Bold indicates the selected configuration within each comparison group.
Student reply
N
All KCs
Reached KCs
Full response
Answer agree
Before tutoring
270
78.1
63.7
58.1
100.0
After tutor feedback
180
81.8
76.1
61.1
97.8
Appendix
Table 14: Luna’s evaluation of student replies on GSM8K (%). All KCs and Reached KCs summarize KC-level judgments. Entire response and Answer agreement summarize response-level judgments. N is the number of replies, not the number of KCs.
Criterion
Luna (%)
Requested strategy appears
96.1
Requested strategy is the main approach
92.8
Mathematically correct feedback
95.0
Correct final answer disclosed
0.0
Appendix
Table 15: Luna’s evaluation of 180 fixed-strategy tutor responses. The strategy-presence and main-approach criteria distinguish including a strategy from using it as the primary assistance.
Judgment
Luna
Human majority
Exact agreement
Student replies: 50 KCs with a resolved human majority
Consistent with assigned knowledge
24/50 (48.0%)
21/50 (42.0%)
42/50 (84.0%)
Violates assigned knowledge
7/50 (14.0%)
10/50 (20.0%)
Relevant step not reached
19/50 (38.0%)
19/50 (38.0%)
Knowledge-state changes: 12 changed events
Feedback supports the change
11/12 (91.7%)
10/12 (83.3%)
11/12 (91.7%)
Appendix
Table 16: Luna and human judgments on the same items. Counts give the number assigned each label, with percentages in parentheses. Exact agreement compares the complete categorical judgments. For student replies, it compares all three labels across the 50 KCs with a human majority. The nine unresolved KCs are excluded from this comparison but retained in the rater-agreement analysis.
Criterion
Human pairwise agreement (%)
Luna–human κ
Student knowledge-state labels
41.8
0.74
Feedback supports the change
77.8
0.62
Feedback addresses the changed KC
94.4
1.0
Requested strategy is the main approach
93.3
0.83
Mathematically correct tutor feedback
86.7
0.64
Appendix
Table 17: Agreement statistics for the human studies. Unweighted Cohen’s κ compares Luna with human majorities. For student knowledge-state labels, human agreement uses all 59 KCs and the Luna comparison uses the 50 majority-resolved KCs. A dash indicates an undefined coefficient.
Feedback
Target KC changed
Relevant
40/40
Retry only
0/40
Irrelevant
0/40
Misaligned
0/40
Answer reveal
0/40
Appendix
Table 18: Recorded KC-state changes under different feedback on GSM8K. Each condition starts from the same initial response and state. Relevant feedback changes only the targeted KC. None of the four control conditions changes the student’s state.
Student state
Accuracy (%)
Before feedback
5.0
After retry only
5.0
After irrelevant feedback
5.0
After misaligned feedback
5.0
After relevant feedback
67.7
After answer reveal
5.0
Appendix
Table 19: Accuracy on 300 new GSM8K problems using the student’s state before or after feedback on a different problem. Control conditions preserve the original state and share the same evaluated answers.
On-policy self-distillation (OPSD) trains a question-only student with token-level feedback from a teacher given training-only privileged information (PI). OPSD therefore provides dense, on-policy supervision, and is free of a larger external teacher, but its effectiveness rests on how PI is designed and utilized. Our preliminary diagnostics suggest a significant gap between teacher utility and student learnability, where a small fraction of high-disagreement tokens dominate the distillation signal, and short teacher continuations at these positions further expose more explicit PI leakage than transferable correction cues, indicating a strong intent on injecting PI-conditioned shortcuts. We propose Adaptive On-Policy Self-Distillation (AOPSD), which adapts what information the teacher receives and how strongly its feedback influences learning. AOPSD encodes each solution as a reasoning DAG, orders problems by the student's evolving capability, and reveals only the affordable subgraph and its next frontier as PI. For high-disagreement tokens, AOPSD utilizes short teacher continuations as probes to encourage useful guidance while mitigating PI-conditioned shortcuts among teacher supervisions. On HMMT25, AIME24, AIME25, and BRUMo25, AOPSD achieves 72.5% Pass@8, which is 6.7 percentage points above OPSD and 4.2 above the strongest competing baseline while reducing 15 percentage points of training time at lower cost.
Jiacheng Du, Weiwei Xie, Tianyi Du +3
National University of Singapore · Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory
LLMs can personalize education, although current static-prompt tutoring systems struggle to adapt to diverse academic disciplines. We develop and test a system with subject-aware prompting, based on 14 pedagogical features (e.g., tutor scaffolding, student understanding) extracted from raw transcripts. We first train a prompt routing model in a simulation environment, and then deploy it for online adaptation with actual high-school students. The simulation benchmark shows the router outperforming two static baselines (0.694 vs. 0.647 and 0.64, p<0.001). A/B testing (N=656 conversations from 359 students) shows sim-to-real transfer where the model switches from analytical to scaffolding learning strategies. Our adaptive prompt selection mechanism improves instructional efficiency, maintains pedagogical quality and reduces interactions by around 3 turns (p=0.007). While a greedy router achieves a comparable exercise conversion rate with the baseline (19.1% vs. 19.6%), a stochastic router that samples strategies leads to a higher conversion rate (28.1%).
Good tutoring adapts to the individual: it tracks what a learner knows, notices why they go wrong, and asks the next question that will help most. Most deployed tutoring tools instead serve fixed item banks and treat a wrong answer as a single bit of signal. We present CoLearn, an interactive, agentic tutor that supports an iterative tutoring loop: the learner practises, and the system builds an evidence-grounded memory of the learner's mastery and misconceptions. This memory is updated as evidence accumulates and is used to generate the next personalised question. CoLearn has three components: (i) a persistent learner-state memory that updates per-topic mastery with a soft-evidence variant of Bayesian Knowledge Tracing, where a large language model acts as a continuous observation function; (ii) adaptive question generation that targets the learner's weakest topic and recurring misconceptions; and (iii) an evidence view that makes personalisation visible and testable through live progress visualisation and blind A/B comparison. In blind A/B evaluation, questions conditioned on this memory are preferred over non-personalised ones 68-69% of the time, and in persona simulations with hidden ground-truth mastery the agent's belief converges toward the learner's true mastery.