The Assistance Dilemma: Learning to Teach via Multi-Turn Reinforcement Learning

Jakub Macina; Manu Kapur; Mrinmaya Sachan

cs.CLOct 5, 2026

The Assistance Dilemma: Learning to Teach via Multi-Turn Reinforcement Learning

Authors: Jakub Macina, Manu Kapur, Mrinmaya Sachan

Organizations: ETH Zurich

Abstract

Large language models (LLMs) trained to answer questions are natively poor at teaching. Reinforcement Learning (RL) against a simulated student is a promising approach to improve their pedagogy, but existing RL-trained tutors reward the student's success on the tutored problem with the tutor's words still in context. The reward is then easiest to raise by telling the student the answer, and a tuned penalty is needed to reduce telling. Drawing on learning sciences, we introduce a masked near-transfer post-test: the student is tested on an unseen variant of the tutored problem with the tutor's utterances masked, so the reward can rise only through what the student wrote in its own turns. This discourages cognitive offloading by the student and allows the continuous penalty to be replaced by two binary reward gates (factual correctness of tutor response, no solution handover). A leave-one-out ablation shows that the learning-gain reward on its own does not separate teaching from telling: the gates reduce solution handover while the near-transfer post-test improves out-of-domain transfer. Using these reward designs we develop Eduardo, a multi-turn RL recipe for training LLM tutors, and use it to train 4B, 9B, 14B and 27B models from two distinct LLM architectures. Our post-trained Eduardo-27B model matches Gemini-3.1-Pro on MathTutorBench and Claude Opus 4.8 on TutorMoments at 2.4-6.2x fewer thinking tokens than frontier models, which matters for interactive tutoring. Without being named in the reward, the model more than doubles its use of the push-for-justification teacher move while support fading (e.g., assigning independent work), whose payoff lies beyond a single-problem dialog episode, is trained out. We open-source our training environment, an 8,671-problem near-transfer dataset, and trained models for further development.

Figures & tables

Figure 1: Mistake diagnostic versus pedagogy on MathTutorBench . Our models (Eduardo) improve pedagogy over their base at every size, while preserving diagnostic accuracy.

	Training environment ( $n{=}494$ , full dialog)					MathDial ( $n{=}500$ , middle turn)					Eedi ( $n{=}500$ , middle turn)
Model	$\Delta_{\text{transfer}}\uparrow$	$\Delta_{\text{same}}$	Leak $\downarrow$	Ped. $\uparrow$	Fact. $\uparrow$	RM $\uparrow$	Win $\uparrow$	Ped. $\uparrow$	Fact. $\uparrow$	Think $\downarrow$	RM $\uparrow$	Win $\uparrow$	Ped. $\uparrow$	Fact. $\uparrow$	Think $\downarrow$
Base models
Qwen3.5-4B ∗†	+0.10	+0.11	17%	2.26	2.88	2.08	30.2%	2.65	2.90	938	3.56	38.7%	2.91	3.27	1028
Qwen3.5-9B ∗†	+0.22	+0.35	23%	3.29	4.20	3.02	38.0%	2.94	3.17	1172	6.04	56.7%	3.04	3.70	1404
Qwen3-14B †	+0.19	+0.20	98%	1.24	3.45	7.01	70.2%	3.05	3.84	461	7.91	66.3%	2.96	4.34	357
Qwen3.8-27B	+0.22	+0.44	9%	4.20	4.77	8.34	82.4%	3.67	4.12	501	9.99	84.3%	3.27	4.50	258

Table 1: Main results. Training environment performance and zero-shot out-of-domain evaluation on MathDial and Eedi. Ped. and Fact. is the 1–5 Gemini-3.1-Pro whole-dialog score; Think is mean thinking tokens per turn. ∗ / † :

{>}15\%

empty turns / truncated dialogs; environment metrics are unreliable (Table 8 ). ‡ Self-judged.

	Environment ( $n{=}494$ )						MathDial ( $n{=}500$ , middle turn)					Eedi ( $n{=}500$ , middle turn)
Configuration	$\Delta_{\text{transfer}}\uparrow$	$\Delta_{\text{same}}$	Leak $\downarrow$	Ped. $\uparrow$	Fact. $\uparrow$	Think $\downarrow$	RM $\uparrow$	Win $\uparrow$	Ped. $\uparrow$	Fact. $\uparrow$	Think $\downarrow$	RM $\uparrow$	Win $\uparrow$	Ped. $\uparrow$	Fact. $\uparrow$	Think $\downarrow$
Base Qwen3.5-4B (no RL) ∗†	+0.10	+0.11	17%	2.26	2.88	596	2.08	30.2%	2.65	2.90	938	3.56	38.7%	2.91	3.27	1028
Eduardo-4B (full recipe)	+0.25	+0.37	31%	3.66	4.19	544	7.92	78.0%	3.44	3.79	456	8.83	72.4%	3.11	4.15	184
w/o Binary reward gates	+0.23	+0.38	61%	2.80	3.43	398	6.10	65.0%	2.80	3.22	345	8.18	74.0%	3.06	3.94	152
w/o Masking ∗†	+0.24	+0.39	13%	3.18	3.99	718	6.48	65.0%	3.15	4.00	1597	7.48	59.0%	3.13	4.34	1041
w/o Near-Transfer	+0.23	+0.40	47%	2.72	3.30	761	5.25	56.0%	2.99	3.39	776	7.36	65.0%	3.03	4.07	512

Table 2: Leave-one-out ablation (4B). While no ablated condition changes

\Delta_{\text{transfer}}

measurably, the conditions determine how the gain is achieved. ∗ / † :

{>}15\%

empty turns / truncated dialogs; environment metrics are unreliable (Table 8 ).

	Plain prompt ( $n{=}100$ )						Evaluation-aware prompt ( $n{=}100$ )
Tutor	Scaff. $\uparrow$	Rigour $\uparrow$	Avoids OH $\uparrow$	Think/turn $\downarrow$	Ratio $\downarrow$	Tok./turn $\downarrow$	Scaff. $\uparrow$	Rigour $\uparrow$	Avoids OH $\uparrow$	Think/turn $\downarrow$	Ratio $\downarrow$	Tok./turn $\downarrow$
Qwen3.8-27B	0.60	0.31	0.52	$\sim$ 970	6.2 $\times$	$\sim$ 43	0.90	0.60	0.81	$\sim$ 1099	8.2 $\times$	$\sim$ 46
Gemini 3.5 Flash	0.94	0.31	0.75	886	5.7 $\times$	60	0.92	0.83	0.94	886	6.6 $\times$	54
Gemini 3.6 Flash	0.85	0.40	0.71	602	3.9 $\times$	44	0.94	0.79	0.91	843	6.3 $\times$	41
Claude Opus 4.8	0.83	0.48	0.67	$\sim$ 369	2.4 $\times$	$\sim$ 84	0.98	0.92	0.97	$\sim$ 774	5.8 $\times$	$\sim$ 88
Eduardo-27B (ours)	0.96	0.44	0.76	$\sim$ 155	1.0 $\times$	$\sim$ 28	0.98	0.67	0.86	$\sim$ 135	1.0 $\times$	$\sim$ 27

Table 3: Out-of-domain generalization on TutorMoments dialogs (with independent Gemini-3.7-Flash student and Gemini-3.1-Pro judge) . Judge scores (Scaffolding, rigour, Avoids OH) are in

[0,1]

. Think/turn is thinking tokens per turn, Ratio is the overhead relative to Eduardo-27B, and Tok./turn is visible output tokens per turn. (

\sim

) denotes estimates from visible reasoning traces (3.28 chars/token). Under a plain prompt, Eduardo-27B achieves top Scaffolding and Avoids OH scores with 2.4–6.2

\times

fewer thinking tokens than baselines.

Figure 2: Two routes to the prompted move distribution. (a) JS distance of each model’s move distribution from human tutors (

x

) and from the centroid of the evaluation-aware frontier models (

y

, leave-one-out) (b) Per-move shares under the plain prompt, base

\rightarrow

Eduardo , against human tutors (diamonds) and the prompted range.

n{=}100

moments per LM cell.

Panel of Figure 3 — Figure 3: Model size versus MathTutorBench tutoring quality . Represents the mean over the three student-understanding and four pedagogical axes. One recipe, one hyperparameter set: our models improve over their base at every size, and the 27B model approaches top-tier models (0.79 vs. 0.80 for Gemini 3.1 Pro) with fewer thinking tokens.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Optimization
Parameter	Value
Algorithm	DPPO ( Qi et al., 2026 )
Group size $G$	16
Mini-batch size	8 (2 asynchronous off-policy steps)
Optimizer	AdamW ( $\beta_{1}{=}0.9$ , $\beta_{2}{=}0.95$ , $\epsilon{=}10^{-15}$ )
Learning rate	$1\times 10^{-6}$ (constant)

Appendix

Table 4: Training hyperparameters. Identical across all model sizes.

	Train ( $n{=}8{,}238$ )			Test ( $n{=}433$ )
	Source	Transfer	Corr.	Source	Transfer	Corr.
Mean	0.289	0.280	0.648	0.283	0.280	0.675
Median	0.250	0.188		0.250	0.250
Std	0.246	0.246		0.240	0.237
Range	[0, 1]	[0, 1]		[0, 1]	[0, 1]

Appendix

Table 5: Dataset solve-rate statistics . Solve rates measured with

k{=}16

samples from frozen Llama-3.1-8B-Instruct.

Source problem	Near-transfer variant
What is the sum of the prime factors of 85,085? Answer: 53 $p_{\text{pre}}$ : 0.250	What is the sum of the prime factors of 34,034? Answer: 50 $p_{\text{pre}}$ : 0.188
What is the sum of all integer solutions to $\|n\|<\|n-3\|<9$ ? Answer: $-14$ $p_{\text{pre}}$ : 0.312	What is the sum of all integer solutions to $\|m\|<\|m-5\|<12$ ? Answer: $-18$ $p_{\text{pre}}$ : 0.188
Given that $m$ is a root of the equation $x^{2}-2x-7=0$ , find $m^{2}-2m+1$ . Answer: 8 $p_{\text{pre}}$ : 0.375	If $r$ is a root of the equation $x^{2}-5x-4=0$ , determine the value of $r^{2}-5r+10$ . Answer: 14 $p_{\text{pre}}$ : 0.375
A triangle has an even perimeter, and two of its sides are 5 and 2,008. How many such triangles exist? Answer: 4 $p_{\text{pre}}$ : 0.250	Two sides of a triangle have lengths 7 and 2,024. If the perimeter of the triangle is an even integer, how many such triangles are possible? Answer: 6 $p_{\text{pre}}$ : 0.188

Appendix

Table 6: Near-transfer pair examples from the test set. Each transfer variant preserves the solution method while changing numbers or surface context, yielding a different answer.

	$\Delta_{\text{transfer}}$			$\Delta_{\text{same}}$			Leak rate
Model	Mean	95% CI		Mean	95% CI		Mean	95% CI
Base models
Qwen3.5-4B ∗†	+0.10	[0.06,	0.13]	+0.11	[0.08,	0.15]	16.8%	[13.6,	20.2]
Qwen3.5-9B ∗†	+0.22	[0.18,	0.26]	+0.35	[0.31,	0.39]	22.7%	[19.0,	26.3]
Qwen3-14B †	+0.19	[0.15,	0.22]	+0.20	[0.17,	0.24]	97.8%	[96.4,	99.0]
Qwen3.8-27B	+0.22	[0.18,	0.25]	+0.44	[0.40,	0.48]	8.9%	[6.5,	11.5]

Appendix

Table 7: Bootstrap 95% CIs for

\Delta_{\text{transfer}}

\Delta_{\text{same}}

, and leak rate (

n{=}494

problems, 10,000 resamples). ∗ / † as in Table 1 .

Model	Avg turns	Empty%	Think tok/turn	Seq. trunc.%	EoC rate
Qwen3.5-4B (base) ∗†	4.40	70.0%	596	31.2%	68.8%
Qwen3.5-9B (base) ∗†	4.74	19.5%	1504	15.2%	84.8%
Qwen3-14B (base) †	6.35	8.1%	1102	26.1%	59.5%
Qwen3.8-27B (base)	7.05	1.3%	427	3.8%	82.0%
Eduardo-4B	8.18	3.7%	544	10.7%	69.6%
Eduardo-9B	7.87	2.1%	496	6.7%	76.3%

Appendix

Table 8: Truncation diagnostics per model on the 494-problem test set. Avg turns : mean tutor turns per dialog. Empty% : fraction of turns where reasoning exceeded the per-turn cap and produced no visible output. Think tok/turn : mean thinking tokens per tutor turn. Seq. trunc.% : fraction of conversations cut short by the 16,384-token sequence budget. EoC rate : fraction of dialogs ending with the tutor’s end-of-conversation token (vs. hitting turn or token limits). Values

{>}15\%

in bold . Rule used in all tables: ∗ Empty%

{>}15\%

, † Seq. trunc.%

{>}15\%

; for flagged rows

\Delta

and Leak are understated and judge scores are depressed.

Watch for: incorrect claims; confirming wrong calculations; rejecting correct reasoning; hints inconsistent with the correct answer; a wrap-up that leaves a wrong result; undefined terms above the problem’s level; imprecise phrasing that creates false general rules (e.g., “multiplying always makes numbers bigger”); vocabulary mismatched to the student’s demonstrated level.
Score	Description
5	Everything stated or implied is correct and consistent with the correct answer; correct student steps are confirmed and wrong ones flagged; terminology and sentence complexity match the student’s level, precise where it matters and simple where possible, with no misconception-inducing shortcuts.
4	Correct overall, with one slightly imprecise or ambiguous statement, or language occasionally pitched a bit too high or too low, without misleading the student.
3	One clear factual error or wrong/missed judgment of a student step, or language persistently mismatched (jargon-heavy for a struggling novice, or so simplified it distorts the math), though the dialog still heads toward the correct answer.
2	Multiple factual errors, validation of an incorrect solution path, or oversimplified explanations that instill a genuine misconception.
1	Guidance is fundamentally wrong: it contradicts the correct answer or leads the student to a wrong final result.

Appendix

Table 9: Factuality rubric (Fact.): Correctness & Language Appropriateness. Is everything the tutor says mathematically correct, and is it expressed at the right level for this problem and this student? Correctness covers hints, confirmations, corrections and intermediate results; the student’s level is judged from their own messages.

Watch for: stating the final answer or key intermediate results; performing calculations or substituting values the student did not provide; hints before the student attempted or struggled; correcting an error instead of prompting the student to find it; funneling (questions so narrow the answer is trivial); never asking for justification; talk-time balance (the student should talk at least as much as the tutor).
Score	Description
5	Excellent: support is contingent (the smallest hint that unblocks, escalated only when needed) and fades over the dialog; the student struggles and self-corrects; errors are probed, not corrected outright; the tutor presses for justification; no final answer or key intermediate result is revealed before the student derives it.
4	Good: mostly contingent guidance with small lapses, such as an occasional premature hint, mild funneling, a minor result confirmed slightly early, or missed chances to ask “why”.
3	Mixed: some genuine guidance, but the tutor rescues too fast after errors, reveals an important intermediate step or does a key calculation, funnels the student through trivial steps, or dominates the dialog.
2	Poor: the tutor reveals the final answer or walks through most of the solution after brief guiding, corrects errors directly, or lectures with little use of the student’s responses.
1	No pedagogy: the final answer or complete solution is given immediately (in any format), or the student’s reasoning is ignored entirely.

Appendix

Table 10: Pedagogy rubric (Ped.). Good tutoring uses contingent scaffolding (help calibrated to the student’s demonstrated need, minimal hint first, fading as competence grows), allows productive struggle (room to attempt, err and self-correct; errors diagnosed and turned into reasoning opportunities), and maintains rigour (pressing for justification and avoiding funneling). The student does the actual reasoning and most of the talking.

Explore similar work

PEARL: Training Socratic Tutors with Pedagogically Aligned Reinforcement Learning
May 28, 2026Qikai Chang, Zhenrong Zhang, Linbo Chen +4Socratic Dialogue Offline Reinforcement Learning
University of Science and Technology of China · iFLYTEK Research
LLMs Are Already Good Tutors: Training-Free Prompt Optimization for Pedagogical Math Tutoring
May 26, 2026Unggi Lee, Minchul Shin, Yeil Jeong +5Tutors Training-Free
Korea University Sejong Campus · 1Korea University Sejong Campus · Gyeonggi Institute of Education +9
Beyond Helpfulness: A Teaching-over-Solving Diagnostic for Measuring Educational Impact in LLM Tutors
Jun 15, 2026Junyi Yao, Zihao Zheng, Baichuan LiTutors Computerized Adaptive Testing
Washington University in St. Louis, St. Louis, MO, USA · Department of Operations Research and Engineering Management, Southern Methodist University, Dallas, TX, USA

May 28, 2026, cs.LGJ/K move · Enter open · S save

PEARL: Training Socratic Tutors with Pedagogically Aligned Reinforcement Learning

Qikai Chang, Zhenrong Zhang, Linbo Chen, Pengfei Hu+3

University of Science and Technology of China · iFLYTEK Research

Socratic Dialogue Offline Reinforcement Learning Reward Functions

Open paper AskPDF arXiv

Large Language Models (LLMs) show strong potential as educational tutors. Existing approaches typically train them to solve problems and provide correct answers, but this problem-solving-centered paradigm overlooks key requirements of effective tutoring: progressive guidance and the coordination of multiple pedagogical objectives across multi-turn interactions. Developing such tutors remains challenging because student behavior varies substantially with individual knowledge states, pedagogical effectiveness depends on multiple factors beyond final-answer correctness, and coordinating these objectives over tutor-student interactions is inherently difficult. To address these challenges, we propose PEARL, a PEdagogically Aligned Reinforcement Learning framework for training Socratic tutoring agents. First, we introduce a controllable student simulator that disentangles latent cognitive states from response generation, enabling simulation of diverse abilities and misconceptions. Second, we develop a pedagogically aligned reward model that jointly assesses pedagogical quality and objective correctness. Finally, we propose a stable multi-objective reinforcement learning approach that balances competing pedagogical objectives during tutor training. Experiments across multiple benchmarks show that PEARL performs competitively against tutoring-specific open-source systems and leading proprietary LLMs.

Abstract

Explore similar work

PEARL: Training Socratic Tutors with Pedagogically Aligned Reinforcement Learning

LLMs Are Already Good Tutors: Training-Free Prompt Optimization for Pedagogical Math Tutoring

Beyond Helpfulness: A Teaching-over-Solving Diagnostic for Measuring Educational Impact in LLM Tutors