DICE: Decoupling Capability from Intervention Necessity in LLM Tutoring

Sayantan Pal; Kaiyi Ji; Rohini K. Srihari

cs.AIOct 4, 2026

DICE: Decoupling Capability from Intervention Necessity in LLM Tutoring

Authors: Sayantan Pal, Kaiyi Ji, Rohini K. Srihari

Organizations: Department of Computer Science and Engineering State University of New York at Buffalo

Abstract

Fluent guidance is not the same as useful intervention. LLM tutors are typically trained to generate the next teacher utterance, implicitly assuming that every student turn warrants a response. However, our experiments indicate that this conflates tutoring capability (what to say) with intervention necessity (whether to say it). We introduce DICE, a framework that decouples intervention decisions from response generation by first selecting an explicit pedagogical action. To calibrate this action selection policy, we define Intervention Value (IV), a rollout-grounded counterfactual metric that compares each action against non-intervention. IV shows that many prescribed interventions provide little or no marginal benefit. We further introduce DICE-Bench, a multi-variant math tutoring benchmark with skill-preserving problem variants for session-level evaluation. Using IV-weighted and KL-regularized policy optimization, DICE learns to intervene selectively while preserving tutoring effectiveness. In simulated tutoring sessions, DICE reduces the over-intervention rate to near zero while guiding students to correct solutions in approximately 3-4 fewer turns on average than existing Socratic tutoring baselines.

Figures & tables

Figure 1: Two failure modes of existing tutors ( SocraticLM used for this case) and how DICE addresses them through explicit action selection. Scenario A: The student solves the problem correctly, but SocraticLM continues guiding for three unnecessary turns before confirming. Scenario B: the student produces an arithmetic error, but SocraticLM accepts the wrong answer; DICE redirects the student to the faulty step. Full dialogue rollouts are in Appendix G.1 .

Turn	Dialogue rollout
Problem: A 400-page manuscript is copied and bound 10 times. Printing costs $\$ 0.05 $per page, binding costs$ $5 $per manuscript, and the final bill has an additional$ 8%$ sales tax. What is the total cost including tax?
S	$270?$
T	Over-intervention It seems you calculated the cost before tax. Try including the $8\%$ sales tax.
S	$250$
T	You may have missed part of the total. First compute printing and binding, then apply tax.
S	Printing is $200$ , binding is $50$ .

Table 1: Illustrative over-intervention rollout from an existing tutor (T), SocraticLM . For readability, we retain only the essential exchanges from the full dialogue between the tutor and a human student (S). Even after the student reaches the correct answer, the tutor continues intervening instead of accepting and advancing.

Figure 2: A qualitative example of DICE-Bench difficulty augmentation. Easy (

q^{-}

), Source (

q

), and Hard (

q^{+}

) variants preserve the same underlying skill while increasing reasoning and operational complexity.

Metric	Easy	Source	Hard	E $\leq$ S	S $\leq$ H
Steps	2.23	3.09	3.63	95.9%	86.8%
Operators	1.96	3.27	4.49	88.9%	79.9%
Equations	2.33	2.93	4.80	87.5%	92.6%
Numeric tokens	7.00	10.71	14.19	94.3%	83.2%
Question tokens	29.38	47.18	47.76	97.4%	60.8%

Table 2: Structural validation of generated difficulty variants in DICE-Bench . Means are computed over 10,273 matched triplets

(q^{-},q,q^{+})

; pairwise

q^{-}\!\leq q

and

q\!\leq q^{+}

differences are significant under paired Wilcoxon signed-rank tests.

Action distribution, all variants, $N=20{,}477$
	$\mathcal{D}_{\mathrm{src}}$	$\mathcal{D}_{\mathrm{easy}}$	$\mathcal{D}_{\mathrm{hard}}$
Provenance	dialogue	step	step
Decision points	7,625	4,728	8,124
$a_{0}$ ( accept )	8,037 (39.2%)
$a_{1}$ ( arith. repair )	4,019 (19.6%)
$a_{3}$ ( handle_q )	3,888 (19.0%)

Table 3: Composition of DICE-Bench . Splits are by source problem ID; action distribution is reported over all variants.

Figure 3: Normalized difficulty trajectory across variants. Values are normalized per metric to highlight relative progression across the triplet.

Figure 4: Policy-model comparison. Left: action macro-F1 . Right: over-intervention rate .

Method	All3 $\uparrow$	Vars. $\uparrow$	Turn $\downarrow$	OInt $\downarrow$	UAcc $\downarrow$
Full	0.564	2.324	11.940	0.028	0.000
w/o Cost	0.532	2.284	13.048	0.036	0.000
w/o Guard	0.552	2.300	11.720	0.026	0.032
w/o Verifier	0.520	2.240	10.850	0.020	0.291
w/o All	0.216	1.496	18.048	0.880	0.076

Table 5: Session-level ablation of DICE . The full system uses cost-sensitive action selection, an acceptance verifier, and a final-answer guard.

Figure 6: Session-level tutoring dynamics on DICE-Bench . Left: Progress curves showing the fraction of sessions where all three variants are solved and the average number of variants solved as the shared turn budget is used. Right: Trade-off plots comparing all-3 success against over-intervention and average variants solved against average turns. DICE-Cost achieves the strongest session-level trade-off, solving more variants in fewer turns while keeping over-intervention near zero.

Mode	Method	All-3 $\uparrow$	$q^{-}$ $\uparrow$	$q$ $\uparrow$	$q^{+}$ $\uparrow$	Vars. $\uparrow$	Turns $\downarrow$	OverInt. $\downarrow$	UnsafeAcc. $\downarrow$
Soc. Tut.	StratL	0.440	0.868	0.700	0.440	2.008	15.112	0.768	0.272
	Ft. MathDial	0.080	0.516	0.284	0.080	0.880	19.164	0.904	0.008
	Soc. Prompting	0.216	0.772	0.508	0.216	1.496	18.048	0.880	0.076
	SocraticLM	0.376	0.844	0.644	0.376	1.864	14.884	0.420	0.400
DICE	MAP	0.532	0.928	0.824	0.532	2.284	13.048	0.036	0.000
	Cost $(\lambda=0.25)$	0.564	0.928	0.832	0.564	2.324	11.940	0.028	0.000

Table 6: Session-level tutoring results on DICE-Bench . Each session contains the triplet

(q^{-},q,q^{+})

under a shared 20-turn budget. All-3 reports full triplet success, Vars. is the average number of variants solved, and UnsafeAcc. is unsafe final-answer acceptance. Higher is better for success metrics and Vars.; lower is better for Turns, OverInt., and UnsafeAcc. Best values are bolded and second-best values are underlined.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Overview of DICE-Bench construction and the DICE tutoring framework. Panel (a) shows how SocraTeach math problems and multi-turn Socratic dialogues are transformed into skill-preserving problem triplets

(q^{-},q,q^{+})

and turn-level action labels. Panel (b) shows the DICE pipeline: a tutoring state is mapped to an explicit pedagogical action policy trained with IV-weighted KL regularization, and the selected action conditions response generation during tutoring.

Panel of Figure 9 — Figure 9: Prompt template used for skill-preserving easy/hard problem augmentation in DICE-Bench .

Panel of Figure 10 — Figure 10: Prompt template used for turn-level tutor-policy annotation in DICE-Bench . Dialogue-derived source states include dialogue history and the observed teacher–student exchange; step-variant states use the same schema but omit dialogue context and are constructed directly from the current subgoal.

Action	Problem and dialogue context	Diagnosis	Rationale
Accept and advance	Problem: John plants a plot of $3$ trees by $4$ trees. Each tree gives $5$ apples, and each apple sells for $\$ 0.5 $. How much money does John make? T: How many trees does John plant? S: John plants$ 3\times 4=12$ trees.	– Correct – On track – No issue	The student answers the current subgoal correctly and shows the needed reasoning, so no intervention is necessary.
Ask for justification	Problem: John plants a plot of $3$ trees by $4$ trees. Each tree gives $5$ apples, and each apple sells for $\$ 0.5 $. How much money does John make? T: How many trees does John plant? S: Isn’t it just$ 3+4 $trees? T: It resembles an array or multiplication. Could you try a different way? S:$ 3\times 4=12 $trees. T: If each of these$ 12 $trees gives$ 5 $apples, how many apples are there? S: I don’t know. Is it$ 60$ ?	– Correct – On track – Weak justification	The answer is correct, but the student gives no reasoning, so the tutor should ask for justification rather than simply advance.
Arithmetic repair	Problem: John plants a plot of $3$ trees by $4$ trees. Each tree gives $5$ apples, and each apple sells for $\$ 0.5 $. How much money does John make? T: How many trees does John plant? S: John plants$ 3\times 4=12 $trees. T: That’s right! Now, if each tree gives$ 5 $apples, how many apples does John get in total? S: Yes, I think John gets$ 12\times 5=70$ apples in total.	– Incorrect – On track – Arithmetic error	The student selects the correct operation but miscomputes $12\times 5$ , so the tutor should repair the arithmetic rather than change the subgoal.

Appendix

Table 7: Qualitative examples of turn-level tutor-policy annotations for one source problem in DICE-Bench . We show three different dialogue rollouts for the same problem and the unique corresponding tutor actions annotated by a frontier model (GPT-5).

Action	Meaning
Accept and advance	The student has satisfied the current subgoal, so the tutor should acknowledge the response and move to the next step without adding extra scaffolding.
Arithmetic repair	The student is following the right reasoning path but makes a numerical or computation error; the tutor should correct or guide the calculation while preserving the current subgoal.
Conceptual hint	The student shows a misunderstanding of the underlying concept or operation; the tutor should provide a light conceptual cue without directly giving away the full solution.
Handle question	The student asks a clarification or meta-level question; the tutor should answer the question and then return the student to the relevant subgoal.
Ask for justification	The student’s answer may be correct or partially correct, but the reasoning is missing or weak; the tutor should prompt the student to explain or justify their step.
Refocus subgoal	The student skips the current subgoal, jumps ahead, or drifts from the intended trajectory; the tutor should redirect attention to the current step.

Appendix

Table 8: Pedagogical action space used for turn-level policy annotation in DICE-Bench .

Table 18

\mathcal{L}_{\textsc{DICE}}=-\frac{1}{\sum_{i}w_{i}}\sum_{i=1}^{N}w_{i}\log\pi_{\theta}(a_{i}\mid x_{i})+\beta D_{\mathrm{KL}}\!\left(\pi_{\theta}(\cdot\mid x_{i})\,\|\,\pi_{\mathrm{SFT}}(\cdot\mid x_{i})\right).

Appendix

Algorithm 1 DICE : IV-Calibrated Tutoring Policy

Table 20

Model	Training setup	Act. Acc.	Act. F1	OverInt. $\downarrow$	$q^{-}$ F1	$q$ F1	$q^{+}$ F1
Llama-3.1-8B Ins.	Pretrained	3.42	3.32	38.42	2.26	4.05	3.43
	SFT ( DA )	74.54	57.64	5.82	60.07	43.50	60.61
	SFT ( A )	76.78	61.83	4.64	64.51	40.02	68.10
	IV calibrated SFT ( A )	84.65	71.43	2.79	74.54	49.75	79.67
Qwen-2.5-3B Ins.	Pretrained	8.99	11.25	38.47	6.78	13.13	11.94
	SFT ( DA )	33.28	23.20	17.16	27.14	18.56	24.92

Appendix

Table 13: Turn-level policy prediction results on the DICE-Bench test split. We report action accuracy, action macro-F1, over-intervention rate, and action macro-F1 separately on the easy variant

q^{-}

, source problem

q

, and hard variant

q^{+}

. Higher is better for accuracy and F1; lower is better for over-intervention. The best value in each column is bolded and the second-best is underlined. The highlighted row indicates the best overall policy by action macro-F1. DA means we trained the model to generate Diagnosis and then Action to be taken, A means we generated only the Action for the given tutor state.

Student Model	Prompting Setup	Overall Acc.	$q^{-}$ Acc.	$q$ Acc.	$q^{+}$ Acc.
Qwen-2.5-0.5B Ins.	Zero-shot	10.25	19.90	7.70	3.15
	CoT	10.92	20.75	9.15	2.85
	CoT + Steps	13.35	23.95	9.10	7.00
Qwen-2.5-1.5B Ins.	Zero-shot	20.73	38.25	17.00	6.95
	CoT	21.22	39.30	17.75	6.60
	CoT + Steps	24.38	43.20	18.70	11.25

Appendix

Table 14: Student-only question-answering accuracy on DICE-Bench . Each model is evaluated on

6{,}000

problems, with

2{,}000

examples each from

q^{-}

q

, and

q^{+}

. These results measure the intrinsic problem-solving capability of the student model without tutor interaction.

Rank	Mode	Teacher	HP	Acc. $\uparrow$	Turns-S $\downarrow$	Int./Prob. $\downarrow$	OverInt. $\downarrow$	FalseAcc. $\downarrow$	Score $\uparrow$
1	DICE-Cost	Qwen-2.5-7B	$\lambda=0.25$	0.865	1.617	0.885	0.000	0.277	0.682
2	DICE-Threshold	Qwen-2.5-7B	$\tau=0.8$	0.851	1.595	1.068	0.034	0.297	0.632
3	DICE-Cost	Qwen-2.5-7B	$\lambda=1.0$	0.815	1.597	1.192	0.000	0.281	0.615
4	DICE-MAP	Qwen-2.5-7B	–	0.836	1.746	1.027	0.007	0.349	0.606
5	DICE-Cost	SocraticLM	$\lambda=0.5$	0.812	1.760	0.758	0.000	0.342	0.603
6	DICE-Cost	SocraticLM	$\lambda=2.0$	0.805	1.842	0.705	0.000	0.389	0.576

Appendix

Table 15: Validation sweep for DICE inference hyperparameters on

150

single-problem tutoring episodes. We include

50

examples each from

q^{-}

q

, and

q^{+}

\lambda

is the intervention cost for DICE-Cost ;

\tau

is the intervention threshold for DICE-Threshold . Turns-S denotes average turns among successfully solved problems; Int./Prob. denotes average active interventions per problem. Higher is better for accuracy (0-1) and score; lower is better for turns, interventions, over-intervention (0-1), and FA(0-1).

Case	Tutor	Problem and dialogue rollout	Behavior
Over-intervention Problem : Mary earns $\$ 46 $for cleaning one home. She earned$ $300 $. How many whole homes could she clean and how much money would be left over? Answer :$ 6 $homes,$ $24$ left.	SocraticLM	S: $300\div 46=6.52$ , so Mary cleaned $6$ homes. The leftover is $300-(46\times 6)=\$ 8 $. T: False acceptance That’s correct! Mary cleaned$ 6 $whole homes and had$ $8 $left over. — Student acknowledges, dialogue continues for 5 turns . T: There seems to be a small error. Let’s try again:$ 300-(46\times 6)=$84-$276=$124 $. S: Mary cleaned$ 6 $homes and had$ $124 $left. T: Recalculate the remainder:$ 300-$276=$24 $. S: Mary cleaned$ 6 $homes and had$ $24 $left. T: Over-intervention Let’s try it again:$ 300-$276=$24$ . — Student gets confused and misguided, and teacher continues intervention	The baseline first accepts an incorrect remainder, then repeatedly intervenes even after the student reaches the correct answer. It continues intervening till set budget.
	DICE	S: Mary could clean $7$ whole homes since $46\times 7=322$ , and have $\$ 18 $left over. T: Arithmetic repair Let’s recheck the division of$ $300 $by$ $46 $. How many whole homes can Mary clean with$ $300 $? S: Mary can clean$ 6 $whole homes, since$ 46\times 6=$276 $. T: Accept and advance That’s correct. Please continue to the next step. S: The leftover is$ 300-276=$24$ . T: Accept and advance That’s correct.	DICE applies one targeted arithmetic repair, then accepts once the student is on track.
False acceptance Problem : A library bookcase held $825$ books. On Monday $247$ were borrowed; on Tuesday $198$ were returned; on Wednesday $156$ more were borrowed. How many books are on the shelf at the end of Wednesday? Answer : $620$ .	SocraticLM	S: $825-247+198-156=420$ . So there are $420$ books remaining. T: False acceptance That’s correct! You’ve successfully calculated the number of books left on the shelf at the end of Wednesday. Well done! S: Thank you for your encouragement! It’s great to see I was able to solve this problem correctly. T: Keep up the good work!	The baseline validates the wrong answer $420$ , reinforcing an incorrect trajectory instead of repairing the arithmetic.
	DICE	S: $825-247+198-156=420$ . So there are $420$ books remaining. T: Arithmetic repair Let’s recheck the subtraction and addition step by step. You started with $825$ books. After Monday, $247$ were borrowed, so you did $825-247$ . Can you verify that calculation? S: $825-247=578$ . Then $578+198=776$ , and $776-156=620$ . T: Accept and advance That’s correct.	DICE detects the arithmetic error, repairs the intermediate step, and accepts only after the student reaches the correct answer.

Appendix

Table 16: Qualitative examples of single-problem dialogue rollouts comparing SocraticLM and DICE . To capture realistic simulator variability, student generations were not seeded, while the student model was kept fixed across systems. For readability, we remove repetitive/noisy turns and retain only the essential exchanges needed to illustrate tutor behavior. The first example shows repeated over-intervention, while the second shows false acceptance of an incorrect student answer. Colored tags indicate the tutor action or failure mode.

Intelligent Tutoring Systems

Intelligent tutoring systems model learner knowledge and adapt instruction, practice, hints, and feedback to support learning.

See all 154 papers

Apr 2, 2026

Do Large Language Models Mentalize When They Teach?

How do LLMs decide what to teach next: by reasoning about a learner's knowledge, or by using simpler rules of thumb? We test this in a controlled task previously used to study human teaching strategies. On each trial, a teacher LLM sees a hypothetical learner's trajectory through a reward-annotated directed graph and must reveal a single edge so the learner would choose a better path if they replanned. We run a range of LLMs as simulated teachers and fit their trial-by-trial choices with the same cognitive models used for humans: a Bayes-Optimal teacher that infers which transitions the learner is missing (inverse planning), weaker Bayesian variants, heuristic baselines (e.g., reward based), and non-mentalizing utility models. In a baseline experiment matched to the stimuli presented to human subjects, most LLMs perform well, show little change in strategy over trials, and their graph-by-graph performance is similar to that of humans. Model comparison (BIC) shows that Bayes-Optimal teaching best explains most models' choices. When given a scaffolding intervention, models follow auxiliary inference- or reward-focused prompts, but these scaffolds do not reliably improve later teaching on heuristic-incongruent test graphs and can sometimes reduce performance. Overall, cognitive model fits provide insight into LLM tutoring policies and show that prompt compliance does not guarantee better teaching decisions.

Sevan K. Harootonian, Mark K. Ho, Thomas L. Griffiths +2

Department of Psychology, Princeton University · Department of Psychology, New York University · Department of Computer Science, Department of Psychology, Princeton University +1

Theory of Mind

Ask PDF

Jul 30, 2026

Rethinking LLM-Judged Helpfulness as a Pedagogy Signal: A Pre-Registered Audit Across Tutor Models

LLM tutoring poses a measurement problem: can a general-purpose helpfulness rubric distinguish direct answer-giving from pedagogical guidance? We audit this signal in a pre-registered study. Within each of three tutor bases, we compare conversational and pedagogical policies instantiated with the same underlying model and paired with one fixed weak simulated student. Deterministic detectors measure answer leakage and next-turn independent work. Claude Opus 4.8 is the frozen, condition-blind primary judge. After the Opus scores were fixed, GPT-5.6 Sol was prospectively specified for a post hoc robustness audit of the same 1,179 confirmatory answer-phase tutor turns under the frozen helpfulness and pedagogy rubrics. On the primary base under Opus, the policies do not differ significantly in helpfulness but are perfectly rank-separated under the pedagogy rubric (Cliff's

|δ|{=}0.10

vs.

1.0

). Across the two judges, pedagogy contrasts retain their direction where detected, whereas the helpfulness ordering is judge-contingent, reversing between judges on two of three bases. In an Opus-only ablation, seven primary-base policies span

2.3

points in mean judged pedagogy within a

0.25

-point band of mean judged helpfulness. Separately, answer-revealing turns are followed by less independent student work on every base, a result that is judge-invariant by construction. In this controlled setting, general-purpose helpfulness is not a reliable pedagogy signal. Tutor evaluation should pair pedagogy-targeted rubrics with deterministic process measures.

Shuyi Fan, Boyuan Deng, Mengyu Xu +4

Columbia University, USA · Johns Hopkins University, USA · The University of Chicago, USA +4

LLM Evaluation LLM-as-a-Judge Intelligent Tutoring Systems Generative AI in Education Educational Dialogue Systems Education

Ask PDF

Aug 24, 2026

LLM Pedagogical Behavior in AI Tutoring Interactions

Students increasingly use LLMs as tutors for coursework and problem solving. Little is known about the level of assistance LLMs provide when students use them as tutors in authentic learning interactions. This matters because tutoring responses can differ substantially in how directly they help students complete a task. We operationalize this dimension as scaffolding level and develop a five-level scale, validated against human annotations, that characterizes responses according to the degree of direct assistance they provide. We apply the scale to 14,637 LLM responses from 203 students in a university AI course. Responses are overwhelmingly concentrated at high levels of assistance, with more than 95% classified as either Explaining or Solving. Scaffolding level is systematically associated with students' subsequent conversational behavior, but provides little additional predictive information about performance on three subsequent exams beyond prior achievement and dialogue behavior. These findings provide an empirical baseline for LLM assistance in tutoring interactions and a measurement framework for evaluating how alternative tutoring designs change that assistance.

Suhyeon Lee, Juneha Baek, Jaehyeong Park +1

KAIST

Instructional Scaffolding AI in Education Intelligent Tutoring Systems

Ask PDF

Abstract

Intelligent Tutoring Systems

Explore similar work

Do Large Language Models Mentalize When They Teach?

Rethinking LLM-Judged Helpfulness as a Pedagogy Signal: A Pre-Registered Audit Across Tutor Models

LLM Pedagogical Behavior in AI Tutoring Interactions