Pedagogical Safety in Educational Reinforcement Learning: Formalizing and Detecting Reward Hacking in AI Tutoring Systems

Oluseyi Olukola; Nick Rahimi

cs.AIApr 5, 2026

Pedagogical Safety in Educational Reinforcement Learning: Formalizing and Detecting Reward Hacking in AI Tutoring Systems

Authors: Oluseyi Olukola, Nick Rahimi

Organizations: School of Computing Sciences and Computer Engineering, Hattiesburg, MS, USA

Abstract

Reinforcement learning (RL) is increasingly used to personalize instruction in intelligent tutoring systems, yet the field lacks a formal framework for defining and evaluating pedagogical safety. We introduce a four-layer model of pedagogical safety for educational RL comprising structural, progress, behavioral, and alignment safety and propose the Reward Hacking Severity Index (RHSI) to quantify misalignment between proxy rewards and genuine learning. We evaluate the framework in a controlled simulation of an AI tutoring environment with 120 sessions across four conditions and three learner profiles, totaling 18{,}000 interactions. Results show that an engagement-optimized agent systematically over-selected a high-engagement action with no direct mastery gain, producing strong measured performance but limited learning progress. A multi-objective reward formulation reduced this problem but did not eliminate it, as the agent continued to favor proxy-rewarding behavior in many states. In contrast, a constrained architecture combining prerequisite enforcement and minimum cognitive demand substantially reduced reward hacking, lowering RHSI from 0.317 in the unconstrained multi-objective condition to 0.102. Ablation results further suggest that behavioral safety was the most influential safeguard against repetitive low-value action selection. These findings suggest that reward design alone may be insufficient to ensure pedagogically aligned behavior in educational RL, at least in the simulated environment studied here. More broadly, the paper positions pedagogical safety as an important research problem at the intersection of AI safety and intelligent educational systems.

Figures & tables

Work	Domain	Formal	Safety	Reward	Empirical
		Def.	Constr.	Hacking	Evidence
Skalse et al. (2022)	General RL	✓	–	✓	–
Pan et al. (2022)	General RL	–	–	✓	✓
Shihab et al. (2025)	General RL	–	–	✓	✓
Achiam et al. (2017)	Robotics	✓	✓	–	–
García and Fernández (2015)	Survey	✓	✓	–	–

Table 1: Comparison with related work. Our contribution (bottom row) appears to be the first to combine formal safety constraints with empirical reward hacking evidence in educational RL.

Figure 1: SmartTutor system architecture. Three layers govern the tutoring loop: the Student Interaction Layer, the SmartTutor Core (five components), and the Safety Monitoring Layer, which evaluates C2–C4 constraint violations and aggregates them into the RHSI.

Profile	Initial Knowledge	Gain Multiplier	Confusion Threshold
Struggling	0.05	$0.8\times$	0.30
Average	0.15	$1.0\times$	0.50
Advanced	0.30	$1.2\times$	0.70

Table 2: Learner profile parameters used in simulation. Gain multiplier scales action-specific base mastery gains. Confusion threshold governs when high confusion suppresses mastery updates during a session.

Action ( $d$ value)	EO	MAS	MO	ST
Encourage ( $0.0$ )	25.8	1.9	32.6	12.2
Explain_Simple ( $0.2$ )	16.6	27.3	13.0	12.2
Provide_Hint ( $0.3$ )	26.8	7.5	6.2	14.2
Explain_Detailed ( $0.4$ )	3.3	40.2	5.7	13.5
Assess_Knowledge ( $0.5$ )	2.6	2.7	5.0	10.9
Provide_Example ( $0.5$ )	20.6	11.0	30.0	18.4

Table 3: Action distribution across conditions (% of 4,500 interactions per condition). See also Figure 2 . Encourage has

d=0.0

and mastery gain

=0.0

: it produces no learning but the largest engagement boost (

+0.8

Figure 2: Action distribution across conditions (% of 4,500 interactions per condition). MO selects Encourage at the highest rate of any condition (32.6%), exceeding even EO (25.8%). ST produces the most balanced distribution, with no action exceeding 18.4% and substantially higher usage of high-demand actions than any unconstrained condition.

Constraint	EO	MAS	MO	ST
C2 (progress)	0.538	0.250	0.517	0.214
C3 (demand floor)	0.708	0.391	0.580	0.058
C4 (coupling)	0.677	0.329	0.704	0.345
$\\|\mathbf{v}\\|_{w}$ (weighted norm)	0.645	0.328	0.606	0.237

Table 4: Constraint violation rates. C1 = 0 for all conditions (action masking enforced in ST; other conditions do not target prerequisite-violating concepts by design). C2:

\varepsilon_{\text{prog}}=0.0023

(calibrated from MAS 25th percentile),

W=10

. C3:

\delta_{\min}=0.4

W=10

. C4:

\rho_{\max}=1.2

, rewards normalized to

[0,1]

W_{0}=10

Cond.	Profile	$\Delta K$	$N_{\text{mastered}}$
EO	Struggling	$0.011\pm 0.006$	$0.0\pm 0.0$
EO	Average	$0.013\pm 0.006$	$0.1\pm 0.3$
EO	Advanced	$0.013\pm 0.007$	$0.3\pm 0.5$
MAS	Struggling	$0.014\pm 0.003$	$0.0\pm 0.0$
MAS	Average	$0.018\pm 0.001$	$0.0\pm 0.0$
MAS	Advanced	$0.021\pm 0.002$	$1.0\pm 0.0$

Table 5: Learning outcomes per condition (mean

\pm

std across 10 seeds).

\Delta K

: mean mastery gain across all 27 concepts.

N_{\text{mastered}}

: concepts reaching

k\geq 0.7

Figure 3: Mean cumulative knowledge level (estimated per-concept mastery averaged across all 27 concepts) over 150 interactions, by condition and learner profile (mean

\pm

SEM across 10 seeds). Note this shows the knowledge level , not mastery gain

\Delta K

reported in Table 5 . MAS and ST consistently achieve the highest knowledge levels across all profiles; the gap widens with learner capability. ST matches MAS on mastery trajectories despite maintaining the engagement signal in its reward function.

Condition	$\hat{V}^{\pi}$	$\hat{V}^{\pi}/\hat{V}^{*}$	$\\|\mathbf{v}\\|_{w}$	$v_{3}$ (C3)	RHSI
EO	148.84	1.000	0.645	0.708	0.645
MO	77.95	0.524	0.606	0.580	0.317
MAS	47.52	0.319	0.328	0.391	0.105
ST	64.42	0.433	0.237	0.058	0.102

Table 6: Reward Hacking Severity Index by condition.

\hat{V}^{*}=148.84

(EO). Equal weights

w_{i}=1/3

for C2, C3, C4; C1 = 0 by construction. Lower RHSI = safer. Note: MAS shows RHSI = 0.105 here (condition-level aggregate) vs. mean 0.117 in Table 7 (per-seed average); the small discrepancy arises from non-linearities in the reward–violation ratio computed at different levels of aggregation.

Condition	Mean $\pm$ SD	Median	IQR
EO (Engagement-Only)	$0.737\pm 0.615$	0.435	[0.312, 0.950]
MO (Multi-Objective)	$0.355\pm 0.240$	0.238	[0.145, 0.577]
No C3 (action filter off)	$0.288\pm 0.190$	0.254	[0.144, 0.305]
No C1 (KG enforcement off)	$0.244\pm 0.111$	0.245	[0.161, 0.269]
MAS (Mastery-Only)	$0.117\pm 0.057$	0.101	[0.065, 0.169]
ST (SmartTutor Full)	$0.095\pm 0.028$	0.091	[0.074, 0.115]

Table 7: Per-seed RHSI descriptives and pairwise Welch t -tests (

n=30

seeds per condition; 3 profiles

\times

10 seeds).

\Delta=\bar{x}_{\text{row}}-\bar{x}_{\text{col}}

; positive

\Delta

indicates the row condition has higher (worse) RHSI than ST. Bonferroni

\alpha_{\text{corrected}}=0.05/6=0.0083

. * Bonferroni-significant;

\dagger

p<.05

uncorrected; ns not significant.

Figure 4: Per-seed RHSI distributions across all conditions (

n=30

per condition; 3 profiles

\times

10 seeds). Box shows IQR, white line is median, whiskers extend to 1.5

\times

IQR, dots are outliers. ST achieves the lowest and most consistent RHSI. Significance brackets show Bonferroni-corrected comparisons. The dashed line at

\varepsilon=0.25

marks the realistic deployment safety threshold discussed in Section 5.5 .

Condition	C3 rate	$\\|\mathbf{v}\\|_{w}$	RHSI	$\Delta$ RHSI	Ped. Approp.
ST (full)	0.058	0.237	0.102	—	1.000
No C1 (KG off)	0.604	0.495	0.240	+0.138	1.000
No C3 (filter off)	0.683	0.574	0.264	+0.162	0.899

Table 8: Ablation study: RHSI and constraint violations when individual constraints are removed.

\Delta

RHSI shows degradation relative to the full framework (ST). Removing C3 (action filter) appears to cause the largest safety degradation.

$W$	$\delta_{\min}$	EO	MAS	MO	ST	Best	2nd Best
5	0.30	0.685	0.101	0.335	0.099	ST	MAS
5	0.35	0.719	0.109	0.344	0.106	ST	MAS
5	0.40	0.729	0.115	0.350	0.122	MAS	ST
5	0.45	0.746	0.169	0.363	0.173	MAS	ST
5	0.50	0.757	0.174	0.369	0.203	MAS	ST
10	0.30	0.712	0.104	0.340	0.096	ST	MAS

Table 9: Parameter sensitivity: mean RHSI by condition across the

W\times\delta_{\min}

grid (20 cells). EO has the strictly highest RHSI in all 20 cells. MO is consistently second worst. ST–MAS ordering reverses at

\delta_{\min}\geq 0.45

(see text).

Figure 5: Parameter sensitivity: mean RHSI across the $W\times\delta_{\min}$ grid (20 cells). Darker shading indicates higher RHSI (worse safety) for EO and MO; darker green indicates higher RHSI for MAS and ST. EO remains the worst condition in all 20 cells. The dashed border on the ST panel marks the $\delta_{\min}\geq 0.45$ region where MAS edges out ST (see text); outside this region ST achieves the lowest RHSI of any condition.

	EO		MAS		MO		ST
Variant	RHSI	C3	RHSI	C3	RHSI	C3	RHSI	C3
Baseline ( $\times 1.0$ )	0.739	0.708	0.117	0.391	0.356	0.580	0.097	0.060
$+20\%$	0.723	0.660	0.108	0.291	0.347	0.514	0.096	0.005
$-20\%$	0.768	0.805	0.175	0.816	0.375	0.725	0.202	0.744

Table 10: Cognitive demand perturbation: mean RHSI and C3 violation rate under

\pm 20\%

scaling of non-anchor demand values (

W=10

\delta_{\min}=0.40

). Anchors:

d(\text{Encourage})=0.0

d(\text{Challenge})=1.0

. Condition ordering appears stable under

+20\%

; ST and MAS exchange positions under

-20\%

(see text).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	vs.	Profile	$\Delta$ [95% CI]	$t$ (df)	$p$	$d$	Bonf.
Pedagogical appropriateness
	EO	Struggling	$+0.13$ [ $+0.07$ , $+0.20$ ]	$t(9)=\phantom{-}4.96$	$.001$	$\phantom{-}2.22$
	EO	Average	$+0.12$ [ $+0.05$ , $+0.18$ ]	$t(9)=\phantom{-}4.17$	$.002$	$\phantom{-}1.87$
	EO	Advanced	$0.00$	—	—	—	—
	MAS	Struggling	$+0.18$ [ $+0.10$ , $+0.26$ ]	$t(9)=\phantom{-}5.21$	$.001$	$\phantom{-}2.33$
	MAS	Average	$+0.16$ [ $+0.06$ , $+0.25$ ]	$t(9)=\phantom{-}3.79$	$.004$	$\phantom{-}1.70$	$\dagger$

Appendix

Table 11: Behavioral metric comparisons: SmartTutor (ST) versus baselines and ablations across all learner profiles (

n=10

seeds per cell).

\Delta=\bar{x}_{\text{ST}}-\bar{x}_{\text{comp}}

; positive values favour ST. Welch t -tests with Bonferroni correction (

\alpha_{\text{corrected}}=0.05/15

per metric). * Bonferroni-significant;

\dagger

p<.05

uncorrected; ns not significant; ‘—’ zero variance across all seeds. EO = Engagement-Only; MAS = Mastery-Only; MO = Multi-Objective (0.3/0.5/0.2). Per-seed RHSI comparisons appear in Table 7 .

ID	Concept	Depth	Direct Prerequisites	Difficulty
c01	Variables and Assignment	0	—	Beginner
c02	Data Types (Basic)	1	c01	Beginner
c03	String Operations	2	c02	Beginner
c04	Arithmetic Operations	2	c02	Beginner
c05	Input and Output	2	c02	Beginner
c08	Lists (Basics)	2	c02	Beginner

Appendix

Table 12: SmartTutor knowledge graph: 27 Python concepts with DAG depth, direct prerequisites, and difficulty tier. Depth = longest path from root (c01). Prerequisite mastery threshold

\theta_{\min}=0.5

gates accessibility. Source: python_kg_research.json .

Reward Hacking

Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.

See all 176 papers

Apr 5, 2026

MC-CPO: Mastery-Conditioned Constrained Policy Optimization for Pedagogically Safe Intelligent Tutoring Systems

Intelligent tutoring systems increasingly rely on reinforcement learning to personalise instruction, yet optimising for observable engagement signals can systematically decouple learner activity from genuine knowledge acquisition. Analysing over 21 million student interactions across two deployed platforms, we find engagement events without corresponding mastery gains occur in 26.5% of interactions on Junyi Academy (72,758 students) and 3.1% on XES3G5M (14,453 students, NeurIPS 2023), confirming this pattern is directly observable in deployed educational technology at scale. We introduce Mastery-Conditioned Constrained Policy Optimisation (MC-CPO), a reinforcement learning framework that addresses this problem structurally. MC-CPO conditions the admissible instructional action space on learner mastery state: a concept becomes available only when prerequisite knowledge meets a mastery threshold, yielding an action space that expands naturally as learners acquire knowledge. Pedagogical safety constraints are enforced by construction, with formal guarantees of structural prerequisite safety, primal-dual convergence, and strict dominance over post-hoc filtering. MC-CPO is the only method to reduce reward hacking severity across all conditions. Mean per-episode mastery gain increases by 18.3% on Junyi Academy and 54.0% on XES3G5M relative to all baselines, while competitive engagement performance is maintained. These results support structural constraint modelling as a principled foundation for safer adaptive instructional policies in deployed tutoring systems.

Oluseyi Olukola, Nick Rahimi

School of Computing Sciences and Computer Engineering University of Southern Mississippi Hattiesburg MS 39406 USA

Intelligent Tutoring Systems Policy Learning Safe RL Constrained RL Reward Hacking

Ask PDF

Mar 18, 2026

SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systems

Large language models are rapidly being deployed as AI tutors, yet current evaluation paradigms assess problem-solving accuracy and generic safety in isolation, failing to capture whether a model is simultaneously pedagogically effective and safe across student-tutor interaction. We argue that tutoring safety is fundamentally different from conventional LLM safety: the primary risk is not toxic content but the quiet erosion of learning through answer over-disclosure, misconception reinforcement, and the abdication of scaffolding. To systematically study this failure mode, we introduce SafeTutors, a benchmark that jointly evaluates safety and pedagogy across mathematics, physics, and chemistry. SafeTutors is organized around a theoretically grounded risk taxonomy comprising 11 harm dimensions and 48 sub-risks drawn from learning-science literature. We uncover that all models show broad harm; scale doesn't reliably help; and multi-turn dialogue worsens behavior, with pedagogical failures rising from 17.7% to 77.8%. Harms also vary by subject, so mitigations must be discipline-aware, and single-turn "safe/helpful" results can mask systematic tutor failure over extended interaction.

Rima Hazra, Bikram Ghuku, Ilona Marchenko +5

Eindhoven University of Technology, Netherlands (TU/e) · Indian Institute of Technology Kharagpur · Taras Shevchenko National University of Kyiv +3

Intelligent Tutoring Systems AI Safety Evaluation LLM Safety Evaluation

Ask PDF

Jun 13, 2026

Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds

Reward hacking, where AI systems exploit misspecified objectives to achieve high reward without satisfying intended goals, remains a central challenge in AI safety. Yet most known instances have been discovered post hoc in frontier systems where controlled study is impractical. We adapt the AI Safety Gridworlds framework into a text-based evaluation suite that reformulates classic reinforcement learning safety tasks for language-based agents. Across frontier and mid-scale models, we find that specification gaming emerges zero-shot: models systematically achieve high observed reward while underperforming on hidden safety objectives, and even apparently safe behaviors can reflect misunderstanding rather than principled safety. Reinforcement learning does not correct these failures: direct reward optimization widens the gap between observed and hidden reward, as the model's initial competence causes it to lock into locally rewarding strategies before discovering safer alternatives. This pattern persists across model scales (1.5B--14B) and is not resolved by finer credit assignment, exploration prompts, or entropy regularization. Our results show that reward hacking arises naturally when optimizing proxy objectives with capable language model agents and resists standard mitigations, suggesting that proxy-reward failures in agentic settings may require approaches beyond standard exploration and credit-assignment fixes. To facilitate reproducibility, the code for this work is available at our public repository.

Ömer Veysel Çağatan, Xuandong Zhao

KUIS AI Center, Koc¸ University · Istanbul, Türkiye · University of California, Berkeley +1

Reward Hacking Language Model Safety Evaluation AI Agent Safety Benchmarks RL for LLM Agents

Ask PDF

Abstract

Reward Hacking

Explore similar work

MC-CPO: Mastery-Conditioned Constrained Policy Optimization for Pedagogically Safe Intelligent Tutoring Systems

SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systems

Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds