Intelligent tutoring systems increasingly rely on reinforcement learning to personalise instruction, yet optimising for observable engagement signals can systematically decouple learner activity from genuine knowledge acquisition. Analysing over 21 million student interactions across two deployed platforms, we find engagement events without corresponding mastery gains occur in 26.5% of interactions on Junyi Academy (72,758 students) and 3.1% on XES3G5M (14,453 students, NeurIPS 2023), confirming this pattern is directly observable in deployed educational technology at scale. We introduce Mastery-Conditioned Constrained Policy Optimisation (MC-CPO), a reinforcement learning framework that addresses this problem structurally. MC-CPO conditions the admissible instructional action space on learner mastery state: a concept becomes available only when prerequisite knowledge meets a mastery threshold, yielding an action space that expands naturally as learners acquire knowledge. Pedagogical safety constraints are enforced by construction, with formal guarantees of structural prerequisite safety, primal-dual convergence, and strict dominance over post-hoc filtering. MC-CPO is the only method to reduce reward hacking severity across all conditions. Mean per-episode mastery gain increases by 18.3% on Junyi Academy and 54.0% on XES3G5M relative to all baselines, while competitive engagement performance is maintained. These results support structural constraint modelling as a principled foundation for safer adaptive instructional policies in deployed tutoring systems.
Figures & tables
Work
CMDP
Feasibility
Dynamic
Edu.
CPO [ 13 ]
Yes
Static
No
No
RCPO [ 15 ]
Yes
Static
No
No
Lyapunov RL [ 14 ]
Yes
Static
No
No
FPI [ 18 ]
Yes
Policy-based
Yes
No
CCPO [ 27 ]
Yes
Threshold-cond.
No
No
KG-A2C [ 38 ]
No
Graph mask
No
No
Table 1: Comparison of Safe RL and Educational RL Approaches
Figure 1: MC-CPO pipeline. Real-world ITS data and a simulated BKT environment are preprocessed into CMDP episodes with prerequisite graphs and mastery trajectories. MC-CPO enforces mastery-conditioned feasibility via masked softmax and optimises a primal–dual Lagrangian with three pedagogical safety constraints. Results are validated under three independent measurement frameworks: RHSI (reward hacking), ΔK mastery gain (educational AI), and constraint satisfaction rate (safe RL).
Figure 2: Training return, π(hack) , and violation rate (mean ± std across 10 seeds).
Figure 3: Dual variable λ and constraint cost Jc during MC-CPO training.
Method
Return
π(hack)
Unconstrained
0.999±0.001
0.999±0.000
Post-hoc
0.0005±0.0006
0.999±0.000
MC-CPO
0.600±0.001
0.0004±0.0001
Table 2: Final performance (mean ± std; last 1,000 episodes).
Method
Jc
Viol. Rate
RHSI
Unconstrained
0.998±0.001
0.998±0.001
0.998±0.001
Post-hoc (train)
0.000±0.000
0.000±0.000
0.998±0.001
MC-CPO
0.0007±0.0008
0.0007±0.0008
0.0004±0.0001
Table 3: Safety and reward hacking severity (mean ± std; last 1,000 episodes).
Figure 4: Final return across methods (mean ± std across 10 seeds).
Method
Return
π(hack)
RHSI (raw)
Unconstrained
4.768±0.215
0.932±0.110
(2.45±2.58)×106
Post-hoc
2.941±0.000
0.000±0.000
2.406±0.508
MC-CPO
3.538±0.553
0.304±0.281
2.179±1.909
Table 4: Extended chain CMDP performance (mean ± std; last 1,000 episodes; 10 seeds). RHSI reported in raw form; the normalized form (Section 3 ) applies where violation rates are available.
Figure 5: Extended chain CMDP learning curves (mean ± std across 10 seeds). Left: return vs episodes. Right: RHSI vs episodes.
Method
Return
RHSI (raw)
Jc2
Jc3
Jc4
Unconstrained
34.64 ± 0.37
69.92 ± 0.85
0.917
19.02
27.14
Reward-Shaped
34.64 ± 0.37
69.92 ± 0.85
0.917
19.02
27.14
Post-hoc
34.64 ± 0.37
69.92 ± 0.85
0.917
19.02
27.14
MC-CPO
32.73 ± 0.48
44.47 ± 0.96
0.871
9.69
23.14
MC-CPO (no frontier)
32.68 ± 0.39
44.51 ± 0.81
0.872
9.69
23.12
Table 6: Neural final evaluation results (mean ± std over 10 seeds; RHSI reported in raw form). Constraint satisfaction: Jˉci≤(1+τ)di , τ=0.1 .
Figure 6: RHSI over training steps (15-concept simulated environment, mean ± std across 10 seeds). MC-CPO is the only method to reduce RHSI below the unconstrained baseline.
Method
Return
RHSI (raw)
Sat. Rate
Unconstrained
34.79±0.30
70.52±0.68
0%
Reward Shaped (all λ )
34.79±0.30
70.52±0.68
0%
Posthoc
34.79±0.30
70.52±0.68
0%
MC-CPO
31.97±0.31
39.16±2.02
80%
MC-CPO (no frontier)
32.11±0.52
40.05±2.93
70%
Table 7: 25-concept neural results (10 seeds, 1M steps). Raw RHSI reported. Constraint satisfaction rate reflects seeds satisfying all three budgets within tolerance τ=0.1 .
Dataset
Method
Return
RHSI
Jc2
Jc3
Jc4
Junyi Academy (16.2M interactions) 14 concepts, 17 edges Mastery: platform signal
Table 8: Real-world validation results (mean across 10 seeds; 200 final evaluation episodes per seed). RHSI reported in raw form. Sat.: fraction of seeds satisfying all three budget constraints within tolerance τ=0.1 . NF: no frontier mixing.
Figure 7: RHSI over training steps on Junyi Academy (mean ± std, 10 seeds). MC-CPO is the only method to reduce RHSI below the unconstrained baseline.
Figure 8: RHSI over training steps on XES3G5M (mean ± std, 10 seeds). MC-CPO reduces RHSI by 39.8% relative to all baselines.
Reinforcement learning (RL) is increasingly used to personalize instruction in intelligent tutoring systems, yet the field lacks a formal framework for defining and evaluating pedagogical safety. We introduce a four-layer model of pedagogical safety for educational RL comprising structural, progress, behavioral, and alignment safety and propose the Reward Hacking Severity Index (RHSI) to quantify misalignment between proxy rewards and genuine learning. We evaluate the framework in a controlled simulation of an AI tutoring environment with 120 sessions across four conditions and three learner profiles, totaling 18{,}000 interactions. Results show that an engagement-optimized agent systematically over-selected a high-engagement action with no direct mastery gain, producing strong measured performance but limited learning progress. A multi-objective reward formulation reduced this problem but did not eliminate it, as the agent continued to favor proxy-rewarding behavior in many states. In contrast, a constrained architecture combining prerequisite enforcement and minimum cognitive demand substantially reduced reward hacking, lowering RHSI from 0.317 in the unconstrained multi-objective condition to 0.102. Ablation results further suggest that behavioral safety was the most influential safeguard against repetitive low-value action selection. These findings suggest that reward design alone may be insufficient to ensure pedagogically aligned behavior in educational RL, at least in the simulated environment studied here. More broadly, the paper positions pedagogical safety as an important research problem at the intersection of AI safety and intelligent educational systems.
Oluseyi Olukola, Nick Rahimi
School of Computing Sciences and Computer Engineering, Hattiesburg, MS, USA
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising approach to improve the reasoning abilities of Large Language Models (LLMs). Among RLVR algorithms, Group Relative Policy Optimization (GRPO) and its variants have demonstrated strong performance and high training efficiency. However, GRPO-style objectives exhibit two issues on high accuracy prompts including mastered prompts (rollout accuracy =1) and majority-correct prompts (rollout accuracy in (0.5,1)). For mastered prompts, group-relative advantages vanish, yielding no training signal and unconstrained policy drift that can cause forgetting. For majority-correct prompts, the induced query weight shrinks as accuracy increases, weakening consolidation from partial correctness to mastery. To alleviate this, we propose Mastery-Consolidated Policy Optimization (MCPO), which introduces (i) a hinge-KL regularizer applied exclusively to mastered prompts to bound harmful policy drift between successive gradient steps, and (ii) a weighting mechanism that prioritizes majority-correct prompts to better allocate optimization effort. Extensive experiments across three mathematical benchmarks demonstrate that MCPO consistently improves pass@1 performance. Counter-intuitively, rather than restricting exploration, MCPO boosts pass@k metrics, indicating that mastery consolidation further catalyzes solution diversity.
Safe reinforcement learning (Safe RL) aims to maximize expected return while satisfying safety constraints, typically modeled as Constrained Markov Decision Processes (CMDPs). While primal-dual methods scale well to deep RL, they often suffer from delayed constraint correction, leading to oscillatory behavior and prolonged safety violations. In this paper, we propose Constraint-Sensitive Policy Optimization (CSPO), a first-order primal-dual method that incorporates local constraint sensitivity into policy updates. CSPO augments the primal objective with a constraint-sensitive correction derived from the shortest signed distance to the safety boundary, enabling smarter recovery steps back to safety, compensating for delayed Lagrange multiplier updates, reducing oscillations near the boundary, and preserving the KKT solutions of the original constrained problem. Experiments on navigation and locomotion benchmarks demonstrate that CSPO achieves faster safety recovery and high reward preservation, resulting in higher constrained returns compared to state-of-the-art primal-dual and penalty-based methods