A Generalized Apprenticeship Learning Framework for Capturing Evolving Student Pedagogical Strategies

Md Mirajul Islam; Xi Yang; Adittya Soukarjya Saha; Rajesh Debnath; Min Chi

cs.LGFeb 24, 2026

A Generalized Apprenticeship Learning Framework for Capturing Evolving Student Pedagogical Strategies

Authors: Md Mirajul Islam, Xi Yang, Adittya Soukarjya Saha, Rajesh Debnath, Min Chi

Organizations: North Carolina State University, Raleigh, NC 27606, USA · IBM Research, Yorktown Heights, NY 10598, USA

Abstract

Reinforcement Learning (RL) and Deep Reinforcement Learning (DRL) have advanced rapidly in recent years and have been successfully applied to e-learning environments like intelligent tutoring systems (ITSs). Despite great success, the broader application of DRL to educational technologies has been limited due to major challenges such as sample inefficiency and difficulty designing the reward function. In contrast, Apprenticeship Learning (AL) uses a few expert demonstrations to infer the expert's underlying reward functions and derive decision-making policies that generalize and replicate optimal behavior. In this work, we leverage a generalized AL framework, THEMES, to induce effective pedagogical policies by capturing the complexities of the expert student learning process, where multiple reward functions may dynamically evolve over time. We evaluate the effectiveness of THEMES against six state-of-the-art baselines, demonstrating its superior performance and highlighting its potential as a powerful alternative for inducing effective pedagogical policies and show that it can achieve high performance, with an AUC of 0.899 and a Jaccard of 0.653, using only 18 trajectories of a previous semester to predict student pedagogical decisions in a later semester.

Figures & tables

Figure 1: Overview of the THEMES Framework.

Methods	Acc	Rec	Prec	F1	AUC	AP	JAC
1. BC [ 15 ]	$.366_{\pm.03}$	$.366_{\pm.03}$	$.384_{\pm.04}$	$.363_{\pm.03}$	$.523_{\pm.03}$	$.362_{\pm.01}$	$.224_{\pm.02}$
2.GP+DQN [ 6 ]	$.389_{\pm.06}$	$.389_{\pm.06}$	$.399_{\pm.10}$	$.297_{\pm.05}$	$.525_{\pm.06}$	$.386_{\pm.03}$	$.193_{\pm.04}$
3. EDM [ 24 ]	$.656_{\pm.05}$	$.656_{\pm.05}$	$.686_{\pm.03}$	$.658_{\pm.05}$	$.801_{\pm.03}$	$.662_{\pm.04}$	$.506_{\pm.05}$
4. MIL [ 19 ]	$.508_{\pm.06}$	$.508_{\pm.06}$	$.486_{\pm.11}$	$.456_{\pm.06}$	$.627_{\pm.06}$	$.465_{\pm.06}$	$.319_{\pm.06}$
5. AHIRL [ 30 ]	$.545_{\pm.07}$	$.545_{\pm.07}$	$.566_{\pm.07}$	$.545_{\pm.07}$	$.688_{\pm.07}$	$.537_{\pm.07}$	$.388_{\pm.07}$
6. EM-EDM [ 7 ]	$\mathbf{.705_{\pm.06}}$	$\mathbf{.705_{\pm.06}}$	$\mathbf{.725_{\pm.04}}$	$\mathbf{.709_{\pm.05}}$	$\mathbf{.847_{\pm.03}}$	$\mathbf{.724_{\pm.05}}$	$\mathbf{.561_{\pm.07}}$

Table 1: Comparing THEMES with baselines and ablations

Panel of Figure 4 — Figure 4: Critical difference diagram for AUC and Jaccard Score

Figure 5: t-SNE Visualization of Expert Student Demonstrations— blue Trajectory (representing a fixed reward function) vs. black Trajectory (indicating evolving reward functions)

AI in Education

AI in Education studies how artificial intelligence can support teaching, learning, assessment, and educational decision-making.

See all 261 papers

Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards
Mar 31, 2026Md Mirajul Islam, Rajesh Debnath, Adittya Soukarjya Saha +1Learning from Suboptimal Demonstrations Learning from Demonstration
North Carolina State University, Raleigh, NC 27606, USA
Pedagogical Safety in Educational Reinforcement Learning: Formalizing and Detecting Reward Hacking in AI Tutoring Systems
Apr 5, 2026Oluseyi Olukola, Nick RahimiReward Hacking Safe RL
School of Computing Sciences and Computer Engineering, Hattiesburg, MS, USA
PEARL: Training Socratic Tutors with Pedagogically Aligned Reinforcement Learning
May 28, 2026Qikai Chang, Zhenrong Zhang, Linbo Chen +4Multi-Objective Reinforcement Learning Intelligent Tutoring Systems
University of Science and Technology of China · iFLYTEK Research

Mar 31, 2026J/K move · Enter open · S save

Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards

Md Mirajul Islam, Rajesh Debnath, Adittya Soukarjya Saha, Min Chi

North Carolina State University, Raleigh, NC 27606, USA

Learning from Suboptimal Demonstrations Learning from Demonstration Reinforcement Learning Intelligent Tutoring Systems Hierarchical RL

Open paper AskPDF arXiv

While apprenticeship learning has shown promise for inducing effective pedagogical policies directly from student interactions in e-learning environments, most existing approaches rely on optimal or near-optimal expert demonstrations under a fixed reward. Real-world student interactions, however, are often inherently imperfect and evolving: students explore, make errors, revise strategies, and refine their goals as understanding develops. In this work, we argue that imperfect student demonstrations are not noise to be discarded, but structured signals-provided their relative quality is ranked. We introduce HALIDE, Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards, which not only leverages sub-optimal student demonstrations, but ranks them within a hierarchical learning framework. HALIDE models student behavior at multiple levels of abstraction, enabling inference of higher-level intent and strategy from suboptimal actions while explicitly capturing the temporal evolution of student reward functions. By integrating demonstration quality into hierarchical reward inference,HALIDE distinguishes transient errors from suboptimal strategies and meaningful progress toward higher-level learning goals. Our results show that HALIDE more accurately predicts student pedagogical decisions than approaches that rely on optimal trajectories, fixed rewards, or unranked imperfect demonstrations.

Abstract

AI in Education

Explore similar work

Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards

Pedagogical Safety in Educational Reinforcement Learning: Formalizing and Detecting Reward Hacking in AI Tutoring Systems

PEARL: Training Socratic Tutors with Pedagogically Aligned Reinforcement Learning