Offline Apprenticeship Learning for Human-Centric Environments

Md Mirajul Islam

cs.LGOct 20, 2026

Offline Apprenticeship Learning for Human-Centric Environments

Authors: Md Mirajul Islam

Abstract

Effective decision-making in human-centric environments such as Intelligent Tutoring Systems (ITSs) and clinical healthcare requires policies that are adaptive, data-efficient, and ethically deployable. Apprenticeship Learning (AL) infers policies directly from expert demonstrations by recovering the underlying reward structure, enabling strictly offline learning without environment interaction. This dissertation develops a progression of offline AL frameworks organized around two challenges: reward modeling and demonstration quality. For reward modeling, EM-EDM clusters demonstrations by latent reward functions and induces cluster-specific policies, and THEMES partitions trajectories into temporally coherent sub-trajectories guided by a hierarchically learned reward regulator; across one educational and three clinical datasets, THEMES outperforms flat and hierarchical AL baselines. For demonstration quality, HALIDE integrates continuous quality signals into hierarchical policy induction, enabling principled reuse of mixed-quality data under distribution shift. CCHAL further attributes quality to individual decisions via Bayesian credit assignment, dual-arm criticality scoring that contrasts expert and non-expert behavior, and a Bradley-Terry contrastive IRL objective over failure trajectories, outperforming baselines including HALIDE on three clinical EHR datasets and an ITS dataset. Finally, a real-world classroom evaluation of StuEDM, an EDM-induced policy that preserves student agency and intervenes only on disagreement, shows significant learning gains over an Expert baseline and a higher pretest-adjusted posttest score than a CQL policy trained on a far larger corpus, while overriding students less than half as often. Together, this work establishes offline AL as a practical, data-efficient, and ethically viable paradigm for human-centric environments.

Figures & tables

Figure 3.2 : The interface of our probability tutor.

Category	Spring 21	Spring 22
$High_{pre}$ $\rightarrow$ $High_{post}$	9	13
$Medium_{pre}$ $\rightarrow$ $High_{post}$	8	9
$Low_{pre}$ $\rightarrow$ $High_{post}$	3	2
$Low_{pre}$ $\rightarrow$ $Medium_{post}$	4	5
Total	24	29

Table 3.1 : No. of Students in High QLG Group.

Category	Methods	Acc	Rec	Prec	F1	AUC	APR	Jaccard
DRL	CQL-T	0.296(0.05)	0.274(0.07)	0.369(0.06)	0.259(0.03)	0.454(0.06)	0.357(0.03)	0.161(0.03)
DRL	CQL	0.376(0.02)	0.358(0.04)	0.329(0.04)	0.325(0.03)	0.526(0.02)	0.382(0.01)	0.221(0.02)
AL	BC	0.354(0.02)	0.342(0.02)	0.346(0.02)	0.333(0.02)	0.508(0.02)	0.339(0.01)	0.204(0.01)
	GAIL	0.387(0.05)	0.355(0.03)	0.360(0.07)	0.311(0.04)	0.515(0.02)	0.343(0.01)	0.197(0.03)
	AIRL	0.365(0.03)	0.375(0.02)	0.382(0.02)	0.330(0.02)	0.533(0.01)	0.353(0.01)	0.204(0.02)
	EDM	0.737(0.05)	0.708(0.05)	0.723(0.06)	0. 705(0.06)	0.836(0.04)	0.700(0.05)	0.563(0.06)

Table 3.2 : Comparing EM-EDM to baselines on predicting student pedagogical actions (5-fold Cross-validation). The best methods are in bold, and the overall best is highlighted with *.

Category	Methods	Acc	Rec	Prec	F1	AUC	APR	Jaccard
DRL	CQL-T	0.217	0.203	0.313	0.199	0.391	0.332	0.115
DRL	CQL	0.321	0.354	0.284	0.308	0.506	0.366	0.198
AL	BC	0.328	0.334	0.344	0.317	0.501	0.335	0.190
	GAIL	0.393	0.362	0.387	0.337	0.522	0.345	0.210
	AIRL	0.341	0.355	0.351	0.331	0.517	0.343	0.202
	EDM	0.669	0.654	0.663	0.656	0.789	0.656	0.498

Table 3.3 : Comparing EM-EDM to baselines on predicting future students’ pedagogical actions. The best methods are in bold, and the overall best is highlighted with *.

Panel of Figure 3 — Figure 3.4 : Critical difference diagram with Conover Friedman test over (a) AUC and (b) Jaccard Score.

Semester	C0	C1	C2	C3	C4	Total
Spring 21	3	9	8	2	2	24
Spring 22	6	5	11	3	4	29

Table 3.4 : Distribution among the Discovered Clusters in Task 2 (C0-C4 refers to the 5 clusters).

Methods	MIMIC-III				CCHS				Mayo
Methods	F1	AUC	AP	JAC	F1	AUC	AP	JAC	F1	AUC	AP	JAC
BC	$.497_{\pm.01}$	$.502_{\pm.00}$	$.485_{\pm.07}$	$.331_{\pm.01}$	$.499_{\pm.01}$	$.500_{\pm.01}$	$.233_{\pm.02}$	$.376_{\pm.01}$	$.503_{\pm.01}$	$.504_{\pm.01}$	$.038_{\pm.01}$	$.475_{\pm.01}$
GP+DQN	$.418_{\pm.04}$	$.516_{\pm.03}$	$.515_{\pm.04}$	$.349_{\pm.05}$	$.333_{\pm.03}$	$.511_{\pm.04}$	$\mathbf{.526}_{\pm.03}$	$.211_{\pm.03}$	$.326_{\pm.01}$	$.460_{\pm.02}$	$.497_{\pm.04}$	$.269_{\pm.05}$
EDM	$.634_{\pm.07}$	$.635_{\pm.07}$	$.590_{\pm.05}$	$.471_{\pm.08}$	$\mathbf{.631}_{\pm.04}$	$.553_{\pm.03}$	$.522_{\pm.01}$	$\mathbf{.479}_{\pm.05}$	$.701_{\pm.10}$	$\mathbf{.640}_{\pm.04}$	$\mathbf{.515}_{\pm.01}$	$.558_{\pm.13}$
EM-EDM	$\mathbf{.778}^{*}_{\pm.05}$	$\mathbf{.779}^{*}_{\pm.05}$	$\mathbf{.717}^{*}_{\pm.05}$	$\mathbf{.641}^{*}_{\pm.07}$	$.607_{\pm.02}$	$\mathbf{.605}^{*}_{\pm.04}$	$\mathbf{.547}^{*}_{\pm.02}$	$.444_{\pm.03}$	$\mathbf{.796}^{*}_{\pm.09}$	$.597_{\pm.04}$	$.511_{\pm.01}$	$\mathbf{.684}^{*}_{\pm.12}$

Table 3.5 : Comparing EM-EDM against its core baselines (BC, GP+DQN, EDM) over three healthcare datasets.

Figure 4.1 : Overview of the THEMES Framework.

Methods	Acc	Rec	Prec	F1	AUC	AP	JAC
1. BC ( Gleave et al. 2022 )	$.366_{\pm.03}$	$.366_{\pm.03}$	$.384_{\pm.04}$	$.363_{\pm.03}$	$.523_{\pm.03}$	$.362_{\pm.01}$	$.224_{\pm.02}$
2.GP+DQN ( Azizsoltani and Sadeghi 2018 )	$.389_{\pm.06}$	$.389_{\pm.06}$	$.399_{\pm.10}$	$.297_{\pm.05}$	$.525_{\pm.06}$	$.386_{\pm.03}$	$.193_{\pm.04}$
3. EDM ( Jarrett et al. 2020a )	$.656_{\pm.05}$	$.656_{\pm.05}$	$.686_{\pm.03}$	$.658_{\pm.05}$	$.801_{\pm.03}$	$.662_{\pm.04}$	$.506_{\pm.05}$
4. MIL ( Hausman et al. 2017 )	$.508_{\pm.06}$	$.508_{\pm.06}$	$.486_{\pm.11}$	$.456_{\pm.06}$	$.627_{\pm.06}$	$.465_{\pm.06}$	$.319_{\pm.06}$
5. AHIRL ( Krishnan et al. 2016 )	$.545_{\pm.07}$	$.545_{\pm.07}$	$.566_{\pm.07}$	$.545_{\pm.07}$	$.688_{\pm.07}$	$.537_{\pm.07}$	$.388_{\pm.07}$
6. EM-EDM	$\mathbf{.705_{\pm.06}}$	$\mathbf{.705_{\pm.06}}$	$\mathbf{.725_{\pm.04}}$	$\mathbf{.709_{\pm.05}}$	$\mathbf{.847_{\pm.03}}$	$\mathbf{.724_{\pm.05}}$	$\mathbf{.561_{\pm.07}}$

Table 4.1 : Comparing THEMES with baselines and ablations.

Panel of Figure 4 — Figure 4.4 : Critical difference diagram for AUC and Jaccard Score.

Figure 4.5 : t-SNE Visualization of Expert Student Demonstrations— blue Trajectory (representing a fixed reward function) vs. black Trajectory (indicating evolving reward functions).

Methods		MIMIC-III				CCHS				Mayo
Methods		F1	AUC	AP	JAC	F1	AUC	AP	JAC	F1	AUC	AP	JAC
Baselines	BC	$.497_{\pm.01}$	$.502_{\pm.00}$	$.485_{\pm.07}$	$.331_{\pm.01}$	$.499_{\pm.01}$	$.500_{\pm.01}$	$.233_{\pm.02}$	$.376_{\pm.01}$	$.503_{\pm.01}$	$.504_{\pm.01}$	$.038_{\pm.01}$	$.475_{\pm.01}$
	GP+DQN	$.418_{\pm.04}$	$.516_{\pm.03}$	$.515_{\pm.04}$	$.349_{\pm.05}$	$.333_{\pm.03}$	$.511_{\pm.04}$	$.526_{\pm.03}$	$.211_{\pm.03}$	$.326_{\pm.01}$	$.460_{\pm.02}$	$.497_{\pm.04}$	$.269_{\pm.05}$
	EDM	$.634_{\pm.07}$	$.635_{\pm.07}$	$.590_{\pm.05}$	$.471_{\pm.08}$	$.631_{\pm.04}$	$.553_{\pm.03}$	$.522_{\pm.01}$	$.479_{\pm.05}$	$.701_{\pm.10}$	$.640_{\pm.04}$	$\mathbf{.515}_{\pm.01}$	$.558_{\pm.13}$
	EM-EDM	$\mathbf{.778}_{\pm.05}$	$.779_{\pm.05}$	$\mathbf{.717}_{\pm 0.05}$	$\mathbf{.641}_{\pm.07}$	$.607_{\pm.02}$	$\mathbf{.605}_{\pm.04}$	$.547_{\pm.02}$	$.444_{\pm.03}$	$.796_{\pm.09}$	$.597_{\pm.04}$	$.511_{\pm.01}$	$\mathbf{.684}_{\pm.12}$
	AHIRL	$.710_{\pm.05}$	$.754_{\pm.04}$	$.692_{\pm.05}$	$.550_{\pm.06}$	$.599_{\pm.04}$	$.601_{\pm.05}$	$\textbf{.549}_{\pm.06}$	$.461_{\pm.05}$	$.795_{\pm.06}$	$.432_{\pm.03}$	$.481_{\pm.01}$	$.660_{\pm.10}$

Table 4.2 : Comparing THEMES with baselines and ablations over three healthcare datasets.

Figure 5.1 : Overview of the HALIDE framework.

Data	Methods	Acc	Rec	Prec	F1	AUC	APR	JAC
E	BC ( Gleave et al. 2022 )	$.434_{\pm.08}$	$.434_{\pm.08}$	$.490_{\pm.17}$	$.453_{\pm.11}$	$.488_{\pm.08}$	$.490_{\pm.18}$	$.314_{\pm.10}$
	EDM ( Jarrett et al. 2020a )	$.673_{\pm.09}$	$.673_{\pm.09}$	$.754_{\pm.06}$	$.695_{\pm.07}$	$.766_{\pm.13}$	$.755_{\pm.07}$	$.548_{\pm.08}$
	AHIRL ( Krishnan et al. 2016 )	$.586_{\pm.02}$	$.586_{\pm.02}$	$.666_{\pm.11}$	$.602_{\pm.05}$	$.716_{\pm.06}$	$.658_{\pm.13}$	$.443_{\pm.05}$
	THEMES	$\mathbf{.724_{\pm.07}}$	$\mathbf{.724_{\pm.07}}$	$\mathbf{.784_{\pm.05}}$	$\mathbf{.742_{\pm.05}}$	$\mathbf{.803_{\pm.11}}$	$\mathbf{.787_{\pm.06}}$	$\mathbf{.603_{\pm.07}}$
E	EDM W	$.674_{\pm.09}$	$.674_{\pm.09}$	$.755_{\pm.06}$	$.696_{\pm.07}$	$.769_{\pm.13}$	$.757_{\pm.07}$	$.550_{\pm.08}$
E	HALIDE 0	$\mathbf{.738_{\pm.08}}$	$\mathbf{.738_{\pm.08}}$	$\mathbf{.785_{\pm.06}}$	$\mathbf{.752_{\pm.06}}$	$\mathbf{.813_{\pm.11}}$	$\mathbf{.809^{*}_{\pm.06}}$	$\mathbf{.617_{\pm.08}}$

Table 5.1 : Semester-based temporal cross-validation results (mean

\pm

std).

Panel of Figure 5 — Figure 5.4 : Critical difference diagram for F-score and Jaccard.

Data	Method	CCHS				Mayo				MIMIC
Data	Method	F1	AUC	AP	JAC	F1	AUC	AP	JAC	F1	AUC	AP	JAC
E	EDM	$.623_{\pm.04}$	$.549_{\pm.02}$	$.519_{\pm.01}$	$.470_{\pm.04}$	$.597_{\pm.10}$	$.560_{\pm.06}$	$.523_{\pm.02}$	$.447_{\pm.11}$	$.622_{\pm.06}$	$.628_{\pm.07}$	$.583_{\pm.05}$	$.456_{\pm.06}$
	EM-EDM	$.601_{\pm.03}$	$.626_{\pm.03}$	$.555_{\pm.02}$	$.436_{\pm.04}$	$.609_{\pm.08}$	$.568_{\pm.05}$	$.527_{\pm.02}$	$.457_{\pm.09}$	$.763_{\pm.04}$	$.757_{\pm.04}$	$.694_{\pm.04}$	$.620_{\pm.05}$
	THEMES	$\mathbf{.696}_{\pm.03}$	$\mathbf{.725}_{\pm.02}$	$\mathbf{.613}_{\pm.02}$	$\mathbf{.541}_{\pm.04}$	$\mathbf{.643}_{\pm.07}$	$\mathbf{.622}_{\pm.05}$	$\mathbf{.552}_{\pm.02}$	$\mathbf{.490}_{\pm.08}$	$\mathbf{.848}_{\pm.09}$	$\mathbf{.845}_{\pm.09}$	$\mathbf{.797}_{\pm.11}$	$\mathbf{.748}_{\pm.13}$
E	EDM w	$.622_{\pm.04}$	$.554_{\pm.03}$	$.521_{\pm.01}$	$.469_{\pm.05}$	$.597_{\pm.10}$	$.560_{\pm.06}$	$.523_{\pm.02}$	$.448_{\pm.11}$	$.623_{\pm.06}$	$.629_{\pm.07}$	$.583_{\pm.05}$	$.457_{\pm.06}$
	EM-EDM w	$.586_{\pm.03}$	$.613_{\pm.06}$	$.549_{\pm.03}$	$.422_{\pm.04}$	$.593_{\pm.08}$	$.565_{\pm.05}$	$.525_{\pm.02}$	$.438_{\pm.08}$	$.768_{\pm.03}$	$.765_{\pm.04}$	$.700_{\pm.04}$	$.627_{\pm.04}$

Table 5.2 : Combined HALIDE vs baselines & ablations results for healthcare datasets.

	Target ( $+$ )	Reference ( $-$ )
Hi-risk (at risk pre-treatment)	$\mathcal{Q}^{+}_{\mathrm{hi}}$ overcame risk	$\mathcal{Q}^{-}_{\mathrm{hi}}$ stayed at risk, shock
Lo-risk (not at risk pre-treatment)	$\mathcal{Q}^{+}_{\mathrm{lo}}$ remained stable	$\mathcal{Q}^{-}_{\mathrm{lo}}$ deteriorated, shock

Table 6.1 : Notation for the four demonstration groups.

Figure 6.1 : Cchal overview. Trajectory-level demonstration ranks are distributed to per-timestep weights via InferGP Bayesian credit assignment in feature-action space (top left; pre-computed, stored as w_t_gp). Dual-arm contrastive criticality scores are estimated from kNN local JSD in clinical feature space (top right; built once per repeat): the high-risk arm contrasts $\mathcal{Q}^{+}_{\mathrm{hi}}$ vs. $\mathcal{Q}^{-}_{\mathrm{hi}}$ and the low-risk arm contrasts $\mathcal{Q}^{+}_{\mathrm{lo}}$ vs. $\mathcal{Q}^{-}_{\mathrm{lo}}$ , each anchored to the same initial risk stratum. Phase 3 weighted EM-EDM uses $\mathcal{Q}^{+}_{\mathrm{hi}}\cup\mathcal{Q}^{+}_{\mathrm{lo}}$ only (failures excluded from the generative model). Phase 4 BT-IRL uses $\mathcal{D}^{+}=\mathcal{Q}^{+}_{\mathrm{hi}}\cup\mathcal{Q}^{+}_{\mathrm{lo}}$ and $\mathcal{D}^{-}=\mathcal{Q}^{-}_{\mathrm{hi}}\cup\mathcal{Q}^{-}_{\mathrm{lo}}$ with arm-routed action-aware weights (bottom).

Method		Mayo				CCHS				MIMIC				ITS
Method		F1	AUC	APR	JAC	F1	AUC	APR	JAC	F1	AUC	APR	JAC	F1	AUC	APR	JAC
Baselines	BC	$.503_{\pm.01}$	$.504_{\pm.01}$	$.038_{\pm.01}$	$.475_{\pm.01}$	$.499_{\pm.01}$	$.500_{\pm.01}$	$.233_{\pm.02}$	$.376_{\pm.01}$	$.497_{\pm.01}$	$.502_{\pm.00}$	$.485_{\pm.07}$	$.331_{\pm.01}$	$.349_{\pm.05}$	$.512_{\pm.04}$	$.359_{\pm.02}$	$.213_{\pm.04}$
	GP+DQN	$.326_{\pm.01}$	$.460_{\pm.02}$	$.497_{\pm.04}$	$.269_{\pm.05}$	$.333_{\pm.03}$	$.511_{\pm.04}$	$.526_{\pm.03}$	$.211_{\pm.03}$	$.418_{\pm.04}$	$.516_{\pm.03}$	$.515_{\pm.04}$	$.349_{\pm.05}$	$.297_{\pm.05}$	$.472_{\pm.04}$	$.423_{\pm.04}$	$.198_{\pm.04}$
	EDM	$.701_{\pm.10}$	$.640_{\pm.04}$	$\mathbf{.515}_{\pm.01}$	$.558_{\pm.13}$	$.631_{\pm.04}$	$.553_{\pm.03}$	$.522_{\pm.01}$	$.479_{\pm.05}$	$.634_{\pm.07}$	$.635_{\pm.07}$	$.590_{\pm.05}$	$.471_{\pm.08}$	$.727_{\pm.05}$	$.855_{\pm.04}$	$\mathbf{.749}_{\pm.06}$	$.584_{\pm.06}$
	EM-EDM	$.796_{\pm.09}$	$.597_{\pm.04}$	$.511_{\pm.01}$	$\mathbf{.684}_{\pm.12}$	$.607_{\pm.02}$	$\mathbf{.605}_{\pm.04}$	$.547_{\pm.02}$	$.444_{\pm.03}$	$\mathbf{.778}_{\pm.05}$	$.779_{\pm.05}$	$\mathbf{.717}_{\pm.05}$	$\mathbf{.641}_{\pm.07}$	$\mathbf{.751}_{\pm.05}$	$\mathbf{.862}_{\pm.04}$	$.747_{\pm.06}$	$.611_{\pm.07}$
	AHIRL	$.795_{\pm.06}$	$.432_{\pm.03}$	$.481_{\pm.01}$	$.660_{\pm.10}$	$.599_{\pm.04}$	$.601_{\pm.05}$	$\mathbf{.549}_{\pm.06}$	$.461_{\pm.05}$	$.710_{\pm.05}$	$.754_{\pm.04}$	$.692_{\pm.05}$	$.550_{\pm.06}$	$.746_{\pm.05}$	$.783_{\pm.04}$	$.735_{\pm.05}$	$.602_{\pm.05}$

Table 6.2 : Comparing baselines (including THEMES, evaluated here purely as a prior baseline) and Cchal variants—including the full ablation study—over three healthcare datasets and one education (ITS) dataset.

Figure 6.2 : A Case Study Comparing Cchal vs. THEMES vs. EDM. The top three panels present temperature, respiratory rate, and heart rate, where the green regions denote normal values and the light-red regions denote abnormal values. The fourth panel compares all treatment actions suggested by the expert, EDM, THEMES, and Cchal . The dashed box is magnified in the bottom panel to show the actions preceding the expert’s first intervention.

Condition	$N$	post	post_iso	NLG_post	NLG_post_iso	NLG_post_SR	Time (h)
StuEDM	26	$\mathbf{0.908}^{*\dagger}$	$\mathbf{0.948}$	$\mathbf{0.685}^{\dagger}$	$\mathbf{0.829}^{\dagger}$	$\mathbf{0.323}^{\dagger}$	$\mathbf{1.52}^{*}$
StuCQL	27	$0.842$	$0.888$	$0.485$	$0.676$	$0.239$	$1.11$
Expert	27	$0.846$	$0.885$	$0.331$	$0.600$	$0.169$	$1.44$

Table 7.1 : Mean outcome measures by condition (students with pre

=1.0

excluded; Expert is the augmented sample, Section 7.5.2 ). Bold marks the highest mean per column. ∗ indicates significant one-sided pairwise comparison vs. StuCQL (pretest-adjusted ANCOVA contrast for post,

t

-test for the remaining columns); † indicates the analogous significant comparison vs. Expert (

\alpha=0.05

). Time is hours spent on ITS training, reported as an engagement outcome (Section 7.6.3 ).

Figure 7.1 : Student’s own presentation-mode choice vs. the policy’s final (post-override) decision, for StuCQL and StuEDM .

Figure 7.2 : Agreement vs. override rate for StuCQL and StuEDM .

	StuCQL (260/390 overridden)			StuEDM (110/390 overridden)
Student chose	PS	WE	CPS	PS	WE	CPS
PS	–	115	0	–	8	1
WE	22	–	0	37	–	2
CPS	15	108	–	53	9	–

Table 7.2 : Override transition counts: student’s original choice (rows) vs. the policy’s override target (cols). Both conditions contributed the same 390 decisions (39 students

\times

10 problems each; Section 7.5 ); this table restricts to the subset that was overridden — 260/390 for StuCQL and 110/390 for StuEDM , reflecting the agreement/override rates in Figure 7.2 . Diagonal is empty by construction (agreements are excluded).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Acronym	Abbreviation
AL	Apprenticeship Learning
BC	Behavior Cloning
BCIL	Batch Contrastive Imitation Learning
BEAR	Bootstrapping Error Accumulation Reduction
CQL	Conservative Q-Learning
DRL	Deep Reinforcement Learning

Appendix

Table A.1 : A summary of acronyms used in alphabetical order.

Variable	Definition
$a$	Action taken by the agent
$A$	Action space
$D$	Offline demonstration dataset
$\mathcal{D}_{E}$	Set of expert demonstrations
$\mathcal{D}_{I}$	Set of imperfect or mixed-quality demonstrations
$E_{\theta}(\cdot)$	Energy function parameterized by $\theta$

Appendix

Table B.1 : A summary of common variables and their definitions in alphabetical order.

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 304 papers

Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards
Mar 31, 2026Md Mirajul Islam, Rajesh Debnath, Adittya Soukarjya Saha +1Learning from Suboptimal Demonstrations Learning from Demonstration
North Carolina State University, Raleigh, NC 27606, USA
A Generalized Apprenticeship Learning Framework for Capturing Evolving Student Pedagogical Strategies
Feb 24, 2026Md Mirajul Islam, Xi Yang, Adittya Soukarjya Saha +2AI in Education Policy Learning
North Carolina State University, Raleigh, NC 27606, USA · IBM Research, Yorktown Heights, NY 10598, USA
JumpStart Your Policy Learning with Lessons from 160,000 Training Runs
Sep 12, 2026Nabil Omi, Eric Bae, Chung Yik Edward Yeung +2Offline RL Algorithm Selection
University of Washington · Microsoft

Mar 31, 2026J/K move · Enter open · S save

Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards

Md Mirajul Islam, Rajesh Debnath, Adittya Soukarjya Saha, Min Chi

North Carolina State University, Raleigh, NC 27606, USA

Learning from Suboptimal Demonstrations Learning from Demonstration Reinforcement Learning Intelligent Tutoring Systems Hierarchical RL

Open paper AskPDF arXiv

While apprenticeship learning has shown promise for inducing effective pedagogical policies directly from student interactions in e-learning environments, most existing approaches rely on optimal or near-optimal expert demonstrations under a fixed reward. Real-world student interactions, however, are often inherently imperfect and evolving: students explore, make errors, revise strategies, and refine their goals as understanding develops. In this work, we argue that imperfect student demonstrations are not noise to be discarded, but structured signals-provided their relative quality is ranked. We introduce HALIDE, Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards, which not only leverages sub-optimal student demonstrations, but ranks them within a hierarchical learning framework. HALIDE models student behavior at multiple levels of abstraction, enabling inference of higher-level intent and strategy from suboptimal actions while explicitly capturing the temporal evolution of student reward functions. By integrating demonstration quality into hierarchical reward inference,HALIDE distinguishes transient errors from suboptimal strategies and meaningful progress toward higher-level learning goals. Our results show that HALIDE more accurately predicts student pedagogical decisions than approaches that rely on optimal trajectories, fixed rewards, or unranked imperfect demonstrations.

Abstract

Offline RL

Explore similar work

Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards

A Generalized Apprenticeship Learning Framework for Capturing Evolving Student Pedagogical Strategies

JumpStart Your Policy Learning with Lessons from 160,000 Training Runs