EVOL: Simulator-Guided Evolutionary Expert Synthesis for Deployment-Free Learning Path Recommendation

Geonwoo Bang; Dongho Kim; Moohong Min

cs.AIOct 2, 2026

EVOL: Simulator-Guided Evolutionary Expert Synthesis for Deployment-Free Learning Path Recommendation

Authors: Geonwoo Bang, Dongho Kim, Moohong Min

Organizations: Department of Computer Education, Sungkyunkwan University Seoul, Republic of Korea · Department of Immersive Media Engineering/Social Innovation Convergence Program, Sungkyunkwan University Seoul, Republic of Korea · Department of Computer Education/Social Innovation Convergence Program, Sungkyunkwan University Seoul, Republic of Korea

Abstract

Reinforcement learning (RL) for learning path recommendation (LPR) faces two coupled obstacles. First, the policy must commit to a sequence of L concepts without intermediate feedback, producing a combinatorial search space that grows super-exponentially with L and provides reward only at the final step. Second, expert learning paths would be the natural cure for sparse-reward RL, but they do not exist in educational data, because student logs record what learners did, not what they should have done. We address both obstacles by importing a recipe from simulator-based demonstration learning in robotics: the knowledge tracing simulator is used both to synthesize per-learner expert demonstrations through evolutionary search and to train a deployment-free policy that distills these demonstrations into a feed-forward learner. Our framework, EVOL, instantiates this pipeline with an asymmetric actor-critic where the actor commits to deployment-realistic blind planning while the critic exploits the privileged simulator state during training. Across three datasets (ASSIST15, Junyi, and EdNet; 39-189 concepts) and path lengths L = 5, 10, and 20, EVOL surpasses 8 baselines spanning heuristic, sequential, RL, graph-enhanced RL, and LLM-enhanced methods. We further compare three imitation strategies (BC, AWR, and DAPG) and show that final performance is governed by the quality of evolutionary experts rather than by the particular imitation objective.

Figures & tables

Figure 1. Overview of EVOL. The KES is consulted only at train time. Stage 1 runs evolutionary search inside the KES to synthesize per-learner expert paths; Stage 2 distills them into a feed-forward actor conditioned only on the initial mastery

\mathbf{h}_{0}

; Stage 2.5 warm-starts the critic on the privileged

\mathbf{h}_{t}

; Stage 3 refines the actor with asymmetric PPO. At deployment the actor alone emits the full

L

-step path with no KES query.

Method	Algorithm preserved	Reward preserved	Inference uses KES (original)	Inference uses KES (ours)
GRU4Rec ( Hidasi et al., 2016 )	yes (GRU)	N/A (BC on experts)	no	no
SASRec ( Kang and McAuley, 2018 )	yes (transformer)	N/A (BC on experts)	no	no
PPO-vanilla ( Schulman et al., 2017 )	yes (PPO + clip)	terminal EP	N/A	no (asymmetric AC)
CSEAL ( Liu et al., 2019 )	yes (A2C + cognitive nav)	EP	yes	no (h 0 -only actor)
GEHRL ( Li et al., 2023 )	yes (hierarchical PPO)	EP	yes	no (h 0 -only actor)
DLELP ( Cheng et al., 2026 )	yes (PPO + plateau S-agent)	EP	yes	no (S-agent off in h 0 mode; graph-aware candidate mask retained)

Table 1. Baseline fidelity. “Original” columns preserve the published method; “Adapted” columns describe our changes for deployment-realistic, capacity-matched comparison.

Heuristic

Sequential

Graph-enhanced RL

LLM

Ours

Dataset

L

Target

GRU4Rec

SASRec

PPO

CSEAL

GEHRL

DLELP

KnowLP

EVOL-BC

EVOL-AWR

EVOL-DAPG

Junyi 39 KCs

+0.203

\pm

.003

+0.237

\pm

.008

+0.256

\pm

.007

+0.247

\pm

.010

+0.054

\pm

.049

+0.106

\pm

.003

+0.110

\pm

.013

+0.089

\pm

.019

+0.286

\pm

.003

+0.286

\pm

.000

+0.285

\pm

.005

+0.244

\pm

.002

+0.266

\pm

.009

+0.320

\pm

.006

+0.364

\pm

.003

+0.178

\pm

.008

+0.138

\pm

.008

+0.247

\pm

.003

+0.230

\pm

.010

+0.390

\pm

.001

+0.388

\pm

.001

+0.390

\pm

.001

+0.286

\pm

.001

-

0.005

\pm

.017

+0.299

\pm

.006

+0.380

\pm

.005

+0.207

\pm

.006

+0.045

\pm

.029

+0.337

\pm

.010

+0.337

\pm

.010

+0.426

\pm

.002

+0.420

\pm

.002

+0.428

\pm

.002

ASSIST15 100 KCs

+0.296

\pm

.011

+0.329

\pm

.007

+0.355

\pm

.008

+0.420

\pm

.011

+0.062

\pm

.022

+0.211

\pm

.007

+0.251

\pm

.015

+0.189

\pm

.034

+0.470

\pm

.003

+0.471

\pm

.005

+0.472

\pm

.004

+0.321

\pm

.009

+0.325

\pm

.010

+0.466

\pm

.011

+0.543

\pm

.009

+0.215

\pm

.013

+0.228

\pm

.003

+0.454

\pm

.002

+0.388

\pm

.011

+0.614

\pm

.002

+0.604

\pm

.001

+0.613

\pm

.003

Table 2. Test EP (mean

\pm

std over 3 seeds) across datasets and path lengths. Best in bold , second best underlined .

Figure 2. Validation EP during training on ASSIST15 ( $L{=}20$ , 3-seed mean). The EVOL variants emerge from Stage 2 already at Val EP $\approx+0.42$ ; PPO-vanilla starts at $0$ and crosses the same level only around episode 6,000 before plateauing near $+0.51$ . EVOL continues to climb to $+0.62$ over the remaining 24,000 episodes.

Variant	Test EP
Search strategy at Stage 1
Greedy 1-step lookahead (no learning)	+0.155 $\pm$ .029
Random-path experts $\to$ BC	+0.354 $\pm$ .011
Evolutionary search $\to$ BC	+0.614 $\pm$ .003
Downstream learner
PPO-vanilla (no experts, no pretraining)	+0.543 $\pm$ .009

Table 3. Component ablations on ASSIST15

L{=}10

. All variants share the same actor-critic architecture and hyperparameters; only the indicated component is removed or replaced. Mean

\pm

std over 3 seeds (729 test learners each, except Greedy which is eval-only on 200 learners).

$L$	Junyi ( $M{=}39$ )	ASSIST15 ( $M{=}100$ )	EdNet ( $M{=}189$ )
5	$6.9\!\times\!10^{7}$	$9.0\!\times\!10^{9}$	$2.3\!\times\!10^{11}$
10	$2.3\!\times\!10^{15}$	$6.3\!\times\!10^{19}$	$4.6\!\times\!10^{22}$
20	$1.7\!\times\!10^{29}$	$1.3\!\times\!10^{39}$	$1.2\!\times\!10^{45}$

Table 4. Combinatorial search space size across datasets and path lengths.

Method	EP	Coverage $\uparrow$	EP ${}_{\text{shuf}}$	SeqDrop $\uparrow$	StepLift $\uparrow$
CSEAL	+0.220	0.625	+0.063	+0.157	+0.010
GEHRL	+0.252	0.784	+0.070	+0.182	+0.011
DLELP	+0.479	0.435	+0.213	+0.266	+0.017
PPO-vanilla	+0.561	0.713	+0.333	+0.227	+0.020
EVOL-BC	+0.627	0.876	+0.330	+0.297	+0.022
EVOL-AWR	+0.616	0.864	+0.327	+0.289	+0.022

Table 5. Pedagogical decomposition on ASSIST15

L\!=\!10

(3 seeds, 200 test learners). “EP

{}_{\text{shuf}}

” is the average EP after uniformly shuffling each recommended path (5 shuffles per path); “SeqDrop” = EP

-

{}_{\text{shuf}}

measures the portion of EP that depends on ordering ; “StepLift” is the average per-step EP gain (EP

/L

		Alternative DKT (different training seed)
Method	Original DKT	seed 7	seed 100	seed 200	seed 333	Mean (4 alts)	$\Delta$ vs PPO
Simulator-guided demonstration learning (Ours)
EVOL-BC	+0.630	+0.314	+0.337	+0.350	+0.364	+0.341	+0.091
EVOL-DAPG	+0.627	+0.306	+0.333	+0.346	+0.362	+0.337	+0.087
EVOL-AWR	+0.619	+0.309	+0.330	+0.339	+0.367	+0.336	+0.086
Reinforcement learning baseline

Table 6. Robustness to DKT-instance shift on ASSIST15

L\!=\!10

. Policies (3 seeds) trained against the original DKT (seed 42) are evaluated by executing their recommended paths on four alternative DKT instances with different training seeds (same architecture and hyperparameters). The state estimator is the original DKT throughout; only the execution simulator changes. All entries are means across 3 policy seeds

\times

200 test learners (same subset and evaluator as Table 7 , so the Original DKT column numbers differ from Table 2 (729 learners) by at most

\pm 0.02

EP per method). Methods are grouped by paradigm and, within each group, sorted by the cross-instance mean (descending). Bold marks the overall best per column.

		Stochastic response model
Method	Determ. (ref.)	Bernoulli	Slip/Guess (0.10)	Slip/Guess (0.20)	Mean (3 stoch.)	$\Delta$ vs PPO
Simulator-guided demonstration learning (Ours)
EVOL-BC	+0.630	+0.305	+0.272	+0.223	+0.267	+0.043
EVOL-DAPG	+0.627	+0.303	+0.270	+0.224	+0.266	+0.042
EVOL-AWR	+0.619	+0.302	+0.272	+0.221	+0.265	+0.041
Reinforcement learning baseline

Table 7. Robustness to stochastic response models on ASSIST15

L\!=\!10

. The deterministic-trained checkpoint of each method is evaluated under Bernoulli and slip/guess response models without re-training (3 policy seeds

\times

200 test learners

\times

20 stochastic rollouts;

\mathbf{h}_{0}

-only paths). Methods grouped by paradigm and sorted within group by the mean across the three stochastic modes. Bold marks the overall best per column.

Recommender Systems

Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.

See all 516 papers

Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies
Sep 27, 2026Duo Wu, Haifeng Wang, Rongwei Lu +6Robotic RL Sparse-Reward RL
Tsinghua University · Yuanxing Robotics · Pengcheng Laboratory
LAR-MoE: Latent-Aligned Routing for Mixture of Experts in Robotic Imitation Learning
Mar 9, 2026Ariel Rodriguez, Chenpan Li, Lorenzo Mazza +5Robot Imitation Learning Agent Skill Learning
Department of Translational Surgical Oncology, NCT/UCC Dresden, a partnership between DKFZ, MF UKD, TUD, HZDR Dresden, Germany. · Cluster of Excellence-CeTI, TUD, Germany. · Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD, Germany.
ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priors
Mar 16, 2026Zifan Xu, Ran Gong, Maria Vittoria Minniti +10Long-Horizon Robotic Manipulation Robot Skill Learning
Robotics and AI Institute · University of Texas at Austin · Sony AI

Sep 27, 2026J/K move · Enter open · S save

Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies

Duo Wu, Haifeng Wang, Rongwei Lu, Jinghe Wang+5

Tsinghua University · Yuanxing Robotics · Pengcheng Laboratory

Robotic RL Sparse-Reward RL Policy Learning Reward Design for RL

Open paper AskPDF arXiv

Reinforcement learning (RL) enables generalist robot policies to improve through trial-and-error interaction, yet its effectiveness is fundamentally constrained by sparse task rewards. Existing general-purpose reward models typically alleviate this issue by learning task progress from expert demonstrations, but introduce a distribution mismatch with the mixed-quality rollouts encountered during policy optimization, making their estimates unreliable on suboptimal and failed behaviors from which the policy must learn. In this work, we introduce a demonstration-free reward learning paradigm where dense reward feedback can be learned directly from sparse task outcomes and policy experience. We theoretically show that terminal task outcomes implicitly define dense success-probability feedback at intermediate timesteps, which can be recursively learned through bootstrapping. Based on this insight, we introduce eVTA

_0

, which learns success probabilities from mixed-quality policy rollouts through temporal-difference-style bootstrapping, without expert demonstrations or intermediate annotations. We further introduce RL with Evolving Rewards (RLER), a closed-loop framework that adapts eVTA

_0

using newly collected rollouts as the policy evolves. Experiments show that eVTA

_0

provides more informative rewards than state-of-the-art reward models and achieves the best average policy performance across all LIBERO task suites under the same RL training budget, improving success rates by 5.4%-13.8% over the initial policy. In real-world manipulation, RLER further improves overall success rates by 20%-26%, with 35%-36% gains under out-of-distribution conditions. These results demonstrate the effectiveness of demonstration-free reward learning and adapting rewards as the policy evolves. Project webpage: https://duowuyms.github.io/evta0.

Abstract

Recommender Systems

Explore similar work

Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies

LAR-MoE: Latent-Aligned Routing for Mixture of Experts in Robotic Imitation Learning

ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priors