Self Paced Gaussian Contextual Reinforcement Learning

Mohsen Sahraei Ardakani; Rui Song

cs.LGMar 24, 2026

Self Paced Gaussian Contextual Reinforcement Learning

Authors: Mohsen Sahraei Ardakani, Rui Song

Organizations: Department of Electrical and Computer Engineering North Carolina State University Raleigh, NC · Department of Statistics North Carolina State University Raleigh, NC

Abstract

Curriculum learning improves reinforcement learning (RL) efficiency by sequencing tasks from simple to complex. However, many self-paced curriculum methods rely on computationally expensive inner-loop optimizations, limiting their scalability in high-dimensional context spaces. In this paper, we propose Self-Paced Gaussian Curriculum Learning (SPGL), a novel approach that avoids costly numerical procedures by leveraging a closed-form update rule for Gaussian context distributions. SPGL maintains the sample efficiency and adaptability of traditional self-paced methods while substantially reducing computational overhead. We provide theoretical guarantees on convergence and validate our method across several contextual RL benchmarks, including the Point Mass, Lunar Lander, and Ball Catching environments. Experimental results show that SPGL matches or outperforms existing curriculum methods, especially in hidden context scenarios, and achieves more stable context distribution convergence. Our method offers a scalable, principled alternative for curriculum generation in challenging continuous and partially observable domains.

Figures & tables

Panel of Figure 1 — Figure 1: Point mass environment with initial context sample Figure 1(a) and target context sample in Figure 1(b)

Curriculum	Average Collected Reward			Success Rate
	mean	se	p-value	mean	se
Default	17.09	1.426	0.008	37	17.1
Self-paced	22.16	0.201	0.071	99	0.7
Gaussian Self-paced	22.64	0.098		100	0.0

Table 1: Performance summary of CL methods for Point Mass environment(setup 1)

Panel of Figure 2 — Figure 2: Point Mass setup 1 experiment with hidden context curriculum learning and training

Curriculum	Average Collected Reward			Success Rate
	mean	se	p-value	mean	se
Default	17.31	1.438	0.011	37.5	17.12
Self-paced	22.57	0.028		99.94	0.00
Gaussian Self-paced	22.25	0.305	0.359	98.39	1.41

Table 2: Performance summary of CL methods for Point Mass environment(setup 2)

Panel of Figure 3 — Figure 3: Point Mass setup 2 experiment with visible context curriculum learning and training

Panel of Figure 4 — Figure 4: Lunar Lander experiment with hidden context curriculum learning and training

Curriculum	Average Collected Reward			Success Rate
	mean	se	p-value	mean	se
Default	246.55	3.458	0.042	89	1.3
Self-paced	252.59	3.199	0.373	88	1.9
Gaussian Self-paced	256.29	1.921		90	0.8

Table 3: Performance summary of CL methods for Lunar Lander environment

Panel of Figure 5 — Figure 5: Ball Catching experiment with hidden context curriculum learning and training

Curriculum	Average Collected Reward			Success Rate
	mean	se	p-value	mean	se
Default	-60.96	4.187	0.0	7	2
Self-paced	-51.11	6.524	0.009	21	2.5
Gaussian Self-paced	-24.56	4.662		31	2

Table 4: Performance summary of CL methods for Ball Catching environment

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Experiment	Target Context Distribution	Initial Context Distribution	$V_{-}$
Point Mass(setup 1)	$\widetilde{\mu}=[2.6,0.7,0.1]$	$\mu_{0}=[0,4,2]$	5
Point Mass(setup 1)	$\widetilde{\theta}=10^{-4}[9,4,1]$	$\theta_{0}=[4,3.5,1]$	5
Point Mass(setup 2)	$\widetilde{\mu}=[2.5,0.7,0.1]$	$\mu_{0}=[0,4,2]$	5
Point Mass(setup 2)	$\widetilde{\theta}=10^{-4}[10000,9,1]$	$\theta_{0}=[4,3.5,1]$	5
Lunar Lander	$\widetilde{\mu}=[-10,15,1.5]$	$\mu_{0}=[-7,10,1]$	1
Lunar Lander	$\widetilde{\theta}=[1,9,0.09]$	$\theta_{0}=[4,16,0.16]$	1

Appendix

Table 5: Experiment parameters summary

Contextual RL

See all 19 papers

Per-Context Reverse Curricula for Sparse-Reward Reinforcement Learning
Jul 29, 2026Siddharth Aphale, Ayushman SinghCurriculum RL Curriculum Learning
Stanford University · Sesame AI
Active Curriculum Refinement for Reinforcement Learning
Aug 26, 2026Zhenya Liu, Yuxin ChenCurriculum RL Curriculum Learning
The University of Chicago, Chicago IL, USA.
Curriculum Generation under Structured Parametric Environments for Robust Navigation Policies
Aug 9, 2026Prishita RayCurriculum RL Robot Policy Generalization
Cornell University

Jul 29, 2026J/K move · Enter open · S save

Per-Context Reverse Curricula for Sparse-Reward Reinforcement Learning

Siddharth Aphale, Ayushman Singh

Stanford University · Sesame AI

Curriculum RL Curriculum Learning Sparse-Reward RL

Open paper AskPDF arXiv

Sparse-reward reinforcement learning often fails because rollouts from the unassisted evaluation start rarely reach later task stages. Reverse curricula address this by starting some training rollouts from easier intermediate states, called scaffolds. The main choices are which intermediate starts to use (scaffold access) and how to distribute training across them (scaffold allocation). Most prior curricula pace removal on one shared schedule, which can fail when task instances, or contexts, learn at different rates. We introduce an online, learner-agnostic reset controller that gives every context its own curriculum. We use binary rollout success to decide whether to move a context's start forward or back. A stalled frontier also gets an occasional harder-start trial. A counting construction shows that synchronized global pacing can be insufficient when contexts need conflicting amounts of assisted practice. Across six navigation and manipulation settings, scaffold access improves learning and enables success in three where unassisted training fails within the reported budget. None of the global schedules we tested solved both groups in the constructed conflict. The per-context controller did. Group-level pacing works if the labels capture how contexts learn. Without those labels, per-context pacing still handles differences within a group. These results suggest pacing assistance separately for contexts that learn at different rates.

Abstract

Contextual RL

Explore similar work

Per-Context Reverse Curricula for Sparse-Reward Reinforcement Learning

Active Curriculum Refinement for Reinforcement Learning

Curriculum Generation under Structured Parametric Environments for Robust Navigation Policies