Offline-to-online reinforcement learning (RL) offers a promising paradigm for robotics by pre-training policies on safe, offline demonstrations and fine-tuning them via online interaction. However, a fundamental challenge remains: how to safely explore online without deviating from the behavioral support of the offline data? While recent methods leverage conditional variational autoencoders (CVAEs) to bound exploration within a latent space, they inherently suffer from an exploitation gap -- a performance ceiling imposed by the decoder's reconstruction loss. We introduce SPAARS, a curriculum learning framework that initially constrains exploration to the low-dimensional latent manifold for sample-efficient, safe behavioral improvement, then seamlessly transfers control to the raw action space, bypassing the decoder bottleneck. SPAARS has two instantiations: the CVAE-based variant requires only unordered (s,a) pairs and no trajectory segmentation; SPAARS-SUPE pairs SPAARS with OPAL temporal skill pretraining for stronger exploration structure at the cost of requiring trajectory chunks. We prove an upper bound on the exploitation gap using the Performance Difference Lemma, establish that latent-space policy gradients achieve provable variance reduction over raw-space exploration, and show that concurrent behavioral cloning during the latent phase directly controls curriculum transition stability. Empirically, SPAARS-SUPE achieves 0.825 normalized return on kitchen-mixed-v0 versus 0.75 for SUPE, with 5x better sample efficiency; standalone SPAARS achieves 92.7 and 102.9 normalized return on hopper-medium-v2 and walker2d-medium-v2 respectively, surpassing IQL baselines of 66.3 and 78.3 respectively, confirming the utility of the unordered-pair CVAE instantiation.
Figures & tables
Method
BC
Safe
Optimal
k -RND
IQL
✓
—
×
—
PLAS
✓
✓
×
—
ExPLORe
✓
×
✓
×
SUPE
×
∼
✓
×
SPAARS
✓
✓
✓
✓
TABLE I : Comparison of offline-to-online RL methods.
Algorithm 1: SPAARS Training
Input: Offline dataset D , CVAE architecture, RL algorithm
Output: Trained policy πraw
// Phase 0: CVAE Pretraining
1. Train CVAE (qϕ,pθ,pψ) on D via ELBO; freeze Dec .
Fig. 1 : Normalized return vs. environment steps on kitchen-mixed-v0. SPAARS -SUPE (gate) warm-starts from the pretrained OPAL IQL policy, reaching SUPE’s asymptotic performance 5 × faster and surpassing it by 300k steps.
Fig. 2 : AntMaze-Medium (left): SPAARS -SUPE matches native SUPE performance. Advantage Gate Firing (right): Mean advantage of raw policy over latent policy increases as training progresses, enabling precise goal-reaching.
Fig. 3 : Gate firing patterns over training. Red crosses indicate states where the Advantage Gate selects πraw . As training progresses, the gate successfully isolates raw-policy execution to the goal region while utilizing the latent policy for general maze exploration.
Environment
IQL [ 9 ]
SPAARS (ours)
hopper-medium-v2
66.3
92.7 ± 3.7
walker2d-medium-v2
78.3
102.9 ± 9.4
TABLE II : Standalone SPAARS (schedule) vs. offline IQL on D4RL locomotion tasks. Normalized return at 300k steps, mean ± 95% CI over 3 seeds.
Fig. 4 : Normalized return vs. environment steps on hopper-medium-v2 (left) and walker2d-medium-v2 (right). Standalone SPAARS exceeds the offline IQL baseline on both tasks, validating that a CVAE trained on unordered (s,a) pairs provides an effective latent manifold for online fine-tuning. Shaded regions indicate 95% confidence intervals over 3 seeds.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Safe exploration remains a fundamental challenge in reinforcement learning (RL), limiting the deployment of RL agents in the real world. We propose Sampling-Based Safe Reinforcement Learning (SBSRL), a model-based RL algorithm that maintains safety throughout the learning process by enforcing constraints jointly across a finite set of dynamics samples. This formulation approximates an intractable worst-case optimization over uncertain dynamics and enables practical safety guarantees in continuous domains. We further introduce an exploration strategy based on constraining epistemic uncertainty, eliminating the need for explicit exploration bonuses. Under regularity conditions, we derive high-probability guarantees of safety throughout learning and a finite-time sample complexity bound for recovering a near-optimal policy. Empirically, SBSRL achieves safe and efficient exploration both in simulation and in real robotic hardware, and readily extends to practical deep-ensemble implementations that scale to high-dimensional continuous control problems.
Offline policy improvement faces an inherent conflict between maximizing value and fitting the data distribution. While in-sample weighted regression is stable, it suffers from over-conservatism that suppresses high-value actions in the distribution tail; conversely, gradient-based approaches often exhibit a fitting-optimization conflict of gradients, which drives the policy off the data manifold. To address this, we propose Support-Preserving Action Rectification (SPAR), which reframes global learning as a local residual rectification anchored to a frozen pure behavior cloning policy. This framework performs fine-grained fitting and local policy improvement in the residual space, thereby contracting the search space. We further introduce Latent Self-Imitation, utilizing a latent-sampling weighted-regression mechanism to address fitting-improvement gradient conflict in the residual space. Theoretically, we prove this mechanism eliminates the manifold-normal drift of standard value gradients, while extensive D4RL experiments show SPAR extracts significant gains from suboptimal baselines to achieve state-of-the-art performance.
Safe exploration is a prerequisite for deploying reinforcement learning (RL) agents in safety-critical domains. In this paper, we approach safe exploration through the lens of epistemic uncertainty, where the actor's sensitivity to parameter perturbations serves as a practical proxy for regions of high uncertainty. We propose Sharpness-Aware Policy Optimization (SHAPO), a sharpness-aware policy update rule that evaluates gradients at perturbed parameters, making policy updates pessimistic with respect to the actor's epistemic uncertainty. Analytically we show that this adjustment implicitly reweighs policy gradients, amplifying the influence of rare unsafe actions while tempering contributions from already safe ones, thereby biasing learning toward conservative behavior in under-explored regions. Across several continuous-control tasks, our method consistently improves both safety and task performance over existing baselines, significantly expanding their Pareto frontiers.
Kaustubh Mani, Yann Pequignot, Vincent Mai +1
Universit´e de Montr´eal · Mila - Qu´ebec AI Institute · Universit´e Laval +2