Escaping Offline Pessimism: Vector-Field Reward Shaping for Safe Frontier Exploration
Authors: Amirhossein Roknilamouki, Arnob Ghosh, Eylem Ekici, Ness B. Shroff
Organizations: Department of Electrical and Computer Engineering, The Ohio State University · Department of Electrical and Computer Engineering, New Jersey Institute of Technology · Department of Computer Science and Engineering, The Ohio State University.
While offline reinforcement learning provides reliable policies for real-world deployment, its inherent pessimism severely restricts an agent's ability to explore and collect novel data online. Drawing inspiration from safe reinforcement learning, exploring near the boundary of regions well covered by the offline dataset and reliably modeled by the simulator allows an agent to take manageable risks--venturing into informative but moderate-uncertainty states while remaining close enough to familiar regions for safe recovery. However, naively rewarding this boundary-seeking behavior can lead to a degenerate parking behavior, where the agent simply stops once it reaches the frontier. To solve this, we propose a novel vector-field reward shaping paradigm designed to induce continuous, safe boundary exploration for non-adaptive deployed policies. Operating on an uncertainty oracle trained from offline data, our reward combines two complementary components: a gradient-alignment term that attracts the agent toward a target uncertainty level, and a rotational-flow term that promotes motion along the local tangent plane of the uncertainty manifold. Through theoretical analysis, we show that this reward structure naturally induces sustained exploratory behavior along the boundary while preventing degenerate solutions. Empirically, by integrating our proposed reward shaping with Soft Actor-Critic on a 2D continuous navigation task, we validate that agents successfully traverse uncertainty boundaries while balancing safe, informative data collection with primary task completion.
Figures & tables
Figure 1: Toy navigation with localized uncertainty. (a) Example trajectories from the same start/goal. A pessimistic baseline (blue, dashed) takes a conservative detour to avoid the high-uncertainty region. Our approach drives the agent to the target manifold U (green), rotates along the boundary to collect informative samples, and then reaches the goal without entering the uncertain interior (shaded). (b) Visualization of the induced vector field for controlled exploration. The blue arrows direct the agent toward the high-uncertainty regions, seeking the target uncertainty boundary Umid . Upon reaching this target level, the orange arrows (Curl Boundary Band) induce a tangential flow, ensuring the agent continuously moves along and explores the surface of the manifold.
Figure 2: Our vector field reward design results in a periodic orbit behavior (the agent rotate around the uncertainty region to collect as much data as it can)
Figure 3: State-based intrinsic reward baseline demonstrating mode collapse. Without the rotational flow component, the agent’s state visitations (blue points) collapse into a highly localized cluster on the target manifold U rather than continuously exploring the frontier.
Figure 4: Balancing target manifold coverage with a primary navigation task. (a) Our method induces a time-splitting strategy, actively circulating the manifold before navigating to the goal. (b) The baseline intrinsic reward fails to achieve diverse boundary coverage, moving almost directly to the goal.
Figure 5: For higher dimensions ( d>2 ), our reward function in Eq. ( 2 ) generates different tangential vector fields for different choices of the skew-symmetric matrix W . Because rotational flow is not unique in these spaces, each arbitrary W dictates a distinct rotational trajectory along the boundary U .
Category
Hyperparameter
Value
Optimization
Actor Learning Rate
1×10−6
Critic Learning Rate
5×10−4
Alpha Learning Rate
2×10−3
Discount Factor ( γ )
0.99
Soft Target Update Rate ( τ )
10−4
Max Gradient Norm
30.0
Table 1: SAC and environment hyperparameters used in our experiments. All models were trained with a replay buffer capacity of 106 transitions.
Safe exploration is a prerequisite for deploying reinforcement learning (RL) agents in safety-critical domains. In this paper, we approach safe exploration through the lens of epistemic uncertainty, where the actor's sensitivity to parameter perturbations serves as a practical proxy for regions of high uncertainty. We propose Sharpness-Aware Policy Optimization (SHAPO), a sharpness-aware policy update rule that evaluates gradients at perturbed parameters, making policy updates pessimistic with respect to the actor's epistemic uncertainty. Analytically we show that this adjustment implicitly reweighs policy gradients, amplifying the influence of rare unsafe actions while tempering contributions from already safe ones, thereby biasing learning toward conservative behavior in under-explored regions. Across several continuous-control tasks, our method consistently improves both safety and task performance over existing baselines, significantly expanding their Pareto frontiers.
Kaustubh Mani, Yann Pequignot, Vincent Mai +1
Universit´e de Montr´eal · Mila - Qu´ebec AI Institute · Universit´e Laval +2
Autonomous exploration of cluttered environments requires efficient exploration strategies that guarantee safety against potential collisions with unknown random obstacles. This paper presents a novel approach combining a graph neural network-based exploration greedy policy with a safety shield to ensure safe navigation goal selection. The network is trained using reinforcement learning and the proximal policy optimization algorithm to maximize exploration efficiency while reducing the safety shield interventions. However, if the policy selects an infeasible action, the safety shield intervenes to choose the best feasible alternative, ensuring system consistency. Moreover, this paper proposes a reward function that includes a potential field based on the agent's proximity to unexplored regions and the expected information gain from reaching them. Overall, the approach investigated in this paper merges the benefits of the adaptability of reinforcement learning-driven exploration policies and the guarantee ensured by explicit safety mechanisms. Extensive evaluations in simulated environments demonstrate that the approach enables efficient and safe exploration in cluttered environments.
Luleå University of Technology · Robotics and AI Group, Department of Computer Science, Electrical and Space Engineering, Luleåa University of Technology, Sweden
Ensuring safe exploration in high-dimensional systems with unknown dynamics remains a significant challenge. Existing safe reinforcement learning methods often provide safety guarantees only in expectation, which can still lead to safety violations. Control-theoretic approaches, in contrast, offer hard constraint-based safety guarantees but typically assume access to known system dynamics or require accurate estimation of control-affine models. In this paper, we propose a safe reinforcement learning framework that learns a probabilistic control-affine dynamics model in an offline setting. The learned model is leveraged to explicitly construct control barrier functions (CBFs) that incorporate model uncertainty to provide conservative safety constraints. These CBF constraints are enforced through an online constraint-based action correction mechanism, enabling safe exploration without overly restricting task performance. Empirical evaluations on nonlinear, complex continuous-control benchmarks demonstrate that our approach achieves returns comparable to those of existing baselines while significantly reducing safety violations.
Rahul Narava, Siddharth Verma, Ojas Jain +2
Indian Institute of Technology Ropar · CRAN, Université de Lorraine