cs.LGMar 18, 2026

Escaping Offline Pessimism: Vector-Field Reward Shaping for Safe Frontier Exploration

Authors: Amirhossein Roknilamouki, Arnob Ghosh, Eylem Ekici, Ness B. Shroff

Organizations: Department of Electrical and Computer Engineering, The Ohio State University · Department of Electrical and Computer Engineering, New Jersey Institute of Technology · Department of Computer Science and Engineering, The Ohio State University.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

While offline reinforcement learning provides reliable policies for real-world deployment, its inherent pessimism severely restricts an agent's ability to explore and collect novel data online. Drawing inspiration from safe reinforcement learning, exploring near the boundary of regions well covered by the offline dataset and reliably modeled by the simulator allows an agent to take manageable risks--venturing into informative but moderate-uncertainty states while remaining close enough to familiar regions for safe recovery. However, naively rewarding this boundary-seeking behavior can lead to a degenerate parking behavior, where the agent simply stops once it reaches the frontier. To solve this, we propose a novel vector-field reward shaping paradigm designed to induce continuous, safe boundary exploration for non-adaptive deployed policies. Operating on an uncertainty oracle trained from offline data, our reward combines two complementary components: a gradient-alignment term that attracts the agent toward a target uncertainty level, and a rotational-flow term that promotes motion along the local tangent plane of the uncertainty manifold. Through theoretical analysis, we show that this reward structure naturally induces sustained exploratory behavior along the boundary while preventing degenerate solutions. Empirically, by integrating our proposed reward shaping with Soft Actor-Critic on a 2D continuous navigation task, we validate that agents successfully traverse uncertainty boundaries while balancing safe, informative data collection with primary task completion.

Figures & tables

Figure 1: Toy navigation with localized uncertainty. (a) Example trajectories from the same start/goal. A pessimistic baseline (blue, dashed) takes a conservative detour to avoid the high-uncertainty region. Our approach drives the agent to the target manifold U\mathcal{U} (green), rotates along the boundary to collect informative samples, and then reaches the goal without entering the uncertain interior (shaded). (b) Visualization of the induced vector field for controlled exploration. The blue arrows direct the agent toward the high-uncertainty regions, seeking the target uncertainty boundary UmidU_{\mathrm{mid}} . Upon reaching this target level, the orange arrows (Curl Boundary Band) induce a tangential flow, ensuring the agent continuously moves along and explores the surface of the manifold.
Figure 2: Our vector field reward design results in a periodic orbit behavior (the agent rotate around the uncertainty region to collect as much data as it can)
Figure 3: State-based intrinsic reward baseline demonstrating mode collapse. Without the rotational flow component, the agent’s state visitations (blue points) collapse into a highly localized cluster on the target manifold U\mathcal{U} rather than continuously exploring the frontier.
Figure 4: Balancing target manifold coverage with a primary navigation task. (a) Our method induces a time-splitting strategy, actively circulating the manifold before navigating to the goal. (b) The baseline intrinsic reward fails to achieve diverse boundary coverage, moving almost directly to the goal.
Figure 5: For higher dimensions ( d>2d>2 ), our reward function in Eq. ( 2 ) generates different tangential vector fields for different choices of the skew-symmetric matrix WW . Because rotational flow is not unique in these spaces, each arbitrary WW dictates a distinct rotational trajectory along the boundary U\mathcal{U} .
CategoryHyperparameterValue
OptimizationActor Learning Rate1×10−61\times 10^{-6}
Critic Learning Rate5×10−45\times 10^{-4}
Alpha Learning Rate2×10−32\times 10^{-3}
Discount Factor ( γ\gamma )0.990.99
Soft Target Update Rate ( τ\tau )10−410^{-4}
Max Gradient Norm30.030.0
Table 1: SAC and environment hyperparameters used in our experiments. All models were trained with a replay buffer capacity of 10610^{6} transitions.
More in this topic

RL Exploration

Reinforcement learning exploration studies how agents choose informative actions and states to learn effective policies under uncertainty.

See all 220 papers

Explore similar work

CardsList

  1. SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration

    Jun 8, 2026Kaustubh Mani, Yann Pequignot, Vincent Mai +1Sharpness-Aware MinimizationRL Exploration

    Universit´e de Montr´eal · Mila - Qu´ebec AI Institute · Universit´e Laval +2

  2. A Graph-Based Reinforcement Learning Approach with Frontier Potential Based Reward for Safe Cluttered Environment Exploration

    Apr 16, 2025Gabriele Calzolari, Vidya Sumathy, Christoforos Kanellakis +1Reward ShapingMobile Robotics

    Luleå University of Technology · Robotics and AI Group, Department of Computer Science, Electrical and Space Engineering, Luleåa University of Technology, Sweden

  3. CAPSULE: Control-Theoretic Action Perturbations for Safe Uncertainty-Aware Reinforcement Learning

    Apr 26, 2026Rahul Narava, Siddharth Verma, Ojas Jain +2Control Barrier FunctionsConstrained RL

    Indian Institute of Technology Ropar · CRAN, Université de Lorraine