Mollified Value Learning

Hrishikesh Viswanath; Juanwu Lu; S. Talha Bukhari; Mihir Chauhan; Damon Conover; Ziran Wang; Aniket Bera

cs.LGFeb 26, 2026

Mollified Value Learning

Authors: Hrishikesh Viswanath, Juanwu Lu, S. Talha Bukhari, Mihir Chauhan, Damon Conover, Ziran Wang, Aniket Bera

Organizations: Department of Computer Science, Purdue University, USA · College of Engineering, Purdue University, USA · DEVCOM Army Research Laboratory, USA

Abstract

Offline goal-conditioned reinforcement learning (GCRL) learns goal-reaching behaviors from static datasets, but accurate value estimation remains challenging under limited state-action coverage. Existing physics-informed approaches address this by imposing pointwise distance-like geometric constraints derived from Hamilton--Jacobi--Bellman (HJB) optimality principles, often through first-order partial differential equations such as the Eikonal equation. However, enforcing local consistency through explicit differential structure can become unstable in complex, high-dimensional environments. Our key insight is to instead reinterpret distance-like constraints as an expectation over a local spatial measure. By aggregating constraints over this measure rather than evaluating them pointwise, the objective acts as a spatial mollifier, inducing distance-like value geometry without requiring expensive differential operators. We refer to this as Mollified Value Learning (MVL). Experiments across navigation and high-dimensional robotic manipulation tasks show that MVL learns structured, value representations, improving goal-reaching performance, when used with implicit value representation learning methods. Open-source codes are available at https://github.com/HrishikeshVish/MVL.

Figures & tables

Panel of Figure 1 — Figure 1: Contrasting Manipulation Tasks. By imposing our proposed formulation on the value function for offline goal-conditioned reinforcement learning , the policy successfully executes complex manipulation tasks where the Eikonal baseline [ 17 ] struggles. As shown (Bottom), our approach yields better trajectories across various manipulation tasks.

Figure 2: Toy example with a single obstacle (blue square) in an arena, with the goal state at the bottom : We solve the exact Eikonal form from [ 17 ] and our integral form on a toy problem.

ENVIRONMENT	ORIG	ORIG -MVL	VIB	VIB -MVL	VIP	TRA	BYOL	DUAL	DUAL -MVL
pointmaze-medium	78 $\pm$ 8	70 $\pm$ 2	69 $\pm$ 13	92 $\pm$ 6	0 $\pm$ 1	3 $\pm$ 6	37 $\pm$ 7	76 $\pm$ 7	96 $\pm$ 3
pointmaze-large	52 $\pm$ 6	48 $\pm$ 4	50 $\pm$ 7	80 $\pm$ 3	0 $\pm$ 0	1 $\pm$ 2	22 $\pm$ 12	46 $\pm$ 6	77 $\pm$ 1
antmaze-medium	71 $\pm$ 4	61 $\pm$ 5	68 $\pm$ 4	75 $\pm$ 4	31 $\pm$ 5	22 $\pm$ 15	39 $\pm$ 5	75 $\pm$ 4	76 $\pm$ 5
antmaze-large	16 $\pm$ 3	23 $\pm$ 2	9 $\pm$ 3	26 $\pm$ 2	9 $\pm$ 2	22 $\pm$ 12	11 $\pm$ 5	28 $\pm$ 11	39 $\pm$ 3
humanoid-medium	27 $\pm$ 3	27 $\pm$ 3	24 $\pm$ 2	30 $\pm$ 3	7 $\pm$ 3	21 $\pm$ 3	18 $\pm$ 5	29 $\pm$ 3	34 $\pm$ 3
humanoid-large	3 $\pm$ 0	4 $\pm$ 1	3 $\pm$ 1	4 $\pm$ 1	1 $\pm$ 0	2 $\pm$ 1	2 $\pm$ 1	3 $\pm$ 2	7 $\pm$ 2

Table 1: RESULTS ON STATE-BASED TASKS WITH GCIVL. Success rates across 4 seeds to highlight the impact of MVL . Soft teal highlights methods within 5% of the best performance, while soft purple highlights an MVL variant when it strictly outperforms its corresponding base representation but falls outside the top 5%.

Panel of Figure 3 — Figure 3: Qualitative value contour ablation on PointMaze-Large. Col 1 (Original): GCIVL Baseline ( 3(a) ) contours aren’t aligned towards goal (bottom-left corner); ours ( 3(e) ) with aligned contours. Col 2 (Dual): Baseline ( 3(c) ) exhibits severe jitter; ours ( 3(g) ) with improved geometry. Col 3 (VIB): Baseline ( 3(b) ); ours ( 3(f) ) enforces value geometry. Col 4 (Eikonal): Eikonal constraints ( 3(h) , 3(d) )

ENVIRONMENT	EIK-HIQL	HIQL+MVL	DUAL+EIK	DUAL+MVL
PointMaze
point-nav-medium	93 $\pm$ 5	95 $\pm$ 3	92 $\pm$ 1	96 $\pm$ 3
point-nav-large	83 $\pm$ 9	89 $\pm$ 5	79 $\pm$ 4	77 $\pm$ 1
point-nav-giant	79 $\pm$ 13	91 $\pm$ 4	73 $\pm$ 3	57 $\pm$ 7
point-nav-teleport	47 $\pm$ 10	45 $\pm$ 10	44 $\pm$ 2	50 $\pm$ 3
point-stitch-medium	96 $\pm$ 3	95 $\pm$ 2	93 $\pm$ 3	93 $\pm$ 2

Table 2: ABLATION EXPERIMENTS. Eik-HIQL applies Eikonal constraint to HIQL policy, DUAL+EIK enforces the first-order Eikonal constraint (

|\nabla V|\approx 1

Table 3: ROBUSTNESS AND SENSITIVITY ABLATIONS. (a) Performance on stochastic environments. (b) Ablations ( pointmaze-large ) across varying sample sizes (

N

) and radius (

\delta

). Additional envs in Table 8

Figure 4: Real-world demonstrations Top In the Reach and Grasp task, without regularization, the arm fails to position itself to grab the apple and instead drags along the table. Bottom In Pick and Place, regularization helps position correctly before releasing the gripper.

Panel of Figure 5 — Figure 5: MVL deployed on Franka Emika Panda. We use Viser to visualize the execution trajectories of our policy, represented as soft teal waypoints (Purple represents base dual policy). This visualizes the routing of the end-effector toward the goal in Reach and Grasp (left) and the more complex Pick and Place (right) task, as shown in Figure 4

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Notations	Explanation
Goal-conditional Reinforcement Learning
${\mathcal{M}}$	A Markov decision process (MDP).
${\mathcal{S}}$	State space of a Markov decision process.
${\mathcal{A}}$	Action space of a Markov decision process.
${\bm{s}}_{t}$	State at time step $t$ .
${\bm{s}}^{\prime}$	Generic notation for next state given current state ${\bm{s}}$ .

Appendix

Table 4: Table of notations in the paper.

Panel of Figure 6 — Figure 6: Benchmark Suite from OGBench We evaluate our method across a wide spectrum of offline GCRL tasks, ranging from standard geometric navigation ( point/ant/humanoid-maze ) and high-dimensional manipulation ( scene-play, puzzle ), to the highly stochastic, pixel-based physics of powderworld . This diversity tests the agent’s ability to handle varying degrees of state dimensionality, transition stochasticity, and dynamic complexity.

Panel of Figure 7 — Figure 7: Comparison of action probability distributions (pink fans) along a trajectory. (a) Ours: The advantage distributions align parallel to the walls, correctly identifying the safe path through corridors. (b) Eikonal: The baseline shows similar geometry trends, highlighting that our approach models the geometry without relying on expensive and unstable first-order autograd operations.

Hyperparameter	Value
Gradient steps	$10^{6}$ (state-based), $5\times 10^{5}$ (pixel-based)
Optimizer	Adam
Learning rate	$0.0003$
Batch size	$1024$ (state-based), $256$ (pixel-based)
MLP size	$[512,512,512]$
Nonlinearity	GELU

Appendix

Table 5: Hyperparameters for experiments.

ENVIRONMENT DOMAIN	Discount ( $\gamma$ )	GCIVL $\alpha$	CRL $\alpha$	Rep Dim	DUAL $\tau$	VIB $\beta$
pointmaze (medium/large)	0.99	10.0	0.03	64	0.9	0.003
antmaze-medium	0.99	10.0	0.1	256	0.9	0.001
antmaze-large	0.99	10.0	0.1	256	0.9	0.003
humanoidmaze (medium/large)	0.995	10.0	0.1	256	0.9	0.001
antsoccer-arena	0.99	10.0	0.3	256	0.9	0.001
manip/play (cube, scene, puzzle, D4RL)	0.99	10.0	3.0	256	0.7	0.001

Appendix

Table 6: HYPERPARAMETERS SUMMARY. Base algorithmic and representation-specific hyperparameters across all evaluated environments. All Dual models utilize a bilinear representation type. Visual environments additionally apply an impala_small encoder, a batch size of 256,

p_{\text{aug}}=0.5

, and are trained for 500K steps with 15 evaluation episodes.

ENVIRONMENT	GCIVL-PIXEL	GCIVL-PIXEL-EIK	GCIVL-PIXEL+MVL (OURS)
powderworld-easy	99 $\pm$ 1	99 $\pm$ 1	99 $\pm$ 1
powderworld-medium	50 $\pm$ 4	61 $\pm$ 4	62 $\pm$ 7
powderworld-hard	4 $\pm$ 3	7 $\pm$ 3	10 $\pm$ 2

Appendix

Table 7: HIGH-DIMENSIONAL STOCHASTIC DYNAMICS (POWDERWORLD). Overall results on pixel-based tasks where environment dynamics are highly stochastic (e.g., sand piling, fire spreading). Since raw pixel perturbation is infeasible, GCIVL-PIXEL+MVL enforces the viscous consistency condition via sampling in the Impala encoder’s latent embedding space.

Varying Number of Sampled States ( $N$ )
ENVIRONMENT	$\mathbf{N=1}$	$\mathbf{N=5}$	$\mathbf{N=10}$	$\mathbf{N=20}$	$\mathbf{N=25}$
pointmaze-large	57 $\pm$ 3	77 $\pm$ 2	76 $\pm$ 2	77 $\pm$ 2	77 $\pm$ 3
antmaze-large	28 $\pm$ 6	33 $\pm$ 7	39 $\pm$ 11	37 $\pm$ 6	39 $\pm$ 9
scene-play	75 $\pm$ 4	78 $\pm$ 5	84 $\pm$ 5	82 $\pm$ 4	83 $\pm$ 6
puzzle-4x4	26 $\pm$ 3	34 $\pm$ 4	33 $\pm$ 5	34 $\pm$ 2	34 $\pm$ 3
Varying Spatial Variance ( $\delta$ )

Appendix

Table 8: ABLATION STUDY ON DUAL-MVL . We evaluate the success rates across 4 seeds, ablating over the number of sampled states

N

and the spatial variance

\delta

across four environments with the largest gains in performance.

ENVIRONMENT/TASK	Orig	Orig-MVL	VIB	VIB-MVL	VIP	TRA	BYOL	DUAL	DUAL-MVL
D4RL/Kitchen-Partial
slide_cabinet	75 $\pm$ 0	58 $\pm$ 12	58 $\pm$ 12	75 $\pm$ 20	50 $\pm$ 0	50 $\pm$ 0	50 $\pm$ 0	75 $\pm$ 0	50 $\pm$ 0
microwave	92 $\pm$ 12	92 $\pm$ 12	92 $\pm$ 12	100 $\pm$ 0	75 $\pm$ 0	100 $\pm$ 0	75 $\pm$ 0	100 $\pm$ 0	100 $\pm$ 0
light_switch	83 $\pm$ 24	75 $\pm$ 20	75 $\pm$ 20	83 $\pm$ 12	50 $\pm$ 0	50 $\pm$ 0	50 $\pm$ 0	67 $\pm$ 12	69 $\pm$ 21
kettle	75 $\pm$ 0	100 $\pm$ 0	75 $\pm$ 0	83 $\pm$ 12	75 $\pm$ 0	88 $\pm$ 12	75 $\pm$ 0	83 $\pm$ 12	94 $\pm$ 11
D4RL/Kitchen-Mixed

Appendix

Table 9: RESULTS ON D4RL/KITCHEN VARIANTS We compare the success rates across multiple tasks and environments. Soft teal highlights methods within 5% of the best performance.

ENVIRONMENT/TASK	BC	FBC	CRL	QRL	TDP	COE	TRL	IVL	MVL
pointmaze-large-navigate-oraclerep-v0
task1	53 $\pm$ 24	98 $\pm$ 3	23 $\pm$ 18	31 $\pm$ 28	40 $\pm$ 19	67 $\pm$ 26	52 $\pm$ 19	94 $\pm$ 6	100 $\pm$ 0
task2	3 $\pm$ 4	17 $\pm$ 6	42 $\pm$ 28	5 $\pm$ 6	0 $\pm$ 0	0 $\pm$ 0	1 $\pm$ 2	0 $\pm$ 0	27 $\pm$ 20
task3	17 $\pm$ 9	76 $\pm$ 11	67 $\pm$ 14	0 $\pm$ 0	4 $\pm$ 4	12 $\pm$ 8	7 $\pm$ 5	100 $\pm$ 0	100 $\pm$ 0
task4	23 $\pm$ 10	79 $\pm$ 8	13 $\pm$ 18	1 $\pm$ 2	56 $\pm$ 12	44 $\pm$ 27	41 $\pm$ 13	3 $\pm$ 5	91 $\pm$ 9
task5	30 $\pm$ 22	87 $\pm$ 5	21 $\pm$ 8	0 $\pm$ 0	49 $\pm$ 16	48 $\pm$ 16	64 $\pm$ 14	44 $\pm$ 39	76 $\pm$ 12

Appendix

Table 10: RESULTS ON ORACLEREP VARIANTS We compare the success rates across multiple tasks and environments. In oraclerep setting, where the goal is denoted by an “Oracle Representation”, which is a subset of state representations. These represent the barebones information required to fulfil the success criteria. Soft teal highlights methods within 5% of the best performance, while soft purple highlights MVL when it improves upon IVL by at least 5%.

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 298 papers

Apr 19, 2026

SVL: Goal-Conditioned Reinforcement Learning as Survival Learning

Standard approaches to goal-conditioned reinforcement learning (GCRL) that rely on temporal-difference learning can be unstable and sample-inefficient due to bootstrapping. While recent work has explored contrastive and supervised formulations to improve stability, we present a probabilistic alternative, called survival value learning (SVL), that reframes GCRL as a survival learning problem by modeling the time-to-goal from each state as a probability distribution. This structured distributional Monte Carlo perspective yields a closed-form identity that expresses the goal-conditioned value function as a discounted sum of survival probabilities, enabling value estimation via a hazard model trained via maximum likelihood on both event and right-censored trajectories. We introduce three practical value estimators, including finite-horizon truncation and two binned infinite-horizon approximations to capture long-horizon objectives. Experiments on offline GCRL benchmarks show that SVL combined with hierarchical actors matches or surpasses strong hierarchical TD and Monte Carlo baselines, excelling on complex, long-horizon tasks. Webpage and Code: https://simple-robotics.github.io/publications/survival-value-learning/

Franki Nguimatsia Tiofack, Fabian Schramm, Théotime Le Hellard +1

Inria and École Normale Supérieure, PSL Research University, Paris, France.

Reinforcement Learning Goal-Conditioned RL

Ask PDF

May 25, 2026

Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning

Offline goal-conditioned reinforcement learning (GCRL) provides a practical framework for obtaining goal-reaching policies from fixed datasets. However, learning a reliable goal-conditioned value function in long-horizon tasks remains challenging. In this paper, we identify erroneous generalization in goal-conditioned value functions as a fundamental bottleneck, and demonstrate that appropriate inductive bias in the value function is crucial for addressing the bottleneck. Building on these findings, we propose Latent-Aligned Value Learning (LAVL), an offline GCRL algorithm that integrates latent-representation-based value generalization with hierarchical planning in a unified framework. Extensive experiments on OGBench demonstrate that LAVL consistently outperforms existing offline GCRL methods, achieving the highest performance on 20 out of 22 datasets. Notably, LAVL exhibits strong performance in long-horizon tasks and trajectory stitching datasets, where prior methods suffer significant performance degradation. Our code is available at https://github.com/oh-lab/LAVL.git.

Hyungkyu Kang, Byeongchan Kim, Min-hwan Oh

Seoul National University, Seoul, South Korea.

Reinforcement Learning Offline RL Representation Learning for RL Goal-Conditioned RL

Ask PDF

Sep 30, 2026

Do Better Goal Representations Improve Goal-Conditioned Reinforcement Learning?

Goal-conditioned reinforcement learning (GCRL) relies heavily on how target goals are represented to the policy. While recent methods encode goals via temporal distance, occupancy, or controllability, it remains unclear how much downstream performance actually depends on representation quality. We study this in offline GCRL by constructing an exact temporal-distance goal representation in deterministic mazes. We then systematically corrupt its geometric quality while keeping the downstream learner fixed. Across OGBench navigation tasks and two algorithms, large changes in goal-representation quality produce almost no change in performance. However, applying the same interventions to the agent's current state more than doubles success, revealing the state pathway as the true bottleneck. Building on this insight, we show that simple random Fourier positional encodings substantially improve performance on the hardest navigation tasks without map information or objective modifications. Overall, our findings suggest that in state-based offline navigation, improving how the agent's current state is represented matters far more than refining the goal representation. Code will be released soon.

Syed Nazmus Sakib, Abdul Monaf Chowdhury, Nafiul Haque +2

Department of Robotics and Mechatronics Engineering, University of Dhaka, Bangladesh · Department of Computer Science, University of Oxford, UK

Offline RL Representation Learning for RL State Representation Learning Goal-Conditioned RL

Ask PDF

Abstract

Offline RL

Explore similar work

SVL: Goal-Conditioned Reinforcement Learning as Survival Learning

Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning

Do Better Goal Representations Improve Goal-Conditioned Reinforcement Learning?