Direction-Conditioned Policies for Online Goal-Conditioned Reinforcement Learning

S K Swaminathan; Damiya Gondha; Theyanesh Eswaramoorthy Rajahkrishnan; Aritra Hazra

cs.LGOct 4, 2026

Direction-Conditioned Policies for Online Goal-Conditioned Reinforcement Learning

Authors: S K Swaminathan, Damiya Gondha, Theyanesh Eswaramoorthy Rajahkrishnan, Aritra Hazra

Organizations: Department of Computer Science and Engineering Indian Institute of Technology Kharagpur, India · Department of Mechanical Engineering Indian Institute of Technology Kharagpur, India

Abstract

Contrastive Reinforcement Learning (CRL) learns representations that estimate goal reachability, yet its policy remains conditioned on raw goals and therefore does not directly exploit the geometry encoded by its critic. We introduce Direction-Conditioned Policies (DCP), a method built around a small modification to CRL: DCP selects previously visited states as waypoints during online training and conditions the policy on their direction and distance in representation space. At deployment, DCP applies the same interface directly to the final goal, requiring neither waypoint selection nor planning. Across nine navigation and manipulation tasks, DCP attains higher final success rates than CRL on seven tasks and spends more time near the goal on seven. Controlled maze experiments further show that DCP captures shortest-path geometry more accurately and that the supplied direction causally influences the actor's behavior. We identify waypoint coverage and ranking as limits to exploration, and show that learned candidate generation improves goal reaching in two controlled mazes.

Figures & tables

Figure 1: Schematic of the policy inputs. CRL uses the raw goal, SSGC uses a selected waypoint, and DCP uses the waypoint’s direction and distance in representation space. At deployment, DCP replaces the waypoint with the final goal. Here $\psi(s)$ abbreviates the encoding of the state’s achieved-goal coordinates.

Stage	Target $y$	Policy input
Online collection	Scored visited waypoint $q(z^{*})$	$[s,d_{\theta}(s,y),r_{\theta}(s,y)]$
Actor update	Achieved future state $q(s_{t+\Delta})$	$[s,d_{\theta}(s,y),r_{\theta}(s,y)]$
Deployment	Requested final goal $g$	$[s,d_{\theta}(s,g),r_{\theta}(s,g)]$

Table 1: Targets used for collection, actor updates, and deployment. Each stage constructs the same direction-and-distance input from its target.

Figure 2: The nine benchmark tasks, rendered from their simulation models. The three Pusher tasks share the same arm and cylinder; their target distributions differ. Eight tasks follow JaxGCRL, while PusherHard Far changes PusherHard’s target distribution.

Figure 3: Online learning across all nine tasks. Each task shows time near goal (upper panel) and success at least once (lower panel). Lines are means over five seeds; shading denotes 95% Student-

t

confidence intervals. Budgets are matched across methods within each task. The comparison includes AntSoccer, where CRL outperforms DCP, and Ant U-Maze, where the final two metrics favor different methods.

	Success rate (%)			Time near goal
Task	CRL	SSGC	DCP	CRL	SSGC	DCP
AntMaze Big	$18.3\!\pm\!6.5$	$23.3\!\pm\!3.0$	$\mathbf{29.9}\!\pm\!2.9$	$89.2\!\pm\!28.0$	$99.8\!\pm\!9.0$	$\mathbf{115.3}\!\pm\!14.6$
AntMaze Hardest	$12.2\!\pm\!5.6$	$14.2\!\pm\!8.0$	$\mathbf{16.1}\!\pm\!2.3$	$48.1\!\pm\!23.7$	$\mathbf{66.0}\!\pm\!41.4$	$63.2\!\pm\!13.3$
Ant U-Maze	$40.8\!\pm\!10.2$	$49.8\!\pm\!4.2$	$\mathbf{55.0}\!\pm\!13.3$	$\mathbf{261.3}\!\pm\!77.2$	$252.7\!\pm\!52.1$	$238.1\!\pm\!76.2$
Humanoid U-Maze	$0.0\!\pm\!0.0$	$0.0\!\pm\!0.0$	$\mathbf{8.4}\!\pm\!9.6$	$0.0\!\pm\!0.0$	$0.0\!\pm\!0.0$	$\mathbf{3.6}\!\pm\!4.5$
AntPush	$45.6\!\pm\!24.1$	$\mathbf{51.4}\!\pm\!15.5$	$42.7\!\pm\!12.6$	$143.4\!\pm\!105.6$	$\mathbf{199.1}\!\pm\!93.3$	$181.7\!\pm\!95.6$

Table 2: Final evaluation across nine tasks, five seeds per method. Entries are means with reported 95% confidence-interval half-widths. Bold marks the largest mean in each task and metric. Time near goal is measured in steps out of 1,000.

Method	Collection target	Open room	U-maze
CRL	Final goal	$0.956\pm 0.037$	$0.846\pm 0.153$
Direct direction conditioning	Final goal	$0.974\pm 0.010$	$0.885\pm 0.059$
SSGC	Scored waypoint	$0.866\pm 0.036$	$0.851\pm 0.040$
DCP	Scored waypoint	$0.869\pm 0.033$	$0.854\pm 0.066$

Table 3: Separating collection targets from actor inputs. Mean success over 20 evaluation checkpoints, averaged over five seeds;

\pm

denotes a 95% Student-

t

confidence-interval half-width across seeds. Higher is better.

Figure 4: Distance to the starred goal in the controlled U-maze: graph shortest-path distance (left), and learned representation distances for CRL and DCP after 10M steps (middle, right; seed 42). Each panel is scaled by its own 98th-percentile distance. The color scale shows relative distance within each panel.

U-maze measure	CRL	DCP
Spearman correlation with graph distance	$0.471\pm 0.031$	$0.756\pm 0.041$
Spearman correlation with Euclidean distance	$0.456\pm 0.033$	$0.690\pm 0.034$
Graph-correct local descent (%)	$64.7\pm 10.1$	$70.9\pm 4.5$

Table 4: Representation geometry after 10M online steps. Means

\pm

sample standard deviations over five independently trained encoders. Distances are computed from the goal encoder; local descent follows the negative spatial gradient of the learned distance.

Figure 5: Changing only the direction supplied to a trained actor reduces success in navigation and puck pushing (left, middle). In matched puck-pushing cases, DCP also makes more useful contacts and goal progress than CRL (right). Bars show means and dots show the three training seeds. The puck task uses 128 quartets per seed, varying the robot’s side of approach while holding puck and goal fixed within each quartet.

Task	Visited pool	Generator	Generator + projection
Two-route maze	$0.457\pm 0.301$	$0.613\pm 0.202$	$0.738\pm 0.128$
Rare-gateway maze	$0.271\pm 0.150$	$0.520\pm 0.113$	$0.204\pm 0.291$

Table 5: Changing the candidate source in two controlled mazes. Mean success over training, averaged across five seeds.

\pm

denotes sample standard deviation across seeds. Projection maps each proposal to the nearest stored state.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Actor / critic hidden width	256
Hidden layers	2
Representation dimension	64
Actor, critic, and temperature learning rates	$3\times 10^{-4}$
Discount	0.99
Minibatch size	256 (Humanoid: 128)

Appendix

Table 6: Main settings. Humanoid U-Maze uses the stated smaller parallelism and batch size. Replay sizes retain the implementation’s buffer configuration units.

Tasks	CRL actor	DCP actor	Actor increase	Total increase
AntMaze Big / Hardest / U-Maze	78,096	94,224	20.65%	6.37%
Humanoid U-Maze	144,162	160,034	11.01%	4.15%
AntPush / AntSoccer	78,608	94,736	20.52%	6.35%
PusherEasy / Hard / Hard Far	75,534	91,406	21.01%	6.40%

Appendix

Table 7: Saved parameter counts for the nine benchmark tasks. Grouped tasks have identical architectures and counts. Percentage increases are relative to CRL.

Task	Goal coordinates	Radius	Training steps
AntMaze Big	Ant position	0.5	50M
AntMaze Hardest	Ant position	0.5	50M
Ant U-Maze	Ant position	0.5	50M
Humanoid U-Maze	Humanoid position	0.5	100M
AntPush	Ant position (movable obstacle)	0.5	100M
AntSoccer	Ball XY	0.5	50M

Appendix

Table 8: The nine evaluation tasks. Training steps are matched across methods within each task.

Experiment	Training and evaluation
Nine-task comparison	Five seeds per method and task; 50M or 100M steps.
Four-way collection/input comparison	Five seeds, 10M steps, 20 evaluations, separately in each layout.
Representation geometry	Five encoders per method and layout, trained for 10M steps.
Candidate-source comparison	Five seeds per candidate source and maze, 10M steps.
Waypoint substitution	Three policies; 20,000 paired probes per policy.
Differential-drive intervention	Three policies; 42 directed pairs and four headings per policy.

Appendix

Table 9: Number of seeds and evaluations in each experiment. The geometry experiment and four-way comparison use separately trained policies.

Figure 6: The controlled environments, rendered from their simulation models with floor textures hidden. The four point-mass environments differ in wall layout and start/goal distributions. The differential-drive robot adds heading-dependent motion. In puck pushing, the blue agent must contact the yellow puck and move it to the green target. Images show illustrative task configurations.

Pool intervention	Pre-motion	Successful path	Paired wins
Reserve 128 of 512 slots	32.39	38.17	3/3
Add 128 to 512 online slots	38.09	36.26	0/3

Appendix

Table 10: PusherHard demonstration-state experiments, three target seeds. Entries are mean evaluation success over training (%). Paired wins compare successful-path against pre-motion seeding. DCP without seeding scores 35.76% in these experiments.

Variant	Pretraining	Final success (%)	Mean training success (%)
Fresh joint DCP	0	25.4	16.72
Frozen CRL goal encoder	50M	22.3	11.60
Both CRL encoders frozen	50M	23.0	12.46

Appendix

Table 11: Frozen-representation transfer on AntMaze Big, seed 42. The transferred variants use 50M CRL pretraining steps and 25M actor-training steps; fresh DCP uses 25M steps in total.

Figure 7: U-maze representation measurements for all five seeds after 10M steps. Connected dots compare CRL and DCP with the same seed; black horizontal marks show means. DCP has higher graph-distance correlation in every seed, both over all pairs and over the longest-distance third. The local-descent comparison shows more variation across seeds.

Figure 8: AntMaze Big distance fields and example trajectories for a seed in which both DCP and CRL reach the goal. Contours show

\|\psi(x,y)-\psi(g)\|

; the blue circle and red star mark start and goal. Both fields reflect the corridor layout. Colors are scaled separately for each method.

Figure 9: A second AntMaze Big example (seed 1), in which DCP reaches the goal and CRL stalls. The color range in each panel is clipped to the 1st–99th percentiles of distances along that method’s trajectory. The field and trajectory illustrate one outcome; the main comparison averages all five seeds.

Panel of Figure 10 — Figure 10: Additional Pusher views. The white cylinder must be moved to the red target. Target placement changes the required approach and pushing direction. The near-target view is an example from PusherHard.

Figure 11: Waypoint selection with five fixed U-maze encoders. Connected points use the same encoder and candidate draws; black marks show means. A forward on-path candidate is available in 77.7% of draws, but the default inner-product rule selects one much less often. Ranking by direction agreement or latent slack improves this fixed-encoder test.

Figure 12: Candidate-source comparison with all five seeds shown. Lines connect runs with the same seed; black marks show means. The generator improves mean success during training in both layouts. Nearest-state projection has different effects in the two mazes and removes the mean benefit in the rare-gateway maze. Each value averages the 20 evaluation checkpoints.

Environment	Method	10M	25M	50M	75M	100M
AntMaze Big	DCP	18.4 $\pm$ 5.4	23.8 $\pm$ 3.3	29.9 $\pm$ 2.9	–	–
	CRL	15.1 $\pm$ 3.7	17.7 $\pm$ 5.3	18.3 $\pm$ 6.5	–	–
	SSGC	18.3 $\pm$ 1.2	22.0 $\pm$ 2.9	23.3 $\pm$ 3.0	–	–
AntMaze Hardest	DCP	9.5 $\pm$ 2.5	14.5 $\pm$ 2.1	16.1 $\pm$ 2.3	–	–
	CRL	5.8 $\pm$ 2.5	10.5 $\pm$ 3.8	12.2 $\pm$ 5.6	–	–
	SSGC	7.7 $\pm$ 1.6	13.2 $\pm$ 4.8	14.2 $\pm$ 8.0	–	–

Appendix

Table 12: Success at least once (%), mean

\pm

reported 95% confidence-interval half-width over five seeds. Bold marks the largest mean at each task/checkpoint.

Environment	Method	10M	25M	50M	75M	100M
AntMaze Big	DCP	78.7 $\pm$ 37.1	94.6 $\pm$ 26.2	115.3 $\pm$ 14.6	–	–
	CRL	75.3 $\pm$ 12.7	83.2 $\pm$ 29.4	89.2 $\pm$ 28.0	–	–
	SSGC	91.9 $\pm$ 19.1	94.6 $\pm$ 20.5	99.8 $\pm$ 9.0	–	–
AntMaze Hardest	DCP	30.3 $\pm$ 12.6	65.1 $\pm$ 12.9	63.2 $\pm$ 13.3	–	–
	CRL	28.4 $\pm$ 15.9	39.9 $\pm$ 24.0	48.1 $\pm$ 23.7	–	–
	SSGC	32.6 $\pm$ 8.1	59.0 $\pm$ 21.0	66.0 $\pm$ 41.4	–	–

Appendix

Table 13: Time near goal (steps per 1,000-step episode), mean

\pm

reported 95% confidence-interval half-width over five seeds.

Explore similar work

Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learning
Jun 15, 2026Swaminathan S K, Damiya Gondha, Theyanesh Eswaramoorthy Rajahkrishnan +1Goal-Conditioned Reinforcement Learning Goal-Conditioned Value Function
Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur, Kharagpur, India · Department of Mechanical Engineering, Indian Institute of Technology Kharagpur, Kharagpur, India.
Do Better Goal Representations Improve Goal-Conditioned Reinforcement Learning?
Sep 30, 2026Syed Nazmus Sakib, Abdul Monaf Chowdhury, Nafiul Haque +2Goal-Conditioned Reinforcement Learning Object Goal Navigation
Department of Robotics and Mechatronics Engineering, University of Dhaka, Bangladesh · Department of Computer Science, University of Oxford, UK
Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
Sep 28, 2026Hengrui Zhang, Yuhu Cheng, C. L. Philip Chen +1Goal-Conditioned Reinforcement Learning High-Level Subgoal Generation
School of Information and Control Engineering, China University of Mining and Technology · School of Computer Science and Engineering, South China University of Technology

Jun 15, 2026, cs.LGJ/K move · Enter open · S save

Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learning

Swaminathan S K, Damiya Gondha, Theyanesh Eswaramoorthy Rajahkrishnan, Aritra Hazra

Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur, Kharagpur, India · Department of Mechanical Engineering, Indian Institute of Technology Kharagpur, Kharagpur, India.

Goal-Conditioned Reinforcement Learning Goal-Conditioned Value Function High-Level Subgoal Generation

Open paper AskPDF arXiv

Hamilton-Jacobi-Bellman theory implies that the optimal goal-conditioned action depends on the goal only through the gradient of the goal-reaching distance at the current state, yet standard online GCRL still conditions the actor on the raw goal -- a signal that is geometrically uninformative when the goal is far from the data distribution. We propose Direction-Conditioned Policies (DCP), a fully online method that decomposes goal-reaching into two components sharing one InfoNCE representation

ψ

: a subgoal-scoring step that selects a visited state

z_t

aligned with the final goal

g

ψ_g

, and a direction-conditioned actor that consumes the unit direction

d_t

and magnitude

r_t

from

ψ(s_t)

ψ(z_t)

. The two components train jointly, factor cleanly at deployment (subgoal scoring is removed, while direction conditioning remains with

g

in place of

z_t

), and admit independent modification at the same

(d_t,r_t)

interface. We prove three results. First, direction sufficiency under HJB: the optimal action under control-affine dynamics depends on the goal only through the value gradient. Second, a quantitative bound showing that, under mild conditions on the learned representation and assuming the scoring rule returns an on-path

z_t

, the actor's conditioning input at training and at deployment coincide up to representation error and geodesic slack. Third, a controllable-subspace characterization of when directional conditioning fails. Across nine environments, DCP improves over Contrastive RL on most final metrics, with the largest gains on manipulation and obstacle-interaction tasks; a qualitative analysis of the learned

ψ

-distance landscape shows the contrastive representation behaves as an online quasimetric encoding environment topology, and the single failure case (AntSoccer) localizes to a learned-gradient pathology that the theory anticipates.

Abstract

Explore similar work

Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learning

Do Better Goal Representations Improve Goal-Conditioned Reinforcement Learning?

Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning