Learning Goal-Reaching Quasimetric Geometry From Finite-Time Reachability

Daisuke Yamada; Travis Pence; Vikas Singh

cs.LGSep 30, 2026

Learning Goal-Reaching Quasimetric Geometry From Finite-Time Reachability

Authors: Daisuke Yamada, Travis Pence, Vikas Singh

Organizations: Department of Computer Science University of Wisconsin–Madison · Department of Biostatistics and Medical Informatics University of Wisconsin–Madison

Abstract

In goal-conditioned reinforcement learning (GCRL), quasimetric learning models goal-reaching costs as quasimetric distances, connecting local constraints to global value geometry. Its local constraints, however, should reflect the direction- dependent effects of control composition over a finite horizon together with environmental feasibility. We propose ReQRL, which constrains the critic's value gradients through finite-horizon reachability. Drawing on state-constrained optimal control, we decouple dynamical reachability from boundary geometry, estimating both from data. On OGBench, our method outperforms or rivals existing quasimetric approaches and other offline GCRL methods.

Figures & tables

Figure 1: ReQRL’s value geometry (top) and Euclidean (bottom). Colors for distance; dark regions denote obstacles, stars the goal.

Figure 2: Composed controls produce sideways motion (a), with net state change outside the initial instantaneous span

\mathcal{H}_{0}

(b).

Figure 3: Toy illustration of ( 7 ), with

d_{\theta}\approx d^{*}

in value and gradient and

d^{*}

affine along

[s,s+\delta^{*}]

. The orange line and black cost contour approximately touch the green reachable set at

s+\delta^{*}

. Contours are spaced by

\tau

Figure 4: Evaluation curves on humanoidmaze for navigate and stitch datasets of all three sizes. The success rate with std shading is plotted every

10^{5}

training steps up to

10^{6}

Env	Data	Size	Other GCRL						Quasimetric
Env	Data	Size	CRL	SAW	GCIQL	HIQL	OTA	GAS	QRL	EikHiQRL	ReQRL
pointmaze	nav	medium	$65\pm 10$	$99\pm 2$	$57\pm 1$	$92\pm 3$	$94\pm 3$	$75\pm 12$	$79\pm 9$	$82\pm 9$	$94\pm 7$
		large	$50\pm 12$	$60\pm 14$	$33\pm 2$	$49\pm 12$	$97\pm 3$	$72\pm 9$	$88\pm 9$	$83\pm 13$	$98\pm 3$
		giant	$26\pm 10$	$12\pm 14$	$0\pm 0$	$6\pm 7$	$83\pm 11$	$86\pm 8$	$73\pm 7$	$71\pm 18$	$91\pm 8$
	sti	medium	$2\pm 3$	$70\pm 14$	$30\pm 5$	$68\pm 12$	$81\pm 7$	$85\pm 11$	$80\pm 8$	$87\pm 12$	$93\pm 10$
		large	$0\pm 0$	$12\pm 10$	$27\pm 7$	$12\pm 8$	$37\pm 14$	$87\pm 13$	$82\pm 8$	$81\pm 11$	$96\pm 8$

Table 1: Success rate % (

\uparrow

) , percentage of episodes in which agent reaches goal

g

from the starting point

s

(mean

\pm

std over 10 seeds of the final evaluation). The agent is tested for 50 episodes per goal. We bold the results within

95\%

of the best and underline the best.

Env	Data	Size	Other GCRL		Quasimetric
Env	Data	Size	OTA	GAS	QRL	EikHiQRL	ReQRL
antmaze	nav	medium	$3\pm 0$	$2\pm 0$	$5\pm 1$	$3\pm 0$	$1\pm 0$
		large	$3\pm 0$	$3\pm 1$	$7\pm 2$	$5\pm 0$	$2\pm 0$
		giant	$9\pm 1$	$8\pm 3$	$14\pm 1$	$6\pm 1$	$4\pm 1$
	sti	medium	$3\pm 0$	$2\pm 0$	$6\pm 1$	$4\pm 0$	$1\pm 0$
		large	$8\pm 1$	$3\pm 1$	$10\pm 1$	$5\pm 1$	$1\pm 0$

Table 2: Collision rate (%) (

\downarrow

): percentage of time steps with agent-boundary contact in MuJoCo (mean

\pm

std over 10 seeds). Full results appear in Appendix D .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	$n$	$m$	Training transitions per dataset
pointmaze	2	2	1M ( navigate , stitch )
antmaze	29	8	1M ( navigate , stitch ); 5M ( explore )
humanoidmaze	69	21	2M ( medium , large ); 4M ( giant )

Appendix

Table 4: Observation/action dimensions and training dataset sizes. M denotes one million transitions; counts exclude terminal-only observations.

n

and

m

denote the full state and action spaces, respectively. The humanoidmaze counts apply to both dataset types.

Panel of Figure 6 — Figure 6: The pointmaze , antmaze , and humanoidmaze agents of OGBench .

Environment	Dataset	$w_{\mathrm{in}}$	$w_{b}$
pointmaze	navigate	$(0.5,1,1)\times 10^{2}$	$(1,1,5)\times 10^{1}$
	stitch	$(2.5,2,2)\times 10^{2}$	$(0.5,1,1.5)\times 10^{1}$
antmaze	navigate	$(3,3,3)$	$(1,1,3)\times 10^{2}$
	stitch	$(2.5,2.5,5)\times 10^{-1}$	$(1,1,0.5)\times 10^{2}$
	explore	$(2.5,2.5)\times 10^{-1}$	$(1,0.5)\times 10^{2}$
humanoidmaze	navigate	$(5,5,5)$	$(3,3,5)\times 10^{2}$

Appendix

Table 5: Selected penalty weights

w_{\mathrm{in}}

and

w_{b}

, ordered as ( medium , large , giant ). For explore , entries correspond to ( medium , large ).

Env	Data	Size	Other GCRL						Quasimetric
Env	Data	Size	CRL	SAW	GCIQL	HIQL	OTA	GAS	QRL	EikHiQRL	ReQRL
carmaze	nav	medium	$50\pm 5$	$52\pm 7$	$28\pm 4$	$53\pm 5$	$63\pm 7$	$81\pm 5$	$36\pm 6$	$72\pm 8$	$90\pm 3$
		large	$13\pm 4$	$2\pm 2$	$4\pm 1$	$19\pm 8$	$32\pm 14$	$67\pm 7$	$4\pm 4$	$45\pm 15$	$81\pm 12$
		giant	$0\pm 0$	$0\pm 0$	$0\pm 0$	$1\pm 1$	$3\pm 3$	$40\pm 8$	$1\pm 1$	$13\pm 9$	$60\pm 14$
	sti	medium	$28\pm 5$	$27\pm 11$	$10\pm 2$	$23\pm 7$	$60\pm 6$	$74\pm 6$	$30\pm 6$	$69\pm 9$	$82\pm 6$
		large	$0\pm 0$	$1\pm 1$	$0\pm 1$	$0\pm 1$	$14\pm 6$	$64\pm 10$	$1\pm 1$	$26\pm 9$	$78\pm 12$

Appendix

Table 6: Success rates (%) on carmaze at the final checkpoint (mean

\pm

std over 10 seeds; 50 episodes per goal). nav and sti denote navigate and stitch . Means at least

95\%

of the best are bold ; the best is underlined .

Env	Data	Size	Other GCRL						Quasimetric
Env	Data	Size	CRL	SAW	GCIQL	HIQL	OTA	GAS	QRL	EikHiQRL	ReQRL
pointmaze	nav	medium	$34\pm 5$	$1\pm 1$	$56\pm 2$	$3\pm 2$	$2\pm 1$	$1\pm 1$	$33\pm 7$	$2\pm 4$	$7\pm 6$
		large	$49\pm 9$	$27\pm 12$	$82\pm 2$	$32\pm 7$	$6\pm 2$	$0\pm 0$	$40\pm 7$	$0\pm 1$	$5\pm 4$
		giant	$58\pm 8$	$50\pm 13$	$93\pm 7$	$59\pm 12$	$14\pm 5$	$2\pm 2$	$43\pm 9$	$9\pm 1$	$10\pm 7$
	sti	medium	$72\pm 12$	$31\pm 8$	$78\pm 3$	$33\pm 8$	$13\pm 3$	$6\pm 6$	$32\pm 7$	$2\pm 4$	$9\pm 7$
		large	$65\pm 13$	$55\pm 8$	$71\pm 3$	$68\pm 10$	$20\pm 10$	$3\pm 5$	$37\pm 4$	$1\pm 1$	$7\pm 5$

Appendix

Table 7: Collision rate (%) (

\downarrow

): percentage of time steps with agent-boundary contact in MuJoCo (mean

\pm

std over 10 seeds). nav , sti , exp are shorthands for navigate , stitch , explore datasets of OGBench , respectively.

Env	Size	Other GCRL		Quasimetric
Env	Size	CRL	HIQL	QRL	EikHiQRL	ReQRL
antsoccer-nav	arena	$24\pm 2$	$60\pm 4$	$10\pm 3$	$61\pm 5$	$65\pm 3$
antsoccer-nav	medium	$4\pm 2$	$13\pm 3$	$2\pm 2$	$13\pm 1$	$17\pm 3$

Appendix

Table 8: Success rates (%) on OGBench ’s antsoccer-navigate after

10^{6}

training updates (mean

\pm

std). For reference, baseline results are taken from Giammarino & Qureshi (2026) under the same training budget.

Figure 7: Success rate averaged across maze sizes versus training time on humanoidmaze navigate and stitch . Wall-clock time covers

10^{6}

training updates on one A100 using compiled PyTorch implementations. Circles denote quasimetric methods; triangles denote other GCRL baselines.

Explore similar work

Beyond Success Rates: Trainability and Extractability for Offline GCRL
Feb 5, 2026Jan Malte Töpperwien, Aditya Mohan, Marius LindauerGoal-Conditioned Reinforcement Learning Task Success Rate
Institute of Artificial Intelligence (LUH|AI), Leibniz University Hannover · L3S Research Center
Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
May 27, 2026Kaiqiang Ke, Shenghong He, Chengdong Xu +3Goal-Conditioned Reinforcement Learning Long-Horizon Task Planning
1Sun Yat-sen University · 2Pengcheng Laboratory
Do Better Goal Representations Improve Goal-Conditioned Reinforcement Learning?
Sep 30, 2026Syed Nazmus Sakib, Abdul Monaf Chowdhury, Nafiul Haque +2Goal-Conditioned Reinforcement Learning Object Goal Navigation
Department of Robotics and Mechatronics Engineering, University of Dhaka, Bangladesh · Department of Computer Science, University of Oxford, UK

Feb 5, 2026, cs.LGJ/K move · Enter open · S save

Beyond Success Rates: Trainability and Extractability for Offline GCRL

Jan Malte Töpperwien, Aditya Mohan, Marius Lindauer

Institute of Artificial Intelligence (LUH|AI), Leibniz University Hannover · L3S Research Center

Goal-Conditioned Reinforcement Learning Task Success Rate Transitions Landscapes

Open paper AskPDF arXiv

Offline goal-conditioned reinforcement learning (GCRL) is typically benchmarked by the best tuned success rate of each method. This score measures attainable performance, but it does not reveal how reliably a learned goal-conditioned signal can be extracted into a policy: a method could succeed across many value-learning and extraction settings, or only at a narrow, hard-to-find configuration. We study this gap across four methods, GCIQL, GCIVL, QRL, and CRL, under a shared advantage-weighted regression (AWR) extractor. For each method, we construct trainability landscapes over the optimizer learning rate, which affects value learning and actor optimization, and AWR temperature, which controls how selectively the actor imitates high-advantage transitions. Across AntMaze, Cube, and Scene, we observe distinct regimes: high-scoring methods may be broadly accessible or brittle, while broad relative basins may still sit below low absolute ceilings. To interpret these differences, we pair landscapes with post-hoc diagnostics of future-vs-random goal discrimination and AWR weight concentration. Their relationship to downstream success is task-dependent. On AntMaze, where future goals align with path-like progress, these diagnostics explain landscape regimes. On Cube and Scene, goal ranking and manipulation control decouple: methods can rank goals well while failing downstream, or succeed through action-conditioned advantages despite weak future-vs-random separation. These results show that peak tuned success alone does not establish broadly extractable goal-conditioned behavior. Trainability landscapes expose this gap, while extraction diagnostics offer a lower-cost lens on how learned signals become policies.

Abstract

Explore similar work

Beyond Success Rates: Trainability and Extractability for Offline GCRL

Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Do Better Goal Representations Improve Goal-Conditioned Reinforcement Learning?