World-Model Policy Arbiter for Goal-Conditioned Reinforcement Learning

Junwei Quan; Evgenii Opryshko; Nicholas Rhinehart; Igor Gilitschenski

cs.LGOct 7, 2026

World-Model Policy Arbiter for Goal-Conditioned Reinforcement Learning

Authors: Junwei Quan, Evgenii Opryshko, Nicholas Rhinehart, Igor Gilitschenski

Organizations: University of Toronto · University of Toronto, Vector Institute

Abstract

Offline goal-conditioned reinforcement learning (GCRL) has produced a diverse set of goal-reaching algorithms, yet no single algorithm performs best across environments, goals, and even different phases of the same task. Rather than deploying only the best-performing policy, we ask whether a set of frozen goal-conditioned policies can be used collectively as a portfolio, deciding at every state which policy should act. Choosing a policy at each state is not straightforward. The policies' own value functions cannot be compared directly: they may use different scales, and some policies have no value function. We need to judge each policy by the states it is likely to reach, even though we can execute only one policy at a time. We also need to avoid switching so often that control becomes unstable. To address these challenges, we introduce World-Model Policy Arbiter (WMPA), a test-time framework that, given a set of frozen policies as input, rolls out each frozen policy in a learned state-space world model, evaluates the imagined futures with a shared goal-conditioned value function, and executes the highest-scoring policy for a short commitment interval before the next round of arbitration (policy selection). WMPA assumes access to a bank of frozen goal-conditioned policies and requires neither policy retraining nor privileged task-specific knowledge. Under the official OGBench evaluation protocol on 18 state-based datasets spanning maze navigation as well as cube, scene, and puzzle manipulation, WMPA improves the macro-average success rate from the 44% achieved by the best policy selected per dataset to 58%, with statistically significant gains on 12 datasets. These gains include +33 percentage points on cube-double-play and +36 percentage points on scene-play.

Figures & tables

Figure 1: WMPA composes policies within one episode. A cube-double-play evaluation episode. Top: the real state at four arbitration steps and at success, with the policy the world model selects there; the label under each frame describes the phase of the task. Bottom: the policy executed over the whole episode, one block per arbitration. The arbiter alternates between HIQL, which moves the free gripper to the next cube, and GCIQL, which grasps and places it.

Figure 2: WMPA overview. (1) We begin with a frozen portfolio of heterogeneous GCRL policies. (2) WMPA imagines the future induced by each policy through a shared learned world model. (3) A shared goal-conditioned value function scores the predicted futures on a common scale. (4) The highest-ranked policy is executed for

c

steps, after which the process repeats.

Family	Dataset	GCBC	GCIVL	GCIQL	QRL	CRL	HIQL	Best-Policy	WMPA
Maze	pointmaze-medium-navigate	$6\pm 4$	$74\pm 3$	$54\pm 16$	$\mathbf{76\pm 17}$	$35\pm 6$	$73\pm 8$	$\mathbf{76\pm 17}$	$\mathbf{79\pm 3}$
Maze	antmaze-large-navigate	$23\pm 3$	$19\pm 5$	$37\pm 8$	$78\pm 5$	$\mathbf{86\pm 3}$	$\mathbf{90\pm 3}$	$\mathbf{90\pm 3}$	$\mathbf{91\pm 3}$
Cube	cube-single-play	$6\pm 3$	$49\pm 4$	$71\pm 3$	$8\pm 4$	$16\pm 3$	$12\pm 6$	$71\pm 3$	$\mathbf{85\pm 3}$
	cube-single-noisy	$8\pm 3$	$71\pm 8$	$\mathbf{99\pm 1}$	$33\pm 10$	$40\pm 5$	$40\pm 10$	$\mathbf{99\pm 1}$	$\mathbf{100\pm 0}$
	cube-double-play	$2\pm 1$	$35\pm 6$	$36\pm 7$	$1\pm 1$	$9\pm 2$	$5\pm 2$	$36\pm 7$	$\mathbf{69\pm 3}$
	cube-double-noisy	$1\pm 1$	$14\pm 3$	$26\pm 9$	$3\pm 2$	$3\pm 2$	$2\pm 1$	$26\pm 9$	$\mathbf{60\pm 7}$

Table 1: Success rate (%) on OGBench under the official protocol (

5

goals

\times

50

episodes,

3

bank seeds). We report every bank policy, the best policy for each dataset which is the bank policy with the highest three-seed mean success on the reported episodes, and WMPA , which arbitrates over the same frozen policies with the

(k,c)

setting of its task family ( Appendix A ). Each entry is the mean success rate

\pm

the half-width of its

95\%

hierarchical-bootstrap interval (resampling bank seeds, then episodes), rounded to the nearest integer. Bold indicates values within

95\%

of the row maximum.

Figure 3: What makes arbitration work? Each point is a dataset. Horizontal axis: success added over the best policy by the per-episode oracle, which picks a successful bank policy per episode but never switches within it, so it bounds per-episode selection, not WMPA . Vertical axis: success added by WMPA . Above the dashed

y=x

line, the extra success must come from switching within an episode; WMPA is significantly above the oracle on 7 datasets and below it on 5 ( Appendix G ).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Family	datasets	value head	$k{=}c$	WM $H$	$\gamma$	$\kappa$	$\lambda_{\mathrm{smooth}}$
Maze	2	metric	1	10	0.999	0.9	10
Cube	6	metric	5	5	0.99	0.7	0
Scene	2	metric	10	5	0.998	0.7	10
Puzzle	8	direct	10	5	0.99	0.9	–

Appendix

Table 2: Settings per task family . The value head is the parameterization of the scorer applied to every candidate’s imagined states (metric: latent-distance value with separate encoders; direct: a single network on the state–goal pair, Section 3.2 ); the single interval

k{=}c

is the imagination horizon and commitment of WMPA and, identically, the re-draw interval of Random-Switch ; the remaining columns (

H

\gamma

\kappa

\lambda_{\mathrm{smooth}}

) are the training settings of the world model and its value head.

			success rate (%)			switches per episode			latency (ms per step)
Dataset	Best-Policy	$k$	$c{=}1$	$c{=}5$	$c{=}10$	$c{=}1$	$c{=}5$	$c{=}10$	$c{=}1$	$c{=}5$	$c{=}10$
cube-double-play	$36\pm 7$	$1$	$63\pm 4$			$147$			$3.9$
		$5$	$65\pm 4$	$69\pm 3$		$107$	$38$		$15.5$	$3.0$
		$10$	$64\pm 3$	$75\pm 3$	$73\pm 4$	$90$	$30$	$19$	$27.6$	$5.9$	$2.9$
scene-play	$52\pm 5$	$1$	$93\pm 3$			$135$			$4.1$
		$5$	$93\pm 2$	$92\pm 2$		$94$	$35$		$15.5$	$3.0$

Appendix

Table 3: Sensitivity to the imagination horizon

k

and the commitment

c

on three datasets with large gains (

k\geq c

). A row fixes

k

, a column fixes

c

, and the diagonal is

k=c

; the cell reported in Table 1 is underlined. Success is the mean

\pm

the half-width of the

95\%

interval, switches are policy changes per episode, and latency is the mean wall-clock time per environment step on the CPU evaluation nodes. A cell costs

18k/c

model calls per step, so only the diagonal keeps the budget of Table 1 .

Dataset	$c$	Best	Random-Switch	WMPA
pointmaze-medium-navigate	$1$	$76\pm 17$	$\mathbf{89\pm 13}$	$79\pm 3$
antmaze-large-navigate	$1$	$\mathbf{90\pm 3}$	$\mathbf{92\pm 2}$	$\mathbf{91\pm 3}$
cube-single-play	$5$	$71\pm 3$	$41\pm 4$	$\mathbf{85\pm 3}$
cube-single-noisy	$5$	$\mathbf{99\pm 1}$	$\mathbf{96\pm 2}$	$\mathbf{100\pm 0}$
cube-double-play	$5$	$36\pm 7$	$24\pm 4$	$\mathbf{69\pm 3}$
cube-double-noisy	$5$	$26\pm 9$	$27\pm 4$	$\mathbf{60\pm 7}$

Appendix

Table 4: Random-Switch versus WMPA at a matched commitment interval (the per-family value of

c

in Table 2 ). Best is the best bank policy on each dataset ( Table 1 ). In each row, entries that reach at least

95\%

of the maximum over the three columns are in bold.

		stall-restart, window $m$
Dataset	Best	$m{=}5$	$10$	$25$	$50$	WMPA
pointmaze-medium-navigate	$76\pm 17$	$\mathbf{99\pm 1}$	$\mathbf{95\pm 2}$	$\mathbf{94\pm 2}$	$\mathbf{96\pm 2}$	$79\pm 3$
antmaze-large-navigate	$\mathbf{90\pm 3}$	$51\pm 4$	$51\pm 4$	$51\pm 4$	$51\pm 4$	$\mathbf{91\pm 3}$
cube-single-play	$71\pm 3$	$24\pm 5$	$24\pm 5$	$24\pm 5$	$24\pm 5$	$\mathbf{85\pm 3}$
cube-single-noisy	$\mathbf{99\pm 1}$	$53\pm 5$	$52\pm 5$	$52\pm 5$	$52\pm 5$	$\mathbf{100\pm 0}$
cube-double-play	$36\pm 7$	$16\pm 4$	$15\pm 4$	$15\pm 4$	$15\pm 4$	$\mathbf{69\pm 3}$

Appendix

Table 5: Stall-restart control : a no-model controller that keeps the incumbent policy and switches to a random other policy only when the normalized displacement over the last

m

steps falls below

\varepsilon

; same bank, episodes, and protocol as Table 1 . Numbers within

95\%

of the row maximum are in bold.

Dataset	$(k,c)$ of WMPA	Best	every-step WM action search on Best	WMPA
pointmaze-medium-navigate	$(1,1)$	$76\pm 17$	$\mathbf{86\pm 9}$	$79\pm 3$
antmaze-large-navigate	$(1,1)$	$\mathbf{90\pm 3}$	$\mathbf{92\pm 3}$	$\mathbf{91\pm 3}$
cube-single-play	$(5,5)$	$71\pm 3$	$52\pm 4$	$\mathbf{85\pm 3}$
cube-single-noisy	$(5,5)$	$\mathbf{99\pm 1}$	$\mathbf{100\pm 0}$	$\mathbf{100\pm 0}$
cube-double-play	$(5,5)$	$36\pm 7$	$39\pm 4$	$\mathbf{69\pm 3}$
cube-double-noisy	$(5,5)$	$26\pm 9$	$49\pm 6$	$\mathbf{60\pm 7}$

Appendix

Table 6: WM action search vs. arbitration with the same horizon

k

and the same value function on every dataset. The action search applies sampling-based MPC on top of the best policy. At every environment step, the candidates are the deterministic action

\pi(s_{t},g)

of the policy and

N{-}1{=}5

Gaussian perturbations of this action (

\sigma{=}0.2

). Each candidate is imagined for

k

steps: only the first action differs across candidates, and the remaining

k{-}1

steps follow the deterministic output of the policy in closed loop on the imagined states, exactly as in the rollout of a candidate policy in WMPA . The candidates are scored as in WMPA , the first action of the best candidate is executed, and the search is repeated at the next step (

c{=}1

). The action search uses

NEk=18k

model steps per environment step, compared with

ME=18

for WMPA . The two budgets are equal on the mazes, whereas the budget of the action search is

5\times

larger on the cube family and

10\times

larger on the scene and puzzle families. On the puzzle family, both methods use the direct value. Selecting the best policy per dataset as a base for MPC serves as oracle. Numbers that are at least

95\%

of the row maximum are shown in bold.

Dataset	$(k,c)$	Best	WMPA	true-sim. branches	$\Delta$ (sim $-$ WMPA )
pointmaze-medium-navigate	$(1,1)$	$\mathbf{76\pm 17}$	$\mathbf{79\pm 3}$	$\mathbf{80\pm 3}$	+0 [-0, +1]
antmaze-large-navigate	$(1,1)$	$90\pm 3$	$\mathbf{91\pm 3}$	$\mathbf{95\pm 2}$	+4 [-0, +9]
cube-single-play	$(5,5)$	$71\pm 3$	$\mathbf{85\pm 3}$	$\mathbf{86\pm 3}$	+2 [-3, +7]
cube-single-noisy	$(5,5)$	$\mathbf{99\pm 1}$	$\mathbf{100\pm 0}$	$\mathbf{100\pm 0}$	+0 [+0, +1]
cube-double-play	$(5,5)$	$36\pm 7$	$\mathbf{69\pm 3}$	$\mathbf{72\pm 4}$	+3 [-1, +7]
cube-double-noisy	$(5,5)$	$26\pm 9$	$\mathbf{60\pm 7}$	$\mathbf{61\pm 4}$	+0 [-10, +9]

Appendix

Table 7: True-simulator branches inside WMPA at the selected cell on every dataset (same value head and commitment; simulator steps replace model calls).

\Delta

is the paired contrast simulator

-

WMPA ; on the puzzle family the branches are scored with the direct value at the reported cell. Numbers within

95\%

of the row maximum are shown in bold ( ∗ : interval excludes zero).

			WMPA (imagined states)		Q-select (no rollout)
Dataset	$(k,c)$	Best	metric value	direct value	$c{=}1$	$c{=}k$	$\Delta$ (direct $-$ Q-select, $c{=}k$ )
pointmaze-medium-navigate	$(1,1)$	$\mathbf{76\pm 17}$	$\mathbf{79\pm 3}$ †	$56\pm 8$	$53\pm 9$	$53\pm 9$	+3 [+1, +4]*
antmaze-large-navigate	$(1,1)$	$\mathbf{90\pm 3}$	$\mathbf{91\pm 3}$ †	$28\pm 4$	$26\pm 6$	$26\pm 6$	+2 [-2, +6]
cube-single-play	$(5,5)$	$71\pm 3$	$\mathbf{85\pm 3}$ †	$76\pm 4$	$67\pm 20$	$58\pm 23$	+18 [+2, +44]*
cube-single-noisy	$(5,5)$	$\mathbf{99\pm 1}$	$\mathbf{100\pm 0}$ †	$\mathbf{100\pm 0}$	$\mathbf{100\pm 0}$	$\mathbf{100\pm 0}$	+0 [-1, +1]
cube-double-play	$(5,5)$	$36\pm 7$	$\mathbf{69\pm 3}$ †	$62\pm 5$	$51\pm 5$	$62\pm 7$	-0 [-5, +5]

Appendix

Table 8: Comparison of value heads and selectors at a fixed cell on all the datasets. The world model, the bank, and the evaluation episodes are held fixed; only the scorer or the selector varies. WMPA scores imagined states with either the metric value or the direct value; † marks the head reported in Table 1 . Q-select uses the same twin-Q value function to rank the action proposed by each policy at the current state,

i^{*}=\arg\max_{i}\min_{j}Q_{\omega,j}(s,\pi_{i}(s,g),g)

, without any imagined states, re-selecting either at every step (

c{=}1

) or every

c{=}k

steps. The last column is the paired contrast between the two selectors that share the same underlying agent, direct-value WMPA minus Q-select at

c{=}k

, with its

95\%

interval: it isolates the contribution of scoring imagined successor states with the value function. Entries are the mean success rate

\pm

the half-width of the

95\%

hierarchical-bootstrap interval, and values within

95\%

of the row maximum are shown in bold ( ∗ : interval excludes zero).

Dataset	$(k,c)$	Best member	WMPA -own	WMPA -shared
pointmaze-medium-navigate	$(1,1)$	$\mathbf{74\pm 3}$	$54\pm 16$	$\mathbf{76\pm 6}$
antmaze-large-navigate	$(1,1)$	$\mathbf{90\pm 3}$	$19\pm 4$	$\mathbf{90\pm 3}$
cube-single-play	$(5,5)$	$71\pm 3$	$71\pm 3$	$\mathbf{81\pm 4}$
cube-single-noisy	$(5,5)$	$\mathbf{99\pm 1}$	$94\pm 4$	$\mathbf{100\pm 0}$
cube-double-play	$(5,5)$	$36\pm 7$	$37\pm 7$	$\mathbf{76\pm 5}$
cube-double-noisy	$(5,5)$	$26\pm 9$	$25\pm 5$	$\mathbf{53\pm 4}$

Appendix

Table 9: Own-value function versus a shared value function on the three bank members whose value networks estimate the same quantity (GCIQL, GCIVL, HIQL; same episodes and protocol as Table 1 , reported cell of each family). WMPA -own scores member

i

’s imagined rollout with member

i

’s own value network; WMPA -shared scores every rollout with the head of Table 1 . Bold marks values within

95\%

of the row maximum.

Dataset	metric	direct	direct $-$ metric [95% CI]	head in Table 1
cube-double-play	$\mathbf{0.75}$	$0.63$	$-0.12$ [ $-0.13$ , $-0.12$ ]*	metric
scene-play	$\mathbf{0.75}$	$0.67$	$-0.08$ [ $-0.08$ , $-0.08$ ]*	metric
puzzle-4x4-play	$0.70$	$\mathbf{0.82}$	$+0.12$ [ $+0.11$ , $+0.12$ ]*	direct
cube-triple-play	$0.64$	$\mathbf{0.66}$	$+0.02$ [ $+0.01$ , $+0.03$ ]*	metric

Appendix

Table 10: Selecting the value head for datasets. We report the concordance of each head with the ranking labels described in Appendix C , that is, the fraction of state pairs on which the head assigns the higher value to the state that reaches the shared goal sooner. The last column indicates the head used in Table 1 , which was chosen per task family before any evaluation episode was run. An asterisk ( ∗ ) indicates that the paired confidence interval excludes zero; in such cases, the head selected by the criterion is shown in bold.

				horizon agg.		ensemble agg.
Dataset	$(k,c)$	Best	WMPA	last	mean	min	mean-std
pointmaze-medium-navigate	$(1,1)$	$\mathbf{76\pm 17}$	$\mathbf{79\pm 3}$	$\mathbf{79\pm 3}$	$\mathbf{79\pm 3}$	$\mathbf{80\pm 4}$	$\mathbf{80\pm 4}$
antmaze-large-navigate	$(1,1)$	$\mathbf{90\pm 3}$	$\mathbf{91\pm 3}$	$\mathbf{91\pm 3}$	$\mathbf{91\pm 3}$	$\mathbf{92\pm 3}$	$\mathbf{91\pm 3}$
cube-single-play	$(5,5)$	$71\pm 3$	$\mathbf{85\pm 3}$	$\mathbf{87\pm 3}$	$\mathbf{84\pm 4}$	$\mathbf{88\pm 3}$	$\mathbf{87\pm 3}$
cube-single-noisy	$(5,5)$	$\mathbf{99\pm 1}$	$\mathbf{100\pm 0}$	$\mathbf{100\pm 0}$	$\mathbf{100\pm 0}$	$\mathbf{100\pm 0}$	$\mathbf{99\pm 1}$
cube-double-play	$(5,5)$	$36\pm 7$	$\mathbf{69\pm 3}$	$\mathbf{71\pm 4}$	$66\pm 4$	$\mathbf{71\pm 5}$	$\mathbf{70\pm 4}$

Appendix

Table 11: Score ablations at the reported cell : horizon aggregation and ensemble reduction of Equation 11 ; same protocol and episodes as Table 1 ; on the puzzle family the ablations are of the direct value at the reported cell. Entries are mean success

\pm

the half-width of the 95% hierarchical-bootstrap interval (bank seeds, then episodes), rounded to integers; numbers within

95\%

of the row maximum are in bold.

Dataset	Bank	$P$	Best member	WMPA	$\Delta$ vs best member	$\Delta$ vs six-member WMPA
cube-double-play	Full bank	6	GCIQL ( $36\pm 7$ )	$\mathbf{69\pm 3}$
cube-double-play	Pooled play+noisy	12	GCIQL ( $36\pm 7$ )	$\mathbf{79\pm 3}$	+43 [+35, +50]*	+10 [+6, +15]*
scene-play	Full bank	6	GCIQL ( $52\pm 5$ )	$\mathbf{87\pm 4}$
scene-play	Pooled play+noisy	12	GCIQL ( $52\pm 5$ )	$\mathbf{88\pm 3}$	+37 [+30, +43]*	+1 [-3, +5]
puzzle-4x4-play	Full bank	6	GCIQL ( $24\pm 5$ )	$\mathbf{51\pm 8}$
puzzle-4x4-play	Pooled play+noisy	12	GCIQL (noisy) ( $34\pm 4$ )	$\mathbf{53\pm 6}$	+18 [+11, +26]*	+2 [-4, +8]

Appendix

Table 12: Pooled play+noisy banks : the six policies of the base dataset plus the six trained on its sibling dataset, arbitrated with the base dataset’s world model, value, and interval on the base dataset’s episodes.

P

: bank size; best policy of that bank and its success; WMPA over that bank; paired contrasts of WMPA against the bank’s best policy and of pooled-bank WMPA against the six-member WMPA of the row above. Numbers within

95\%

of the row maximum are in bold. ( ∗ : interval excludes zero).

Dataset	$B_{\max}$	$D_{\mathrm{gap}}$	$U$ (oracle)	$U-B_{\max}$	WMPA	$\Delta$ WMPA [95% CI]	WMPA $-U$ [95% CI]
pointmaze-medium-navigate	$76\pm 17$	$2$	$\mathbf{98\pm 1}$	$22$	$79\pm 3$	+3 [-17, +17]	-19 [-22, -16]*
antmaze-large-navigate	$90\pm 3$	$4$	$\mathbf{99\pm 1}$	$9$	$91\pm 3$	+0 [-3, +4]	-9 [-12, -6]*
cube-single-play	$71\pm 3$	$22$	$\mathbf{81\pm 3}$	$10$	$\mathbf{85\pm 3}$	+14 [+10, +18]*	+4 [-1, +9]
cube-single-noisy	$\mathbf{99\pm 1}$	$28$	$\mathbf{100\pm 0}$	$1$	$\mathbf{100\pm 0}$	+1 [+0, +2]*	+0 [-0, +1]
cube-double-play	$36\pm 7$	$1$	$57\pm 6$	$21$	$\mathbf{69\pm 3}$	+33 [+25, +40]*	+12 [+6, +18]*
cube-double-noisy	$26\pm 9$	$11$	$31\pm 6$	$6$	$\mathbf{60\pm 7}$	+35 [+30, +40]*	+29 [+24, +34]*

Appendix

Table 13: Bank statistics and the static per-episode oracle (%) . Here,

U

is the success rate of a hindsight selector that picks a successful policy for each episode whenever one exists, and

U-B_{\max}

is the headroom available to any per-episode selector that does not switch policies within an episode. The WMPA column reports the scores from Table 1 , and

\Delta

WMPA is the difference between WMPA and

B_{\max}

. The WMPA

-U

column reports the paired per-episode difference between WMPA and the oracle on the same episodes, together with a hierarchical bootstrap confidence interval computed over bank seeds and episodes. Among

B_{\max}

U

, and WMPA , entries at or above

95\%

of the row maximum are shown in bold; note that

U

is a hindsight bound rather than a deployable method. The symbol ∗ indicates that the confidence interval excludes zero.

Dataset	$(k,c)$	ep. len	WMPA	Rand.	decisions/ep	WM calls/step	usage $H$ (bits)	WMPA	Rand.	fixed
			switches/ep					ms/step (CPU)
pointmaze-medium-navigate	$(1,1)$	295	24.8	246.5	294.8	18.0	1.30	3.76	0.21	0.18
antmaze-large-navigate	$(1,1)$	475	335.9	425.8	475.2	18.0	2.57	3.88	0.19	0.15
cube-single-play	$(5,5)$	90	13.0	27.3	18.3	18.3	2.54	3.29	0.35	0.23
cube-single-noisy	$(5,5)$	46	3.7	13.0	9.5	18.8	1.61	3.24	0.24	0.30
cube-double-play	$(5,5)$	264	37.8	73.8	53.1	18.1	2.55	2.97	0.23	0.22

Appendix

Table 14: Selection behavior and runtime of WMPA (selected

(k,c)

) and Random-Switch (same

c

). ep. len and decisions/ep belong to WMPA .

Explore similar work

Feb 5, 2026cs.LG

Beyond Success Rates: Trainability and Extractability for Offline GCRL

Offline goal-conditioned reinforcement learning (GCRL) is typically benchmarked by the best tuned success rate of each method. This score measures attainable performance, but it does not reveal how reliably a learned goal-conditioned signal can be extracted into a policy: a method could succeed across many value-learning and extraction settings, or only at a narrow, hard-to-find configuration. We study this gap across four methods, GCIQL, GCIVL, QRL, and CRL, under a shared advantage-weighted regression (AWR) extractor. For each method, we construct trainability landscapes over the optimizer learning rate, which affects value learning and actor optimization, and AWR temperature, which controls how selectively the actor imitates high-advantage transitions. Across AntMaze, Cube, and Scene, we observe distinct regimes: high-scoring methods may be broadly accessible or brittle, while broad relative basins may still sit below low absolute ceilings. To interpret these differences, we pair landscapes with post-hoc diagnostics of future-vs-random goal discrimination and AWR weight concentration. Their relationship to downstream success is task-dependent. On AntMaze, where future goals align with path-like progress, these diagnostics explain landscape regimes. On Cube and Scene, goal ranking and manipulation control decouple: methods can rank goals well while failing downstream, or succeed through action-conditioned advantages despite weak future-vs-random separation. These results show that peak tuned success alone does not establish broadly extractable goal-conditioned behavior. Trainability landscapes expose this gap, while extraction diagnostics offer a lower-cost lens on how learned signals become policies.

Jan Malte Töpperwien, Aditya Mohan, Marius Lindauer

Institute of Artificial Intelligence (LUH|AI), Leibniz University Hannover · L3S Research Center

RL Benchmarks Offline RL Goal-Conditioned RL

Ask PDF

Jun 19, 2026cs.LG

Inverting the Bellman Equation: From $Q$ -Values to World Models

Model-based and model-free reinforcement learning are traditionally viewed as separate paradigms: instead of learning a model of the transition kernel

P

, model-free agents typically estimate value functions tied to a specific policy and reward. In this paper, we challenge this dichotomy by proving that value-based agents trained on a sufficiently rich set of reward functions, e.g. using goal-conditioned RL, implicitly encode a unique and accurate world model. To extract this model in practice, we introduce \textit{

P

-learning}, an inverse analogue to

Q

-learning that samples from an agent's

Q

-values, policies and rewards to decode its internal model of the environment. We then provide sufficient conditions on the type and number of goals for which agents encode the true kernel

P

, covering both stochastic and deterministic MDPs over finite or continuous state spaces. Even when our assumptions are violated, we empirically demonstrate that agents trained on a handful of reward functions encode accurate dynamics in

\texttt{Reacher}

\texttt{MountainCar}

and stochastic variants of

\texttt{FourRooms}

. Surprisingly, we find that policies trained exclusively on a \texttt{Reacher} agent's implicit world model are quasi-optimal on out-of-distribution, velocity-based goals despite position-only training -- suggesting that agents contain hidden generalisation capabilities and providing a new lens into the connection between model-based, model-free, and goal-conditioned RL.

Alistair Letcher, Mattie Fellows, Alexander D. Goldie +3

FLAIR, University of Oxford · Google DeepMind · Mila, University of Montreal

Reinforcement Learning Model-Free RL Goal-Conditioned RL Model-Based RL

Ask PDF

May 25, 2026cs.LG

Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning

Offline goal-conditioned reinforcement learning (GCRL) provides a practical framework for obtaining goal-reaching policies from fixed datasets. However, learning a reliable goal-conditioned value function in long-horizon tasks remains challenging. In this paper, we identify erroneous generalization in goal-conditioned value functions as a fundamental bottleneck, and demonstrate that appropriate inductive bias in the value function is crucial for addressing the bottleneck. Building on these findings, we propose Latent-Aligned Value Learning (LAVL), an offline GCRL algorithm that integrates latent-representation-based value generalization with hierarchical planning in a unified framework. Extensive experiments on OGBench demonstrate that LAVL consistently outperforms existing offline GCRL methods, achieving the highest performance on 20 out of 22 datasets. Notably, LAVL exhibits strong performance in long-horizon tasks and trajectory stitching datasets, where prior methods suffer significant performance degradation. Our code is available at https://github.com/oh-lab/LAVL.git.

Hyungkyu Kang, Byeongchan Kim, Min-hwan Oh

Seoul National University, Seoul, South Korea.

Reinforcement Learning Offline RL Representation Learning for RL Goal-Conditioned RL

Ask PDF

Abstract

Explore similar work

Beyond Success Rates: Trainability and Extractability for Offline GCRL

Inverting the Bellman Equation: From QQQ-Values to World Models

Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning

Inverting the Bellman Equation: From $Q$ -Values to World Models