VaDar: Implicit Goal Conditioning via Value Disaggregation

Shashwat Saxena; Mehul Goel; Sreyas Venkataraman; Sarvesh Patil; Max Simchowitz

cs.LGOct 19, 2026

VaDar: Implicit Goal Conditioning via Value Disaggregation

Authors: Shashwat Saxena, Mehul Goel, Sreyas Venkataraman, Sarvesh Patil, Max Simchowitz

Organizations: Carnegie Mellon University

Abstract

Long-horizon tasks are important for modern AI and robotics, yet pose a challenge for reinforcement learning. Historically, a popular approach has been to decompose a task into subgoals, and learn a subgoal-conditioned policy to execute each subtask in sequence. We introduce an alternative called Value Disaggregation (VaDar), which trains a single subgoal-independent policy to maximize a linear combination of per-subgoal critic values. Because subgoal information is needed only at training-time, VaDar removes the need for possibly costly or cumbersome subgoal-selection during policy deployment. Moreover, through careful experiments, we show that VaDar (1) is always competitive with, and frequently outperforms, natural baselines like goal-conditioning and (2) succeeds on tasks with non-sequential goal structure and multiple optimization objectives. In particular, VaDar is the first algorithm to solve challenging, long-horizon tasks in the RoboCasa suite, even when starting from a policy pretrained from behavior cloning with near-zero initial task success. Taken together, VaDar's successes suggest that subgoal information benefits actor-critic reinforcement learning by enhancing the efficacy of critic learning, whereas policy goal-conditioning is often unnecessary.

Figures & tables

Figure 2: VaDar training and inference. Left: Subgoal completion indicators weight the per-subgoal critic values, whose aggregate guides updates to a subgoal-independent policy. Right: The policy selects actions directly from the state, without subgoal information.

Config.	Weight $\mathbf{w}[g]$	Interpretation
Current-myopic	$\mathbb{I}[g=g^{\mathrm{curr}}]$	Only the current subtask contributes.
Current-future	$\mathbb{I}[g\succeq g^{\mathrm{curr}}]$	All incomplete subtasks contribute.
Next-myopic	$\mathbb{I}[g=g^{\mathrm{next}}]$	Focus on the next subtask, gated by the current subtask.
Next-future	$\mathbb{I}[g\succeq g^{\mathrm{next}}]$	Incomplete subtasks with incomplete parents contribute.

Table 1: Value-weighting variants used for policy extraction in VaDar .

Figure 3: Subgoal weighting used by GetWeighting .

Figure 4: Success rate of VaDar - NF (solid) and VaDar - CF (dotted) against HGC , its implicit variant i HGC , the Dense preference-reward baseline, and the Sparse backbone across Robocasa365 , Robomimic , Libero and OGBench , for our three instantiations OGPO , FQL and SAC .

Figure 5: Weighting ablation of VaDar ( VaDar - GM , VaDar - CF , VaDar - NF ) against HGC and i HGC on Robocasa365 with the OGPO backbone and Robomimic Tool-Hang with the FQL backbone.

Figure 7: Success rate of VaDarLast-Q compared with VaDar - NF and Sparse on selected tasks. Figure 21 gives the results across all benchmarks and instantiations.

Figure 8: Best-of-

N

steering (

N=4

) of a frozen BC policy with VaDar - NF and Sparse critics as critic training progresses. The dashed line marks the unsteered policy’s success rate. We use the BC-pretrained flow policy from OGPO , without online policy updates.

Figure 9: All-subgoal TD targets update shared critic representations.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Tool-Hang with FQL : i HGC learns more smoothly and achieves higher final success than HGC .

Figure 11: Myopic failure modes of goal-conditioned policies. Rollouts of policies trained to optimize only the current subgoal on Robocasa365 ArrangeTea (a, top) and PreSoakPan (b, bottom); frames are chronological within each row. In (a), the kettle is placed on the tray without leaving room for the mug, so the subsequent mug placement collides with the kettle and knocks it off the counter. In (b), the pan is dropped into the sink in a position that does not permit a stable placement of the sponge on top of it

Signal	Observed issue
$\displaystyle\sum_{s\in\tau}V^{*}(s)$	Uses the value function $V^{}$ from an optimal policy $\pi^{}$ , but produced noisy preferences because $V^{}$ gave poor estimates outside the distribution induced by $\pi^{}$ .
$\displaystyle\sum_{t,s_{t}\in\tau}\gamma^{t}r_{\mathrm{sparse}}(s_{t})$	Monte Carlo returns depend on the policy used to generate the trajectory, making the induced reward signal noisy and not policy-invariant at the state level.
$\displaystyle\sum_{r_{\mathrm{dense}}\in\tau}r_{\mathrm{dense}}$	Uses $r_{\mathrm{dense}}$ as defined from the QIPM main algorithm, but it preferred longer trajectories, which introduced a harder and less reliable preference signal for the reward model to learn.

Appendix

Table 3: Alternative auto generated preference signals tested for Pbrl .

Panel of Figure 12 — Figure 12: Comparison of success-only (top) and mixed (bottom) dataset based reward rollouts. We see that the success-only dataset is unable to accurately estimate failure cases, not having a clear signal for reward. Both models were using preference based on $V^{*}$

Hyperparameter	Value
Reward model
ensemble size	$3$
model architecture	MLP of ( $512,512,512,512$ )
learning rate	$3\times 10^{-4}$
training epochs	$50$
segment size	episode

Appendix

Table 4: Reward-model hyperparameters used for PBRL experiments.

Figure 13: Success rate of VaDar - NF against Pref-Subgoal , which learns a dense reward per subgoal from preferences, on Robocasa365 with the OGPO backbone and Robomimic Tool-Hang with the FQL backbone.

Panel of Figure 14 — Figure 14: Transport Subgoal Structure.

Config.	Weight $w_{g}$	Interpretation
Current-myopic	$\mathbb{I}[g\in G^{\mathrm{curr}}(\mathbf{d})]$	Only currently available subtasks contribute.
Current-future	$\mathbb{I}[g\in\overline{\mathrm{desc}}(G^{\mathrm{curr}}(\mathbf{d}))]$	Current subtasks and their descendants contribute.
Next-myopic	$\mathbb{I}[g\in G^{\mathrm{next}}(\mathbf{d})]$	Only direct successors of current subtasks contribute.
Next-future	$\mathbb{I}[g\in\overline{\mathrm{desc}}(G^{\mathrm{next}}(\mathbf{d}))]$	Next subtasks and their descendants contribute.

Appendix

Table 5: DAG value-weighting variants used for policy extraction in VaDar .

Suite	Task	$\|G\|$	Structure
Robocasa365	PreSoakPan	5	sequential chain
Robocasa365	LoadDishwasher	5	sequential chain
Robocasa365	ArrangeTea	6	sequential chain
Robomimic	Transport	7	DAG, two parallel chains, joint handoff
Robomimic	Tool-Hang	4	sequential chain
Libero	KITCHEN_SCENE3	3	sequential chain

Appendix

Table 6: Summary of subgoal counts and dependency structure across benchmark tasks.

Environment	Max episode length
Robocasa365 (all envs: ArrangeTea , PreSoakPan , LoadDishwasher )	$1200$
Robomimic Transport	$800$
Robomimic Tool-Hang	$1000$
Libero KITCHEN_SCENE3/4/6	$500$
OGBench scene-play-singletask-task4	$100$
PushT-Obstacle	$300$

Appendix

Table 7: Maximum episode length per environment (used both for evaluation and as the truncation horizon during online finetuning)

Hyperparameter	Value
Optimization
critic learning rate	$3\times 10^{-4}$
actor / BC learning rate	$3\times 10^{-4}$
batch size	256
replay buffer size	$2\times 10^{6}$
Networks

Appendix

Table 8: Hyperparameters shared across all tasks and backbones ( OGPO , FQL , SAC )

Hyperparameter	Value
$h$ (action chunk length)	$8$
q_warmup_steps	$20{,}000$
flow_steps	$10$
best_of_n	$8$
critic target EMA $\tau$	$0.05$

Appendix

Table 9: OGPO hyperparameters held fixed across all five OGPO -backed tasks ( PreSoakPan , LoadDishwasher , ArrangeTea , Transport , Tool-Hang )

	Transport (low)	Transport (full)	Tool-Hang (low)	Tool-Hang (full)
dataset.proportion	$0.1$	$1.0$	$0.30$	$1.0$
offline_steps	300K	300K	300K	300K
online_steps	3M	3M	3M	3M
q_warmup_steps	20K	20K	20K	20K
start_training	40K	40K	10K	10K
$\bar{\tau}_{\pi}$	0.005	0.005	0.005	0.005

Appendix

Table 10: Low-data BC ablation on Robomimic . Both columns use the OGPO backbone with VaDar -style critic disaggregation. dataset.proportion controls the fraction of offline demonstrations used for BC pretraining and offline replay; all other hyperparameters remain the same.

	Transport	Tool-Hang
offline_steps	1M	500K
online_steps	6M	3M
q_warmup_steps	50K	50K
start_training	10K	10K
$h$ (chunk)	8	8
$\gamma$	0.999	0.999

Appendix

Table 11: Per-task hyperparameters for the FQL backbone on Robomimic . Values not listed match Table 8 .

Figure 15: VaDar - NF on Robocasa365 PreSoakPan with the OGPO base algorithm, trained with oracle subgoal completions vs. with completions judged by a VLM ( gemini-3.6-flash ).

Figure 16: Multi-Objective Optimization using VaDar on a variant of PushT created: PushT-Obstacle . Here the goal of the agent is to align the T with the given position whilst avoiding contact with the obstacle

Figure 17: Depiction of our PushT-Obstacle env. The agent needs to place the T into the T shaped slot (measured by IoU $\geq 95\%$ ) whilst avoiding the red shaped obstacle placed in the vicinity of the final position.

Figure 18: Plots of success rate for different values of

\lambda_{\mathrm{safety}}

comparing both VaDar and Aggregated - SAC on our PushT-Obstacle environment. Across all the values of

\lambda_{\mathrm{safety}}

that VaDar is more performative and stable than Aggregated - SAC

Figure 19: Best-of-

N

steering of the frozen OGPO BC policy with the Sparse and VaDar critics for

N\in\{4,16,64\}

(rows) on Robocasa365 ArrangeTea , PreSoakPan and Robomimic Tool-Hang (columns), as a function of the critic training budget. Error bars are one standard deviation over seeds; the dashed line is the unsteered BC policy.

		Sparse critic			VaDar critic
Task	Critic budget	$N=4$	$N=16$	$N=64$	$N=4$	$N=16$	$N=64$
ArrangeTea	$N=1$	$0.23\pm 0.04$
	40K	$0.17\pm 0.03$	$0.05\pm 0.04$	$0.01\pm 0.00$	$\mathbf{0.20\pm 0.07}$	$0.10\pm 0.06$	$0.02\pm 0.02$
	100K	$0.24\pm 0.12$	$0.11\pm 0.15$	$0.05\pm 0.08$	$\mathbf{0.33\pm 0.10}$	$0.23\pm 0.11$	$0.05\pm 0.05$
	200K	$0.27\pm 0.08$	$0.09\pm 0.03$	$0.05\pm 0.06$	$\mathbf{0.48\pm 0.07}$	$0.30\pm 0.11$	$0.16\pm 0.10$
	280K	$0.23\pm 0.04$	$0.09\pm 0.01$	$0.02\pm 0.02$	$\mathbf{0.55\pm 0.05}$	$0.38\pm 0.05$	$0.22\pm 0.06$

Appendix

Table 12: Success rate of best-of-

N

steering of the frozen OGPO BC policy with the Sparse and VaDar critics, as a function of the critic training budget (mean

\pm

std over seeds). The

N=1

row is the unsteered BC policy, and the best cell in each row is highlighted.

Figure 20: Q-values of the VaDar (middle) and Sparse (right, labelled OGPO in the panel) critics, each trained with the largest critic budget of 280K steps, at a single step of Robomimic Tool-Hang before the frame is inserted into the stand (left), plotted over the plane of the first two principal components of the BC policy’s 64 candidate action chunks. Dots are the candidates and the green square is the executed chunk; streamlines follow the gradient of $Q$ with respect to the action, and the contour spacing is the same in both panels.

Figure 21: Success rate of VaDarLast-Q compared with VaDar - NF and Sparse across all benchmarks and instantiations.

Figure 22: Subgoal reward shaping ablation. VaDar - NF against VaDar-Shaped , which sums the per-subgoal rewards into a single shaped reward before the Bellman backup and trains a standard scalar critic on it, on Robocasa365 with OGPO and Robomimic Tool-Hang with FQL .

Figure 23: Critic parameterization ablation. VaDar with its goal-conditioned critic

S{\times}A{\times}G\to\mathbb{R}

(blue) against Vector-VaDar , which uses a vector-valued critic

S{\times}A\to\mathbb{R}^{|G|}

(red), under next-future (solid) and current-future (dotted) weightings on Robocasa365 with OGPO and Robomimic Tool-Hang with FQL .

Figure 24: Iterative Critic Prediction

Figure 25: MIQ + VaDar (orange/purple) vs. monolithic vector critics (red/blue) at critic UTD

q=1

(solid) and

q=4

(dashed); the policy update frequency is held fixed across all runs. The largest MIQ effect is on OGPO Transport , where the monolithic critic degrades sharply at

q=4

and MIQ restores parity with

q=1

Figure 26: We compare monolithic VaDar critics with MIQ + VaDar at critic UTD

q\in\{1,4\}

, holding the actor update frequency fixed. MIQ substantially reduces total and per-subgoal TD residuals and produces lower-variance mean-

Q

estimates, while maintaining comparable or slightly improved success rates.

Long-Horizon RL

See all 32 papers

Locally-Guided Actor-Critic: Training a Goal-conditioned Actor with a Subgoal-aware Critic
Aug 31, 2026Olivier Serris, Stéphane Doncieux, Olivier SigaudReinforcement Learning Sparse-Reward RL
ISIR · Sorbonne Université, CNRS
Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning
Oct 1, 2026Pedro Robles Dutenhefner, Dikshant Shehmar, Wagner Meira +1Reinforcement Learning Offline RL
Universidade Federal de Minas Gerais · University of Alberta · Alberta Machine Intelligence Institute (Amii) +1
Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning
May 25, 2026Hyungkyu Kang, Byeongchan Kim, Min-hwan OhReinforcement Learning Offline RL
Seoul National University, Seoul, South Korea.

Aug 31, 2026J/K move · Enter open · S save

Locally-Guided Actor-Critic: Training a Goal-conditioned Actor with a Subgoal-aware Critic

Olivier Serris, Stéphane Doncieux, Olivier Sigaud

ISIR · Sorbonne Université, CNRS

Reinforcement Learning Sparse-Reward RL Actor-Critic Methods Reward Design for RL Goal-Conditioned RL

Open paper AskPDF arXiv

Goal-conditioned reinforcement learning struggles with long horizons when rewards are sparse. While a planner can provide subgoals to guide a low-level policy, its use at test time may introduce practical subgoal management difficulties. An alternative paradigm utilizes a high-level planner to assist learning, while the policy remains conditioned only on the final goal, enabling planner-free deployment. Among these methods, Reinforcement Learning with Imagined Subgoals (RIS) introduces a regularization term that encourages the policy to take the same actions for the final goal as it does for an intermediate goal. This regularization, however, may lead to goal-chaining issues when intermediate goals are low-dimensional. Potential-based reward shaping (PBRS) translates plans into an additional reward while ensuring that the optimal policy remains unchanged. Yet, it can generate deceptive rewards in terminal states. We study these failure cases and first propose an alternative reward shaping method (RS) that removes these deceptive rewards at the expense of theoretical guarantees of PBRS. Similar to this RS variant, we then propose another method named Locally-Guided Actor Critic (LG-AC) that rewards the agent for reaching intermediate goals. Unlike RS, where intermediate rewards are implicit in the shaping signal, we explicitly condition a value estimator on the full sequence of intermediate goals but represent the value function as a sum of subgoal-conditioned value functions, enabling dense hindsight relabeling. We evaluate all these methods in tasks with challenging goal-chaining requirements and empirically highlight specific cases in which either action regularization or reward shaping yield low performance, while LG-AC achieves the best overall performance across tasks.

Abstract

Long-Horizon RL

Explore similar work

Locally-Guided Actor-Critic: Training a Goal-conditioned Actor with a Subgoal-aware Critic

Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning

Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning