Long-horizon tasks are important for modern AI and robotics, yet pose a challenge for reinforcement learning. Historically, a popular approach has been to decompose a task into subgoals, and learn a subgoal-conditioned policy to execute each subtask in sequence. We introduce an alternative called Value Disaggregation (VaDar), which trains a single subgoal-independent policy to maximize a linear combination of per-subgoal critic values. Because subgoal information is needed only at training-time, VaDar removes the need for possibly costly or cumbersome subgoal-selection during policy deployment. Moreover, through careful experiments, we show that VaDar (1) is always competitive with, and frequently outperforms, natural baselines like goal-conditioning and (2) succeeds on tasks with non-sequential goal structure and multiple optimization objectives. In particular, VaDar is the first algorithm to solve challenging, long-horizon tasks in the RoboCasa suite, even when starting from a policy pretrained from behavior cloning with near-zero initial task success. Taken together, VaDar's successes suggest that subgoal information benefits actor-critic reinforcement learning by enhancing the efficacy of critic learning, whereas policy goal-conditioning is often unnecessary.
Figures & tables
Figure 2: VaDar training and inference. Left: Subgoal completion indicators weight the per-subgoal critic values, whose aggregate guides updates to a subgoal-independent policy. Right: The policy selects actions directly from the state, without subgoal information.
Config.
Weight w[g]
Interpretation
Current-myopic
I[g=gcurr]
Only the current subtask contributes.
Current-future
I[g⪰gcurr]
All incomplete subtasks contribute.
Next-myopic
I[g=gnext]
Focus on the next subtask, gated by the current subtask.
Next-future
I[g⪰gnext]
Incomplete subtasks with incomplete parents contribute.
Table 1: Value-weighting variants used for policy extraction in VaDar .
Figure 3: Subgoal weighting used by GetWeighting .
Figure 4: Success rate of VaDar - NF (solid) and VaDar - CF (dotted) against HGC , its implicit variant i HGC , the Dense preference-reward baseline, and the Sparse backbone across Robocasa365 , Robomimic , Libero and OGBench , for our three instantiations OGPO , FQL and SAC .
Figure 5: Weighting ablation of VaDar ( VaDar - GM , VaDar - CF , VaDar - NF ) against HGC and i HGC on Robocasa365 with the OGPO backbone and Robomimic Tool-Hang with the FQL backbone.
Figure 6
Figure 7: Success rate of VaDarLast-Q compared with VaDar - NF and Sparse on selected tasks. Figure 21 gives the results across all benchmarks and instantiations.
Figure 8: Best-of- N steering ( N=4 ) of a frozen BC policy with VaDar - NF and Sparse critics as critic training progresses. The dashed line marks the unsteered policy’s success rate. We use the BC-pretrained flow policy from OGPO , without online policy updates.
Figure 10: Tool-Hang with FQL : i HGC learns more smoothly and achieves higher final success than HGC .
Figure 11: Myopic failure modes of goal-conditioned policies. Rollouts of policies trained to optimize only the current subgoal on Robocasa365 ArrangeTea (a, top) and PreSoakPan (b, bottom); frames are chronological within each row. In (a), the kettle is placed on the tray without leaving room for the mug, so the subsequent mug placement collides with the kettle and knocks it off the counter. In (b), the pan is dropped into the sink in a position that does not permit a stable placement of the sponge on top of it
Signal
Observed issue
s∈τ∑V∗(s)
Uses the value function V∗ from an optimal policy π∗ , but produced noisy preferences because V∗ gave poor estimates outside the distribution induced by π∗ .
t,st∈τ∑γtrsparse(st)
Monte Carlo returns depend on the policy used to generate the trajectory, making the induced reward signal noisy and not policy-invariant at the state level.
rdense∈τ∑rdense
Uses rdense as defined from the QIPM main algorithm, but it preferred longer trajectories, which introduced a harder and less reliable preference signal for the reward model to learn.
Appendix
Table 3: Alternative auto generated preference signals tested for Pbrl .
Figure 12: Comparison of success-only (top) and mixed (bottom) dataset based reward rollouts. We see that the success-only dataset is unable to accurately estimate failure cases, not having a clear signal for reward. Both models were using preference based on V∗
Hyperparameter
Value
Reward model
ensemble size
3
model architecture
MLP of ( 512,512,512,512 )
learning rate
3×10−4
training epochs
50
segment size
episode
Appendix
Table 4: Reward-model hyperparameters used for PBRL experiments.
Figure 13: Success rate of VaDar - NF against Pref-Subgoal , which learns a dense reward per subgoal from preferences, on Robocasa365 with the OGPO backbone and Robomimic Tool-Hang with the FQL backbone.
Figure 14: Transport Subgoal Structure.
Config.
Weight wg
Interpretation
Current-myopic
I[g∈Gcurr(d)]
Only currently available subtasks contribute.
Current-future
I[g∈desc(Gcurr(d))]
Current subtasks and their descendants contribute.
Next-myopic
I[g∈Gnext(d)]
Only direct successors of current subtasks contribute.
Next-future
I[g∈desc(Gnext(d))]
Next subtasks and their descendants contribute.
Appendix
Table 5: DAG value-weighting variants used for policy extraction in VaDar .
Suite
Task
∣G∣
Structure
Robocasa365
PreSoakPan
5
sequential chain
Robocasa365
LoadDishwasher
5
sequential chain
Robocasa365
ArrangeTea
6
sequential chain
Robomimic
Transport
7
DAG, two parallel chains, joint handoff
Robomimic
Tool-Hang
4
sequential chain
Libero
KITCHEN_SCENE3
3
sequential chain
Appendix
Table 6: Summary of subgoal counts and dependency structure across benchmark tasks.
Table 7: Maximum episode length per environment (used both for evaluation and as the truncation horizon during online finetuning)
Hyperparameter
Value
Optimization
critic learning rate
3×10−4
actor / BC learning rate
3×10−4
batch size
256
replay buffer size
2×106
Networks
Appendix
Table 8: Hyperparameters shared across all tasks and backbones ( OGPO , FQL , SAC )
Hyperparameter
Value
h (action chunk length)
8
q_warmup_steps
20,000
flow_steps
10
best_of_n
8
critic target EMA τ
0.05
Appendix
Table 9: OGPO hyperparameters held fixed across all five OGPO -backed tasks ( PreSoakPan , LoadDishwasher , ArrangeTea , Transport , Tool-Hang )
Transport (low)
Transport (full)
Tool-Hang (low)
Tool-Hang (full)
dataset.proportion
0.1
1.0
0.30
1.0
offline_steps
300K
300K
300K
300K
online_steps
3M
3M
3M
3M
q_warmup_steps
20K
20K
20K
20K
start_training
40K
40K
10K
10K
τˉπ
0.005
0.005
0.005
0.005
Appendix
Table 10: Low-data BC ablation on Robomimic . Both columns use the OGPO backbone with VaDar -style critic disaggregation. dataset.proportion controls the fraction of offline demonstrations used for BC pretraining and offline replay; all other hyperparameters remain the same.
Transport
Tool-Hang
offline_steps
1M
500K
online_steps
6M
3M
q_warmup_steps
50K
50K
start_training
10K
10K
h (chunk)
8
8
γ
0.999
0.999
Appendix
Table 11: Per-task hyperparameters for the FQL backbone on Robomimic . Values not listed match Table 8 .
Figure 15: VaDar - NF on Robocasa365 PreSoakPan with the OGPO base algorithm, trained with oracle subgoal completions vs. with completions judged by a VLM ( gemini-3.6-flash ).
Figure 16: Multi-Objective Optimization using VaDar on a variant of PushT created: PushT-Obstacle . Here the goal of the agent is to align the T with the given position whilst avoiding contact with the obstacle
Figure 17: Depiction of our PushT-Obstacle env. The agent needs to place the T into the T shaped slot (measured by IoU ≥95% ) whilst avoiding the red shaped obstacle placed in the vicinity of the final position.
Figure 18: Plots of success rate for different values of λsafety comparing both VaDar and Aggregated - SAC on our PushT-Obstacle environment. Across all the values of λsafety that VaDar is more performative and stable than Aggregated - SAC
Figure 19: Best-of- N steering of the frozen OGPO BC policy with the Sparse and VaDar critics for N∈{4,16,64} (rows) on Robocasa365 ArrangeTea , PreSoakPan and Robomimic Tool-Hang (columns), as a function of the critic training budget. Error bars are one standard deviation over seeds; the dashed line is the unsteered BC policy.
Sparse critic
VaDar critic
Task
Critic budget
N=4
N=16
N=64
N=4
N=16
N=64
ArrangeTea
N=1
0.23±0.04
40K
0.17±0.03
0.05±0.04
0.01±0.00
0.20±0.07
0.10±0.06
0.02±0.02
100K
0.24±0.12
0.11±0.15
0.05±0.08
0.33±0.10
0.23±0.11
0.05±0.05
200K
0.27±0.08
0.09±0.03
0.05±0.06
0.48±0.07
0.30±0.11
0.16±0.10
280K
0.23±0.04
0.09±0.01
0.02±0.02
0.55±0.05
0.38±0.05
0.22±0.06
Appendix
Table 12: Success rate of best-of- N steering of the frozen OGPO BC policy with the Sparse and VaDar critics, as a function of the critic training budget (mean ± std over seeds). The N=1 row is the unsteered BC policy, and the best cell in each row is highlighted.
Figure 20: Q-values of the VaDar (middle) and Sparse (right, labelled OGPO in the panel) critics, each trained with the largest critic budget of 280K steps, at a single step of Robomimic Tool-Hang before the frame is inserted into the stand (left), plotted over the plane of the first two principal components of the BC policy’s 64 candidate action chunks. Dots are the candidates and the green square is the executed chunk; streamlines follow the gradient of Q with respect to the action, and the contour spacing is the same in both panels.
Figure 21: Success rate of VaDarLast-Q compared with VaDar - NF and Sparse across all benchmarks and instantiations.
Figure 22: Subgoal reward shaping ablation. VaDar - NF against VaDar-Shaped , which sums the per-subgoal rewards into a single shaped reward before the Bellman backup and trains a standard scalar critic on it, on Robocasa365 with OGPO and Robomimic Tool-Hang with FQL .
Figure 23: Critic parameterization ablation. VaDar with its goal-conditioned critic S×A×G→R (blue) against Vector-VaDar , which uses a vector-valued critic S×A→R∣G∣ (red), under next-future (solid) and current-future (dotted) weightings on Robocasa365 with OGPO and Robomimic Tool-Hang with FQL .
Figure 24: Iterative Critic Prediction
Figure 25: MIQ + VaDar (orange/purple) vs. monolithic vector critics (red/blue) at critic UTD q=1 (solid) and q=4 (dashed); the policy update frequency is held fixed across all runs. The largest MIQ effect is on OGPO Transport , where the monolithic critic degrades sharply at q=4 and MIQ restores parity with q=1 .
Figure 26: We compare monolithic VaDar critics with MIQ + VaDar at critic UTD q∈{1,4} , holding the actor update frequency fixed. MIQ substantially reduces total and per-subgoal TD residuals and produces lower-variance mean- Q estimates, while maintaining comparable or slightly improved success rates.
Goal-conditioned reinforcement learning struggles with long horizons when rewards are sparse. While a planner can provide subgoals to guide a low-level policy, its use at test time may introduce practical subgoal management difficulties. An alternative paradigm utilizes a high-level planner to assist learning, while the policy remains conditioned only on the final goal, enabling planner-free deployment. Among these methods, Reinforcement Learning with Imagined Subgoals (RIS) introduces a regularization term that encourages the policy to take the same actions for the final goal as it does for an intermediate goal. This regularization, however, may lead to goal-chaining issues when intermediate goals are low-dimensional. Potential-based reward shaping (PBRS) translates plans into an additional reward while ensuring that the optimal policy remains unchanged. Yet, it can generate deceptive rewards in terminal states. We study these failure cases and first propose an alternative reward shaping method (RS) that removes these deceptive rewards at the expense of theoretical guarantees of PBRS. Similar to this RS variant, we then propose another method named Locally-Guided Actor Critic (LG-AC) that rewards the agent for reaching intermediate goals. Unlike RS, where intermediate rewards are implicit in the shaping signal, we explicitly condition a value estimator on the full sequence of intermediate goals but represent the value function as a sum of subgoal-conditioned value functions, enabling dense hindsight relabeling. We evaluate all these methods in tasks with challenging goal-chaining requirements and empirically highlight specific cases in which either action regularization or reward shaping yield low performance, while LG-AC achieves the best overall performance across tasks.
Existing approaches to offline goal-conditioned reinforcement learning (GCRL) struggle with long-horizon tasks. Discounting shrinks value differences between distant states until they fall below the function approximation error, leaving the agent with no signal for ranking states. Temporal abstraction, which treats k environment steps as a single transition, restores this signal at long range, but no single fixed k suits all state-goal distances: large k preserves value differences across long temporal distances while collapsing distinctions between nearby states, and small k does the reverse. We make this trade-off explicit and introduce Generalized Implicit Temporal Abstraction (GITA), which conditions a single value function on k. GITA trains one policy by aggregating advantage-weighted supervision across multiple k values, so scales assigning larger positive advantages to a state-goal pair contribute more strongly to its update. GITA does not need to choose between local resolution and long-range signal; it retains both without committing to a single k. On OGBench, GITA outperforms a broad range of offline GCRL baselines, raising average success rate across all tasks by 25 percentage points (73% relative improvement) over HIQL. It also improves over the strongest fixed-k method, OTA, by 7 percentage points (14% relative).
Pedro Robles Dutenhefner, Dikshant Shehmar, Wagner Meira +1
Universidade Federal de Minas Gerais · University of Alberta · Alberta Machine Intelligence Institute (Amii) +1
Offline goal-conditioned reinforcement learning (GCRL) provides a practical framework for obtaining goal-reaching policies from fixed datasets. However, learning a reliable goal-conditioned value function in long-horizon tasks remains challenging. In this paper, we identify erroneous generalization in goal-conditioned value functions as a fundamental bottleneck, and demonstrate that appropriate inductive bias in the value function is crucial for addressing the bottleneck. Building on these findings, we propose Latent-Aligned Value Learning (LAVL), an offline GCRL algorithm that integrates latent-representation-based value generalization with hierarchical planning in a unified framework. Extensive experiments on OGBench demonstrate that LAVL consistently outperforms existing offline GCRL methods, achieving the highest performance on 20 out of 22 datasets. Notably, LAVL exhibits strong performance in long-horizon tasks and trajectory stitching datasets, where prior methods suffer significant performance degradation. Our code is available at https://github.com/oh-lab/LAVL.git.