Reinforcement Learning via Advantage Guidance

Zhiyuan Zhou; Sergey Levine

cs.LGOct 14, 2026

Reinforcement Learning via Advantage Guidance

Authors: Zhiyuan Zhou, Sergey Levine

Organizations: UC Berkeley

Abstract

Reinforcement learning (RL) holds the promise of learning a policy that can outperform what is shown in the data. However, in practice, such systems suffer from instability and hyperparameter sensitivity. In contrast, supervised learning provides a stable and scalable training objective, but is fundamentally limited by the quality of the demonstrated behavior. In this paper, we ask whether we can perform policy optimization with something as simple and stable as supervised learning but as performant as RL. Unfortunately, naively training the policy with supervised learning and conditioning on future rewards or advantages typically underperform dedicated RL methods. We introduce a new policy extraction method, Advantage Guidance, that retains the simple supervised learning structure with advantage conditioning while outperforming prior RL methods through effective conditional training and contrastive diffusion guidance during sampling. We show that Advantage Guidance can significantly improve a model's ability to respond to conditional advantage signals, and the contrastive guidance during sampling can use the learned advantage-conditioned policy to target the KL-regularized optimal policy. Empirically, we find that Advantage Guidance significantly outperforms prior state-of-the-art RL methods on 26 long-horizon, sparse reward tasks across datasets with varying degrees of optimality.

Figures & tables

Figure 2: Naïve advantage conditioned policy suffers from the advantage alignment problem , generating actions whose advantages don’t align with the requested value.

5

tasks per domain.

Figure 3: Naïve advantage conditioned policies often ignore advantage inputs. Averaged over

20

OGBench tasks.

Training	Sampling
Input: dataset $\mathcal{D}$ , advantage $A(s,a)$ , $K$ , advantage label std $\sigma_{y}$	Input: state $s$ , trained velocities $v_{\hat{\pi}},v_{\theta}$
while not converged do	conditional query pair $y_{+}>y_{-}$ ,
$(s,a)\sim\mathcal{D}$ , $\epsilon\sim\mathcal{N}(0,I)$ , $t\sim\mathcal{U}(0,1)$	guidance weight $w$ , steps $T$
Train ref. policy $\hat{\pi}$ with $\displaystyle\mathcal{L}_{\mathrm{BC}}=\left\\|v_{\hat{\pi}}-(a-\epsilon)\right\\|_{2}^{2}$	$x_{0}\sim\mathcal{N}(0,I)$
$a_{0}\leftarrow a$ ; $a_{i}\sim\hat{\pi}(\cdot\mid s)$ , $i=1,\ldots,K$ $\blacktriangleleft$ action imagination	for $n=0,\ldots,T-1$ do
$A_{i}\leftarrow A(s,a_{i})$ , $i=0,\ldots,K$	$t\leftarrow n/T$

Algorithm 1 Advantage Guidance (AG)

Figure 4: Offline RL performance on OGBench (20 tasks) and OCBench (6 tasks) with

10

seeds: Advantage Guidance ( blue ) outperforms all previous methods, including strong actor-critic methods ( grey ), supervised-learning-based methods ( green ), and diffusion guidance methods ( shaded green ).

Figure 5: Comparison with best-of-

N

sampling . Advantage Guidance outperforms BFN at small

N

on OGBench and consistently on OCBench, despite using much less test-time compute.

Figure 8: Ablation of sampling method for Advantage Guidance. Contrastive guidance significantly outperforms CFG and no guidance on

6

OCBench tasks.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Ablating the number of imagined actions,

K

, used for training the conditional policy in Advantage Guidance. A small

K

underperforms larger values, but performance quickly plateaus. Averaged over

5

tasks per domain and

4

seeds.

Figure 10: Ablating different advantage normalization schemes: our per-state advantage normalization outperforms batch-level and dataset-level normalization. Averaged over

5

tasks per domain and

4

seeds.

Figure 11: Sensitivity to contrastive conditions and guidance weight: each contrastive condition performs similarly given the best guidance weight for that condition.

Figure 12: Sensitivity to different advantage label standard deviation: performance is not sensitive to the label standard deviations as long as the best guidance weight is picked. Averaged over

5

tasks per domain and

4

seeds.

Figure 13: Naïve advantage conditioned policy RCP-A has the same performance across

20

OGBench tasks (

5

tasks per domain) when queried with different advantage values. Averaged over

4

seeds.

Figure 14: Using a transformer for the policy architecture does not significantly improve advantage alignment (left), and does not improve policy performance overall on OGBench (right).

Figure 15: Pure conditional sampling of Advantage Guidance: querying with large advantage values leads to a sharp performance drop on OCBench and hard long horizon task puzzle-3x3 . Averaged over different tasks in the same domain with

10

seeds.

Figure 16: Comparison of two critic sizes on cube-single in OGBench on four different methods.

2\times 128

significantly outperforms

4\times 512

nearly for all methods, suggesting that the commonly used

4\times 512

architecture suffers from critic overfitting.

Hyperparameter	OGBench	OCBench
Batch size	$256$	$1024$
Discount factor $\gamma$	$0.99$	$0.999$
IQL expectile $\tau$	$0.9$	$0.5$
Critic ensemble size	2	2
Critic aggregation method	min	min
Target-network update rate	$0.005$	$0.005$

Appendix

Table 1: Common hyperparameters used for the OGBench and OCBench experiments in Figure 4 .

Hyperparameter	OGBench	OCBench
Positive condition $y_{+}$	$1$	$1$
Negative condition $y_{-}$	$-1$	$-1$
Advantage label standard deviation $\sigma_{y}$	$0.1$	$0.01$
Number of Imagined Actions $K$	$7$	$7$

Appendix

Table 2: Common hyperparameters for Advantage Guidance . As shown in Figure 12 and Figure 11 , different values of

y_{+},y_{-},\sigma_{y}

hyperparameters lead to very similar results. We therefore fix these parameters across tasks on the same benchmark and only tune the guidance weight for Advantage Guidance.

Suite	Domain	EDP ( $\alpha_{\mathrm{EDP}}$ )	FQL ( $\alpha_{\mathrm{FQL}}$ )	QGF ( $w_{\mathrm{QGF}}$ )	CFGRL ( $w_{\mathrm{CFGRL}}$ )	FAWR ( $\beta$ )	RCP-A ( $k$ )	RCP-R ( $k$ )	CRR	AG ( $w$ )
OGBench	Cube Single	$100$	$1000$	$0.08$	$2$	$0.3$	1	1	–	$1$
OGBench	Cube Double	$10$	$100$	$0.16$	$2$	$0.3$	1	1	–	$1.5$
OGBench	Scene	$30$	$300$	$0.16$	$2$	$10$	1	1	–	$1$
OGBench	Puzzle $3{\times}3$	$30$	$300$	$0.16$	$3$	$0.3$	1	1	–	$2.5$
OCBench	Block Single	$1$	$300$	$0.02$	$10$	$3$	1	1	–	$1.5$
OCBench	Block Double	$(1,1)$	$(100,100)$	$(0.32,0.32)$	$(15,15)$	$(3,0.3)$	1	1	–	$(3,4)$

Appendix

Table 3: Domain-specific hyperparameters used for OGBench and OCBench results in Figure 4 . For OCBench, parenthesized values are ordered by task number. For example,

(3,4)

denotes the values used for Tasks 1 and 2 of Block Double. A dash indicates that the method has no tuned domain-specific hyperparameter.

Method	Hyperparameter	Tunning Range
EDP	BC coefficient $\alpha_{\mathrm{EDP}}$	OGBench: $\{10,30,100,300\}$ ; OCBench: $\{1,10\}$
FQL	BC coefficient $\alpha_{\mathrm{FQL}}$	$\{100,300,1000\}$
QGF	Guidance weight $w$	$\{0,0.002,0.004,0.01,0.02,0.04,0.08,0.16,0.32\}$
CFGRL	Guidance weight $w$	OGBench: $\{0,0.5,1,2,3,5,10,15\}$ ; OCBench: $\{0,0.5,1,1.5,2,3,5,10,15\}$
FAWR	Inverse temperature $\beta$	OGBench: $\{0.3,1,3,10\}$ ; OCBench: $\{0.3,1,3\}$
RCP-A	inference std multiplier $k$	OGBench: $\{0.5,0.7,1,1.5,2,3\}$ ; OCBench: $\{1\}$

Appendix

Table 4: Hyperparameter ranges for OGBench and OCBench experiments.

Figure 17: Task breakdown of Figure 4 : Comparing Advantage Guidance against prior methods in offline RL setting on 20 OGBench tasks and 6 OCBench tasks. Averaged over 10 random seeds. See exact plotted numbers in Table 5 .

Figure 18: Task breakdown of Figure 7 : Comparing (1) gradient similarity between gradient of policy loss conditioned on true and random advantage values and (2) the variance of policy output conditioned on different advantage values or initial noise.

Figure 19: Task breakdown of Figure 5 , comparing Advantage Guidance with best-of-N (BFN) sampling at different values of

N

on OGBench and OCBench. Averaged over

10

random seeds.

Figure 20: Domain breakdown of Figure 7 : ablation of residual parameterization and action imagination.

Figure 21: Task breakdown of Figure 11 : sensitivity to contrastive condition pair

(y_{+},y_{-})

and guidance weight

w

. Top: Performance at different guidance weights. Bottom: Performance at best guidance weight.

Figure 22: Domain breakdown of Figure 8 : Ablation of sampling method for Advantage Guidance. 10 random seeds.

Figure 23: Compute Cost of BFN agents. The result is measured in terms of FLOPs per primitive action in the action chunk, and is averaged over OGBench and OCBench environments.

Figure 24: Offline RL performance on cube-double with discount factor

0.999

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Reward-Conditioned Reinforcement Learning
Mar 5, 2026Michal Nauman, Marek Cygan, Pieter AbbeelReinforcement Learning Off-Policy RL
University of Warsaw · University of Warsaw, Nomagic · UC Berkeley, Amazon FAR
Adaptive Expert Guidance for Efficient On-Policy Reinforcement Learning
Oct 5, 2026Daniele Affinita, Ming Xu, Rudolf Reiter +2Sample-Efficient RL
EPFL, Switzerland · University of Zurich, Switzerland
Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RL
Jul 1, 2026Juliette Decugis, Sean O'Brien, Francis Bach +2Policy Gradient RL for Language Model Reasoning
FAIR at Meta · Work done at Meta, now working at the University of California, San Diego · Inria, Ecole Normale Supérieure

Mar 5, 2026J/K move · Enter open · S save

Reward-Conditioned Reinforcement Learning

Michal Nauman, Marek Cygan, Pieter Abbeel

University of Warsaw · University of Warsaw, Nomagic · UC Berkeley, Amazon FAR

Reinforcement Learning Off-Policy RL Multi-Objective Reinforcement Learning

Open paper AskPDF arXiv

Single-task RL agents are typically trained under a fixed reward function, which limits their robustness to reward misspecification and their ability to adapt to changing preferences. We introduce Reward-Conditioned Reinforcement Learning (RCRL), an off-policy method that conditions agents on reward parameterizations while collecting experience under a single nominal objective. By recomputing counterfactual rewards from shared replay data, RCRL exposes the agent to multiple reward objectives without additional environment interaction, connecting single-task RL with ideas from multi-objective and multi-task learning. Across single-task, multi-task, and vision-based benchmarks, RCRL improves sample efficiency under the nominal reward parameterization, enables efficient adaptation to new parameterizations, and supports zero-shot behavioral adjustment at deployment. Our results show that RCRL provides a scalable mechanism for learning robust, steerable policies without sacrificing the simplicity of single-task training.

Abstract

Reinforcement Learning

Explore similar work

Reward-Conditioned Reinforcement Learning

Adaptive Expert Guidance for Efficient On-Policy Reinforcement Learning

Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RL