AWLS: Advantage-Weighted Latent Steering for One-Step Generative Robotic Manipulation Policies

Jiahong Tang; Jiaju He; Zhiyuan Yang; Ruyang Han; Haiyue Zhu

cs.ROOct 20, 2026

AWLS: Advantage-Weighted Latent Steering for One-Step Generative Robotic Manipulation Policies

Authors: Jiahong Tang, Jiaju He, Zhiyuan Yang, Ruyang Han, Haiyue Zhu

Organizations: Nanyang Technological University, Singapore · Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR), Singapore

Abstract

Human-collected real-world robotic manipulation demonstrations often contain mixed-quality trajectories, including hesitant, indirect, or inefficient action segments. While diffusion- and flow-based policies can model such multimodal data, standard behavior cloning (BC) assigns equal likelihood to all demonstrated actions and may therefore reproduce suboptimal modes. Offline RL value estimates offer a natural signal for biasing policies toward higher-value behavioral modes, but directly fine-tuning the generative policy or using critic gradients for policy steering can amplify critic errors and induce prior drift. We propose Advantage-Weighted Latent Steering (AWLS), an offline post-training method for improving frozen one-step generative manipulation policies without updating their decoders. AWLS learns a residual noise shaper, an observation-conditioned residual correction to the policy's latent noise, and trains it with an advantage-weighted action reconstruction objective. The advantages are detached, scale-normalized, and used only as supervised weights on demonstrated actions, avoiding critic-gradient policy updates while preserving fast single-step inference. We further introduce a rollout-free decoder-sensitivity criterion for selecting pretrained checkpoints that remain responsive to latent-noise perturbations. Experiments on simulated and real-world robotic manipulation tasks with clean and mixed-quality demonstrations show that AWLS improves over the frozen base policy while limiting drift from the pretrained behavior prior. Project page: https://awls-corl2026.github.io/

Figures & tables

Figure 1: Trajectory segments from human-collected robomimic can demonstrations [ 5 ] , with policy rollouts. Thin dashed lines are demonstration segments from a shared start to the goal, and thick curves are illustrative policy rollouts. Behavior cloning treats every demonstration the same, so its rollouts reproduce the hesitant detours as faithfully as the efficient paths. Q-maximization steering pushes actions toward high-

Q

estimates (arrows), so its rollouts leave the demonstrated region. AWLS weights demonstrations by advantage (green high, blue low) and reshapes the latent noise of the frozen decoder toward the high-advantage demonstrated mode, so its rollouts follow the efficient paths.

Figure 2: Overview of AWLS. Top: the one-step decoder $G_{\theta}$ is pretrained by behavior cloning. Bottom: with $G_{\theta}$ frozen, $T_{\phi}$ reshapes the latent noise via advantage-weighted reconstruction using detached, normalized offline advantages.

Paradigm	Method	Denoising Steps	Adroit (3 tasks)	DexArt (4 tasks)	RoboTwin (5 tasks)	Average
Behavior Cloning	DP3	10	67.5 $\pm$ 3.3	50.3 $\pm$ 1.2	45.0 $\pm$ 3.8	52.4 $\pm$ 2.8
	FlowPolicy	1	71.3 $\pm$ 2.0	50.4 $\pm$ 2.2	48.3 $\pm$ 4.2	54.8 $\pm$ 3.0
	MeanFlow	1	72.1 $\pm$ 4.2	54.7 $\pm$ 2.2	53.2 $\pm$ 4.6	58.4 $\pm$ 3.7
Q-maximization	DSRL	10	72.5 $\pm$ 2.5	56.4 $\pm$ 3.7	55.8 $\pm$ 4.8	60.2 $\pm$ 3.9
	FQL	1	73.6 $\pm$ 2.6	57.4 $\pm$ 2.4	56.1 $\pm$ 4.9	60.9 $\pm$ 3.5
Ours	FP-AWLS	1	74.8 $\pm$ 2.6	58.5 $\pm$ 2.1	57.6 $\pm$ 4.5	62.2 $\pm$ 3.2

Table 3

Panel of Figure 3 — Figure 3: Left: Average success rate on four DexArt tasks as the noisy-demonstration fraction increases. Right: PCA projection of flattened Bucket action trajectories with 50 clean and 50 noisy demonstrations.

Panel of Figure 4 — Figure 4: Real-world manipulation tasks. The first three are single-arm tasks: Drawer-Cube opens a drawer, places a cube, and closes the drawer; Tube Insertion removes a jar and inserts a tube; Screw Scooping scoops screws and pours them into a target container. The last two are dual-arm tasks: Gear Packing opens a box, places a gear, and closes the box; Ball Weighing presses the scale button with one arm while placing the ball on the scale with the other arm.

Method	Drawer-Cube	Tube Ins.	Screw Scoop.	Gear Pack.	Ball Weigh.	Total
MeanFlow	12/20	11/20	13/20	4/20	5/20	45/100
FQL	13/20	14/20	15/20	5/20	4/20	51/100
DSRL	12/20	15/20	13/20	6/20	4/20	50/100
Ours	15/20	17/20	16/20	9/20	8/20	65/100

Table 2: Real-world success counts over 20 hardware rollouts per task. MeanFlow is the frozen behavior-cloning base used by MF-AWLS.

(a) Decoder adaptation
	DexArt SR
MF-AWLS (ours)	$\mathbf{62.2}$
BC (no post-training)	$-6.7$
Q-max steering	$-6.7$
Unfreeze decoder	$-4.0$
Unfreeze enc. + dec.	$-9.7$

Table 3: DexArt ablations (MeanFlow, clean demonstrations); compact

\Delta

SR values are relative to MF-AWLS (

62.2

). Variants were trained in separate batches under the same protocol.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Task	$\sigma_{\text{OU}}$	$\tilde{A}_{\text{clean}}$	$\tilde{A}_{\text{noisy}}$	$w_{\text{clean}}/w_{\text{noisy}}$	AUROC	$\rho(A,R)$
Laptop	$0.4$	$0.66\pm 0.15$	$0.39\pm 0.17$	$2.26$	$0.89$	$0.47$
Faucet	$0.6$	$0.63\pm 0.15$	$0.39\pm 0.17$	$2.09$	$0.86$	$0.49$
Toilet	$0.6$	$0.66\pm 0.14$	$0.35\pm 0.17$	$2.55$	$0.92$	$0.56$
Bucket	$0.15$	$0.65\pm 0.15$	$0.40\pm 0.18$	$2.10$	$0.85$	$0.46$
DexArt avg.	–	$0.65$	$0.38$	$2.25$	$0.88$	$0.49$

Appendix

Table 4: The IQL critic ranks clean demonstrations above noisy ones (four

50/50

clean/noisy mixed DexArt datasets, MeanFlow backbone; advantages detached). For each task we report the mean scale-normalized advantage

\tilde{A}\in[0,1]

(mean

\pm

std over three seeds) on the clean vs. OU-perturbed noisy subset, the induced weight ratio

w_{\text{clean}}/w_{\text{noisy}}=\exp\!\big(\beta(\bar{\tilde{A}}_{\text{clean}}-\bar{\tilde{A}}_{\text{noisy}})\big)

\beta=3

, the AUROC (empirical rank-order) of separating clean from noisy actions by advantage, and the Spearman correlation

\rho(A,R)

between advantage and demonstrator return. Clean actions receive higher advantage on every task; the separation is strong and roughly uniform across tasks because each per-task

\sigma

is tuned to drive its noisy pool to a similar low quality (

<20\%

success), so

\sigma

reflects task sensitivity to noise, not the quality gap. Raw

A

is in return units (goal-conditioned,

\gamma=0.99

);

\tilde{A}

is its per-task min–max normalization, so the

[0,1]

scale is the per-task advantage range by construction. The bottom row is the per-task mean of each column.

Figure 5: The IQL critic ranks clean demonstrations above noisy ones (four

50/50

mixed DexArt datasets, MeanFlow backbone). Top: per-task histograms of the scale-normalized advantage

\tilde{A}\in[0,1]

on clean (green) vs. OU-perturbed noisy (orange) demonstrated actions; dashed lines mark the subset means. Clean actions carry systematically higher advantage on every task, and advantage separates the two subsets with empirical rank-order AUROC

0.85

–

0.92

—even though clean/noisy provenance is a label the critic never observes during training. Bottom: the detached advantage

A

(return units) vs. the demonstrator’s episode return

R

, with the linear trend; advantage is positively correlated with return on every task (Spearman

\rho=0.46

–

0.56

). Summary statistics match Table 4 ; histograms and scatters pool all demonstrated pairs over the three seeds.

Category	Detour	Hesitation	Jerk/abrupt
In $\mathrm{Low}$ - $A$	$626/978$ ( $\mathbf{64.0\%}$ )	$345/484$ ( $\mathbf{71.3\%}$ )	$149/294$ ( $50.7\%$ )

Appendix

Table 5: Advantage–quality alignment on

70

real-world Tube Insertion demonstrations: share of manually flagged steps in each category that fall in the low-advantage half of their trajectory (

50\%

= chance).

	Gaussian	Direct NN	Residual
$J_{T}$	$\operatorname{diag}(\bm{\sigma}_{\phi})$	$J_{f_{\phi}}$	$I+J_{\Delta}$
Nonsingularity	$\min_{i}\sigma_{i}>0$	Requires a condition	If $L_{v}<1$
$\kappa_{2}(J_{T})$	$\max_{i}\sigma_{i}/\min_{i}\sigma_{i}$	Unconstrained	$\leq(1+L_{v})/(1-L_{v})$

Appendix

Table 6: Noise-shaper structure and conditional geometric properties. The residual bounds require

\|J_{\Delta}\|_{\mathrm{op}}\leq L_{v}<1

Figure 6: Residual vs. direct noise shaper on clean DexArt (ablation batch, mean $\pm$ std over three seeds, best checkpoint). The residual parameterization $T_{\phi}=\bm{\epsilon}+\Delta_{\phi}$ matches or exceeds the unconstrained direct map $T_{\phi}=f_{\phi}$ on every task. This comparison supports the residual parameterization empirically; its local geometric measurements are reported separately in Table 11 .

	BC/Base	Direct	Gaussian	Gaussian $+$ reg.	Residual
$D_{\mathrm{rel}}$ $\uparrow$	$1.000$	$5.4{\times}10^{-4}$	$5.2{\times}10^{-5}$	$0.380$	$\mathbf{0.457}$
SR (%) $\uparrow$	$55.2$	$59.8$	$59.5$	$60.2$	$\mathbf{61.9}$

Appendix

Table 7: Latent diversity and success of noise-shaper variants on DexArt (two-seed matched reruns; shared frozen base per task and seed).

D_{\mathrm{rel}}

is the mean pairwise action spread across latent samples relative to the BC backbone (

1

: base,

0

: collapse).

Task	BC-best SR	$S_{J}$ -sel. SR (ours)	Oracle SR	$\Delta_{\text{reg}}$	$\Delta_{\text{BC}}$	$\rho(S_{J},\mathrm{SR})$
Laptop	$79.0$	$86.0\pm 1.4$	$87.0$	$1.0$	$+7.0$	$0.78$
Faucet	$40.0$	$48.0\pm 1.2$	$48.8$	$0.8$	$+8.0$	$0.72$
Toilet	$65.0$	$78.0\pm 0.7$	$79.5$	$1.5$	$+13.0$	$0.84$
Bucket	$32.0$	$36.8\pm 1.9$	$37.2$	$0.4$	$+4.8$	$0.16$
DexArt avg.	$54.0$	$62.2\pm 1.3$	$63.1$	$0.9$	$+8.2$	$0.63$

Appendix

Table 8: Measured checkpoint-selection validation (oracle sweep) on the four DexArt tasks (success rate %, means over three training seeds,

20

evaluation episodes per task per seed). For each task, full AWLS post-training is run at every in-tolerance checkpoint;

k^{\star}

is the oracle (highest AWLS success),

\hat{k}

the

S_{J}

-selected checkpoint (highest

S_{J}

within the

L_{\text{BoM}}

tolerance), and BC-best the highest-BC-quality checkpoint.

\Delta_{\text{reg}}=\mathrm{SR}(k^{\star})-\mathrm{SR}(\hat{k})

is the selection regret,

\Delta_{\text{BC}}=\mathrm{SR}(\hat{k})-\mathrm{SR}(\text{BC-best})

the gain over BC-quality selection, and

\rho

the Spearman correlation between

S_{J}

and AWLS success across the sweep. The

S_{J}

-selected column is the deployed MF-AWLS model (per-task

\pm

: std over seeds); the DexArt-average

\rho

includes the flat-

S_{J}

Bucket task.

	Adroit	DexArt	RoboTwin
Observation modality	point cloud	point cloud	point cloud
Point-cloud points	512	1024	1024
Encoder feature dim	64	64	192
Prediction horizon	16	16	8
Observation steps	2	2	3
Action steps	8	8	6

Appendix

Table 9: Per-benchmark observation and one-step decoder configuration. All benchmarks use point-cloud observations through a shared DP3-style encoder.

Architecture
Network $\Delta_{\phi}$ type	MLP, $\bm{z}=\bm{\epsilon}+\text{scale}\cdot\Delta_{\phi}(\bm{o},\bm{\epsilon})$
Hidden dimensions	$[512,512,512,512]$
Activation / normalization	GELU / LayerNorm
Conditioning input	observation encoding, latent noise $\bm{\epsilon}$
Residual scale	$1.0$
Output initialization	orthogonal (gain $0.1$ ), near-identity start

Appendix

Table 10: Residual noise shaper

T_{\phi}

(

\Delta_{\phi}

) architecture and AWLS post-training configuration.

Residual size	Residual Jacobian $J_{\Delta}$			Transport Jacobian $I+J_{\Delta}$
Rel. norm $\\|\Delta_{\phi}\\|_{2}/\\|\bm{\epsilon}\\|_{2}$	Median $\\|J_{\Delta}\\|_{2}$	$Q_{.95}$ $\\|J_{\Delta}\\|_{2}$	Samples with $\\|J_{\Delta}\\|_{2}<1$	$Q_{.05}$ $\sigma_{\min}(I+J_{\Delta})$	Median $\mathrm{cond}_{2}$
$\approx 0.175$	$0.036$	$0.065$	$3{,}072/3{,}072$	$0.966$	$1.04$

Appendix

Table 11: Measured residual-map statistics of the trained shaper at

3{,}072

evaluated state–noise samples (full SVD of the exact Jacobian

J_{\Delta}=\partial\Delta_{\phi}/\partial\bm{\epsilon}

Inverse temperature $\beta$	$3$
Advantage normalization	per-batch min–max (Eq. ( 6 ))
Normalization constant $\varepsilon$	$10^{-8}$
Normalized advantage range $\tilde{A}$	$[0,1]$
Per-sample weight $w$	$\exp(\beta\tilde{A})$ (mean-normalized)
Weight bounds	$[1,\,e^{\beta}]\approx[1,\,20.09]$
Within-batch weight ratio bound $R$	$e^{\beta}\approx 20.09$

Appendix

Table 12: AWLS advantage-weighting objective hyperparameters.

Q-ensemble size	$2$
Pessimistic aggregation	$\min_{k}Q_{\psi_{k}}$
Value baseline	$V_{\nu}$ , expectile regression
Expectile $\tau$	$0.7$
Discount $\gamma$	$0.99$
CQL penalty weight $\alpha_{\mathrm{CQL}}$	$0.5$
Reward	goal-conditioned binary

Appendix

Table 13: IQL critic architecture and training configuration.

Latent samples per pair $M$ ( $L_{\text{BoM}}$ )	$16$
Perturbation magnitude $h$ ( $S_{J}$ )	$0.01$
Probe directions per state ( $S_{J}$ )	$50$
Validation observations ( $S_{J}$ )	$32$
$L_{\text{BoM}}$ tolerance $\delta$	$0.2$
Checkpoint saving / diagnostic interval	every $50$ epochs
Validation set	offline held-out $\mathcal{D}_{\text{val}}$

Appendix

Table 14: Checkpoint-selection diagnostic hyperparameters.

Method	Generative backbone	NFE	Latency (ms)	Throughput (Hz)
DP3	diffusion (DDIM)	$10$	$17.8$	$56$
DSRL	DP3 $+$ noise steering	$10$	$17.8$	$56$
FlowPolicy	one-step flow	$1$	$3.4$	$294$
FQL	one-step flow	$1$	$3.4$	$294$
MeanFlow	one-step	$1$	$3.4$	$294$
FP-AWLS (ours)	FlowPolicy $+\,T_{\phi}$	$1$	$3.7$	$270$

Appendix

Table 15: Inference cost per action chunk. NFE = neural function evaluations of the decoder; latency is the mean over

1000

calls on a single NVIDIA RTX 4090 (batch size

1

, DexArt configuration:

1024

points, prediction horizon

16

), encoder included; throughput is the implied closed-loop rate. One-step methods are

\approx\!4.8\times

faster than the

10

-step diffusion methods, and AWLS matches its one-step backbone up to the

\approx\!0.3

ms shaper.

Method	Adroit (3 tasks, 10 demos)				DexArt (4 tasks, 100 demos)
	Hammer	Door	Pen	Avg.	Laptop	Faucet	Toilet	Bucket	Avg.
DP3	100.0 $\pm$ 0.0	55.8 $\pm$ 5.7	46.8 $\pm$ 4.2	67.5 $\pm$ 3.3	75.8 $\pm$ 0.8	35.8 $\pm$ 1.5	61.8 $\pm$ 1.1	27.8 $\pm$ 1.3	50.3 $\pm$ 1.2
FlowPolicy	100.0 $\pm$ 0.0	61.0 $\pm$ 3.4	53.0 $\pm$ 2.6	71.3 $\pm$ 2.0	76.6 $\pm$ 3.0	36.4 $\pm$ 2.1	63.4 $\pm$ 1.8	25.4 $\pm$ 1.9	50.4 $\pm$ 2.2
MeanFlow	100.0 $\pm$ 0.0	62.2 $\pm$ 7.1	54.2 $\pm$ 5.4	72.1 $\pm$ 4.2	80.0 $\pm$ 2.2	40.2 $\pm$ 2.5	68.6 $\pm$ 1.8	29.8 $\pm$ 2.2	54.7 $\pm$ 2.2
DSRL	100.0 $\pm$ 0.0	62.6 $\pm$ 4.2	55.0 $\pm$ 3.2	72.5 $\pm$ 2.5	81.4 $\pm$ 3.6	41.8 $\pm$ 4.5	71.4 $\pm$ 3.5	30.8 $\pm$ 3.1	56.4 $\pm$ 3.7
FQL	100.0 $\pm$ 0.0	64.0 $\pm$ 5.1	56.8 $\pm$ 2.6	73.6 $\pm$ 2.6	83.0 $\pm$ 2.2	42.8 $\pm$ 2.3	72.8 $\pm$ 1.8	31.2 $\pm$ 3.1	57.4 $\pm$ 2.4

Appendix

Table 16: Full per-task success rates (%) on the Adroit and DexArt simulation benchmarks. Each entry is mean

\pm

std over three random seeds (

20

evaluation episodes per task per seed); the Avg. column under each benchmark is the per-benchmark task average and matches the benchmark summary (Table 4.1 ). RoboTwin per-task results and the cross-benchmark Overall average are in Table 17 . The Avg. (and Overall)

\pm

values are the average of the per-task standard deviations.

Method	RoboTwin (5 tasks, 50 demos)						Overall Avg.
	Div. Bottles	Dual Bot. H	Empty Cup	Shoe Place	Beat Hammer	Avg.
DP3	32.8 $\pm$ 4.0	38.8 $\pm$ 1.8	59.8 $\pm$ 1.3	38.0 $\pm$ 8.0	55.8 $\pm$ 3.9	45.0 $\pm$ 3.8	52.4 $\pm$ 2.8
FlowPolicy	47.6 $\pm$ 5.0	38.6 $\pm$ 2.2	65.4 $\pm$ 3.0	32.4 $\pm$ 5.9	57.4 $\pm$ 4.8	48.3 $\pm$ 4.2	54.8 $\pm$ 3.0
MeanFlow	52.4 $\pm$ 8.2	42.2 $\pm$ 2.6	69.8 $\pm$ 1.6	39.6 $\pm$ 5.0	62.2 $\pm$ 5.4	53.2 $\pm$ 4.6	58.4 $\pm$ 3.7
DSRL	55.0 $\pm$ 5.3	43.6 $\pm$ 3.1	72.0 $\pm$ 3.2	43.8 $\pm$ 6.6	64.6 $\pm$ 5.9	55.8 $\pm$ 4.8	60.2 $\pm$ 3.9
FQL	56.4 $\pm$ 8.3	44.8 $\pm$ 2.3	74.0 $\pm$ 2.2	39.4 $\pm$ 9.1	66.0 $\pm$ 2.7	56.1 $\pm$ 4.9	60.9 $\pm$ 3.5

Appendix

Table 17: Full per-task success rates (%) on the RoboTwin simulation benchmark. Each entry is mean

\pm

std over three random seeds (

20

evaluation episodes per task per seed). Avg. is the RoboTwin task average and Overall is the cross-task aggregate over all 12 tasks (Adroit, DexArt, and RoboTwin); both match the benchmark summary (Table 4.1 ). Adroit and DexArt per-task results are in Table 16 . The Avg. (and Overall)

\pm

values are the average of the per-task standard deviations.

Variant	BC	Selected	Uniform	Shuffled	Adv-filter	AWLS
SR (%) $\uparrow$	$55.2$	$54.0$	$55.9$	$55.3$	$60.2$	$\mathbf{62.2}$

Appendix

Table 18: Isolating advantage weighting on DexArt (MeanFlow; mean success rate over the four tasks, clean demonstrations). All post-training variants share the same selected frozen base; AWLS is the main-result MF-AWLS.

Figure 7: Advantage-temperature sweep on DexArt (MeanFlow, clean demonstrations; DexArt-average success rate, mean

\pm

std over three seeds). Approximate reconstruction of the submitted sweep figure: marker values are accurate to about

\pm 0.5

points and the shaded

\pm 1

std bands are reproduced approximately. The scale-normalized advantage (ours) is stable across

\beta

, whereas the raw advantage is lower and degrades as

\beta

grows. The star marks

\beta=3

, used in all experiments.

Figure 8: Per-task breakdown of the mixed-quality robustness study. Post-adaptation success rate (%) versus the fraction of low-quality demonstrations in the training mix, shown separately for each of the four DexArt tasks. This is the per-task decomposition of the DexArt-averaged curves reported in main-paper Figure 3 ; each subplot title lists the task’s OU noise scale

\sigma

used to corrupt the low-quality demonstrations (larger

\sigma

= noisier). Across every task and at every contamination level, the AWLS variants (green) degrade most gracefully, with MF-AWLS (thickest, dark green) retaining the highest success rate even at 100% low-quality data, whereas the behavior-cloning backbones (gray/blue) fall off fastest. The Q-maximization baselines (orange) lie in between. The consistent green-over-rest ordering within each panel shows the averaged result is not driven by any single task.

Task	Arms	Episode steps	#Stages
Drawer-Cube	Single	300	3
Tube Insertion	Single	300	2
Screw Scooping	Single	300	2
Gear Packing	Dual	300	3
Ball Weighing	Dual	300	2

Appendix

Table 19: Per-task real-world configuration. Episode steps is the rollout cap (

300

steps

=30

s at

10

Hz); #Stages is the number of ordered stages that must all be completed for a rollout to count as a success.

Method	Drawer	Tube	Screw	Gear	Ball	Agg. (%)
MeanFlow	$60\,[39,78]$	$55\,[34,74]$	$65\,[43,82]$	$20\,[8,42]$	$25\,[11,47]$	$45\,[36,55]$
FQL	$65\,[43,82]$	$70\,[48,85]$	$75\,[53,89]$	$25\,[11,47]$	$20\,[8,42]$	$51\,[41,61]$
DSRL	$60\,[39,78]$	$75\,[53,89]$	$65\,[43,82]$	$30\,[15,52]$	$20\,[8,42]$	$50\,[40,60]$
MF-AWLS (ours)	$\mathbf{75}\,[53,89]$	$\mathbf{85}\,[64,95]$	$\mathbf{80}\,[58,92]$	$\mathbf{45}\,[26,66]$	$\mathbf{40}\,[22,61]$	$\mathbf{65}\,[55,74]$

Appendix

Table 20: Real-world success rate (%) with Wilson score

95\%

confidence intervals, computed from the

20

hardware rollouts per task of Table 2 (

n\!=\!100

in aggregate).

Robot Policy Learning

Robot policy learning develops controllers that map sensory observations, task goals, and interaction history to actions for embodied robots.

See all 369 papers

Sep 10, 2026

Beyond Noise Steering: Dual-Latent Space Reinforcement Learning for Generative Robot Policy

Pretrained generative robot policies learn expressive action priors from demonstrations. However, existing reinforcement learning methods only steer the noisy space but fail to modulate intermediate action representations during the generation process, resulting in performance degradation and inefficiency. To address this limitation, we propose a novel Dual-Latent Space Reinforcement Learning (DLSRL) framework, which complements initial-noise steering with representation-level control inside the frozen generator. Specifically, our actor network predicts two distinct latent variables: an initial-noise latent variable that steers behavior generation, and an action-representation latent variable for intermediate feature modulation. Moreover, this representation latent variable is mapped to adapter features and ingeniously injected into the hidden states of intermediate action tokens via residual connections. Our dual-control design enables direct adjustment of action representations without updating the base policy. Experiments across generative policy architectures and robotic manipulation tasks show that DLSRL effectively accelerates online robot policy adaptation and achieves competitive performance. Our code is available at https://github.com/xianchaoxiu/DLSRL.

Pengfei Zhang, Teng Sun, Xianchao Xiu

School of Mechatronic Engineering and Automation, Shanghai University, Shanghai 200444, China

Latent Action Learning Robot Policy Learning Robotic RL RL for Generative Models

Ask PDF

May 31, 2026

Lagrangian Perturbation Diffusion Steering: Latent Reinforcement Learning for Generative Policies

Behavior cloning with high-capacity generative policies achieves strong imitation performance, but is often limited by demonstration coverage and distribution shift. Direct reinforcement learning fine-tuning can improve performance, but updating large action decoders is frequently unstable and sample inefficient. We propose Lagrangian Perturbation Diffusion Steering (LP-DS), a lightweight adaptation method that improves a frozen generative policy by learning a compact noise-space perturbation before decoding. LP-DS optimizes this perturbation with a Lagrangian trust-region objective, improving downstream value while constraining deviation from the latent prior. Across RoboMimic manipulation, OpenAI Gym locomotion, and Adroit dexterous manipulation benchmarks, LP-DS improves sample efficiency, success, and return while maintaining higher action-space entropy than unconstrained noise-space steering, with return improvements of up to 25% over prior baselines. Additional evaluations with flow-matching backbones, a large vision-language-action model, and physical Franka deployment show that LP-DS is not limited to compact diffusion policies or simulated benchmarks. Project page: https://sites.google.com/view/lp-ds/home.

Hikmet Simsir, Ozgur S. Oguz

Department of Computer Engineering, Bilkent University, Ankara, Türkiye.

Robot Policy Adaptation Reinforcement Learning Constrained RL Robotic RL Diffusion Policy RL for Diffusion Models+2

Ask PDF

Mar 5, 2026

Latent Policy Steering through One-Step Flow Policies

Offline reinforcement learning (RL) allows robots to learn from offline datasets without risky exploration. Yet, offline RL's performance often hinges on a brittle trade-off between (1) return maximization, which can push policies outside the dataset support, and (2) behavioral constraints, which typically require sensitive hyperparameter tuning. Latent steering offers a structural way to stay within the dataset support during RL, but existing offline adaptations commonly approximate action values using latent-space critics learned via indirect distillation, which can lose information and hinder convergence. We propose Latent Policy Steering (LPS), which enables high-fidelity latent policy improvement by backpropagating original-action-space Q-gradients through a differentiable one-step MeanFlow policy to update a latent-action-space actor. By eliminating proxy latent critics, LPS allows an original-action-space critic to guide end-to-end latent-space optimization, while the one-step MeanFlow policy serves as a behavior-constrained generative prior. This decoupling yields a robust method that works out-of-the-box with minimal tuning. Across OGBench and real-world robotic tasks, LPS achieves state-of-the-art performance and consistently outperforms behavioral cloning and strong latent steering baselines.

Hokyun Im, Andrey Kolobov, Jianlong Fu +1

Department of Artificial Intelligence, Yonsei University · Microsoft Research

RL for Robotics Latent Action Learning Offline RL

Ask PDF

Abstract

Robot Policy Learning

Explore similar work

Beyond Noise Steering: Dual-Latent Space Reinforcement Learning for Generative Robot Policy

Lagrangian Perturbation Diffusion Steering: Latent Reinforcement Learning for Generative Policies

Latent Policy Steering through One-Step Flow Policies