Noise Space Learning for VLA Adaptation

Junjie Lu; Chuheng Zhang; Yucen Wang; Yuhua Jiang; Xinyao Qin; Min Xu; Li Zhao

cs.ROOct 9, 2026

Noise Space Learning for VLA Adaptation

Authors: Junjie Lu, Chuheng Zhang, Yucen Wang, Yuhua Jiang, Xinyao Qin, Min Xu, Li Zhao

Organizations: University of Technology Sydney · Microsoft Research · Nanjing University · Tsinghua University

Abstract

Pretrained vision-language-action (VLA) models encode broad manipulation priors, yet adapting them to specific tasks requires balancing limited robot interaction, computational resources, and safety constraints. Noise space learning (NSL) adapts flow-matching-based VLAs by learning a context-conditioned distribution over noise while keeping the pretrained VLA fixed. Building on prior noise-steering methods, we formulate NSL as an interface for VLA adaptation that accommodates different learning algorithms and instantiate it with PPO, SAC, and AWR. Theoretically, we provide an information-theoretic analysis of how NSL enables efficient adaptation. Empirically, we study these NSL algorithms on LIBERO, RoboTwin, and four real-robot manipulation tasks under interaction, computation, and execution constraints. Across these settings, we develop a suite of effective NSL variants and derive practical recommendations for choosing among them under different resource and deployment regimes.

Figures & tables

Figure 1: Different VLA adaptation approaches for a deterministic generator. NSL learns a context-conditioned noise distribution, generator adaptation updates the noise-to-action map, and residual adaptation modifies generated actions before execution. Orange modules are trainable; blue modules are fixed.

Figure 2: Noise space learning on LIBERO-10 with SmolVLA and

\pi_{0.5}

under Full and One demonstration settings. Curves show success rates over 500 evaluation episodes at each epoch; dashed lines denote the corresponding base-policy references. Values below each panel report the best observed success rate within 60 epochs.

Model / metric	Base policy	Action-AWR	RLT	NSL-AWR	NSL-PPO	NSL-SAC
SmolVLA-Full SR (%)	50.6	93.0	94.2	94.2	95.8	95.8
$\pi_{0.5}$ -One SR (%)	40.4	65.8	67.8	64.0	67.4	71.2
Avg. SR (%)	45.5	79.4	81.0	79.1	81.6	83.5
Update time (s/batch)	—	210.20	111.60	1.09	2.34	9.93

Table 1: Performance and update time on two model settings. Adapted-policy SR is the best observed within 60 epochs, with 500 evaluation episodes per epoch. Avg. is the equal-weight mean of the two model settings; Base policy gives the unadapted reference. Time is the gradient-update time per collection batch averaged over both models, excluding collection and evaluation.

Figure 3: Effect of update budget, averaged equally over SmolVLA-Full and

\pi_{0.5}

-One. Top:

1\times

2\times

, and

4\times

update budgets within each algorithm. Bottom: algorithms compared at each update budget. The rollout protocol is fixed. Dashed lines show the mean base-policy success rate (45.5%).

Figure 4: Effect of rollout count, averaged equally over SmolVLA-Full and

\pi_{0.5}

-One. Top: one, four, and eight rollouts per task–reset pair within each algorithm. Bottom: algorithms compared at each rollout count. Update ratios remain AWR

1\times

, PPO

2\times

, and SAC

4\times

. Dashed lines show the mean base-policy success rate (45.5%).

Model	Method	Baseline (%)	w/o Noise Limit	w/o Task Balancing	w/o Success/Failure Balancing	All Removed
SmolVLA Full	SAC	93.4	$32.0\;(-61.4)$	$80.6\;(-12.8)$	$85.2\;(-8.2)$	$0.0\;(-93.4)$
	PPO	92.2	$92.2\;(0.0)$	$87.2\;(-5.0)$	$91.8\;(-0.4)$	$84.0\;(-8.2)$
	AWR	87.2	$87.2\;(-0.0)$	$87.2\;(-0.0)$	$87.0\;(-0.2)$	$90.2\;(+3.0)$
$\pi_{0.5}$ -One	SAC	66.6	$56.6\;(-10.0)$	$60.4\;(-6.2)$	$61.8\;(-4.8)$	$39.8\;(-26.8)$
	PPO	58.2	$58.2\;(0.0)$	$50.4\;(-7.8)$	$58.0\;(-0.2)$	$52.4\;(-5.8)$
	AWR	59.0	$59.0\;(-0.0)$	$61.6\;(+2.6)$	$58.4\;(-0.6)$	$60.2\;(+1.2)$

Table 2: Component ablations. Values are the best success rates (%) within the first 30 epochs, with changes from the corresponding baseline in parentheses (pp). PPO and AWR already use unbounded noise, so their no-limit entries reuse the baseline.

Figure 5: Real-robot tasks. Arrows indicate operations, and numbers denote their order. The inset shows a completed bead-placement goal from a successful demonstration.

Task (SR, %)	Initial policy	Autonomous			Human-guided
Task (SR, %)	Initial policy	Action-AWR	RLT	NSL-AWR	DAgger	UniSteer	NSL-AWR + human
Cup stacking	75.0	100.0	100.0	100.0	100.0	100.0	100.0
Toy fishing	55.0	70.0	75.0	80.0	70.0	90.0	90.0
Sequential insertion	15.0	50.0	75.0	70.0	70.0	85.0	90.0
Bead placement	5.0	15.0	30.0	35.0	25.0	20.0	55.0
Avg.	37.5	58.8	70.0	71.3	66.3	73.8	83.8

Table 3: Real-robot success rates over 40 trials per task and training time. Avg. SR and cumulative model-update time are averaged over the four tasks.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	NSL-AWR	NSL-PPO	NSL-SAC
Actor learning rate	$10^{-5}$	$10^{-5}$	$10^{-5}$
Value/critic learning rate	$2\times 10^{-5}$	$2\times 10^{-5}$	$2\times 10^{-5}$
Training minibatch size	2,048	2,048	2,048
Discount per chunk	0.99	0.99	0.99
Replay success sampling fraction	0.5	0.5	0.5
Random seed	42	42	42

Appendix

Table 4: Hyperparameters for the main LIBERO experiments. SAC learns its entropy temperature, initialized at

0.006

, with Adam at learning rate

3\times 10^{-4}

. Its target entropy is specified per noise dimension.

Method	Main update multiplier	Actor updates	Value/critic updates
NSL-AWR	$1\times$	100	200
NSL-PPO	$2\times$	200	400
NSL-SAC	$4\times$	400	800

Appendix

Table 5: Scheduled updates for a collection batch containing 20,480 new chunk transitions. Counts scale with the actual number of collected transitions. Each SAC critic update jointly optimizes both Q networks.

Task	Demonstrations	SFT steps
Cup stacking	25	30,000
Toy fishing	35	30,000
Sequential insertion	35	50,000
Bead placement	20	140,000

Appendix

Table 6: Task-specific

\pi_{0.5}

SFT data and training steps. Bead placement uses 20 complete demonstration trajectories.

Figure 6: Mean trajectories collected per adaptation round. Bars separate model and human trajectories; labels above bars give totals, and side labels give human counts. Autonomous denotes the shared schedule for Action-AWR, RLT, and NSL-AWR.

Figure 7: NSL-AWR actor-mean evolution on 500 fixed LIBERO observations. Rows show the four backbone–data settings; colors denote tasks. Each point is a projected 32-dimensional conditional actor mean. Each model uses one PCA basis fitted to all 30,500 means from 61 checkpoints, with fixed axis limits across its panels. Axis percentages give the explained variance over those checkpoints. The 500 initial means overlap and are shown as one black point.

Figure 8: NSL-AWR output statistics on the same fixed observations. Mean shift and epoch-to-epoch drift average per-observation L2 distances of actor means relative to initialization and the preceding checkpoint, respectively. Actor standard deviation is averaged over observations and noise coordinates. Between-task variance is the fraction of total actor-mean squared variation explained by task centroids, computed in the original 32-dimensional space. Curves use all 61 checkpoints without smoothing; the initial drift and between-task fraction are undefined and omitted.

Figure 9: Effect of the noise limit on NSL-SAC with SmolVLA-Full. Left: mean L2 displacement of the underlying Gaussian mean from initialization over 500 fixed contexts per run, measured before any tanh transformation. Right: mean minibatch-maximum Bellman target, weighted by the number of critic updates. The dotted line marks the return upper bound of 1 under terminal binary rewards without entropy backup.

Figure 10: SAC training diagnostics for real-robot bead placement. (a) Mean and maximum absolute deterministic noise output

5\tanh(\mu)

over minibatch contexts and coordinates; the dotted line marks the amplitude limit. (b) Mean absolute twin-critic disagreement for actor-proposed and replay noise. (c) Logged actor-proposal minus replay Q value, using the arithmetic mean of the two critics for both. Thin lines show minibatch statistics; thick lines show trailing 50-update means. The Q difference compares the separately logged actor and replay evaluations.

Setting	Method	SFT	Epoch 30	$\Delta$	Best (epoch)
Easy	NSL-AWR $4\times$	31.50	73.50	$+42.00$	73.50 (30)
	NSL-PPO $2\times$	31.50	70.50	$+39.00$	73.50 (29)
	NSL-SAC $4\times$	31.50	84.25	$+52.75$	84.25 (30)
Hard	NSL-AWR $4\times$	32.75	65.00	$+32.25$	67.00 (29)
	NSL-PPO $2\times$	32.75	69.25	$+36.50$	72.25 (27)
	NSL-SAC $4\times$	32.75	77.25	$+44.50$	78.50 (26)

Appendix

Table 7: RoboTwin main results with

R=8

. SR (%) is the equal-weight mean over all eight tasks at the same epoch.

\Delta

is epoch-30 SR minus SFT, in percentage points. Best is the highest mean SR within epochs 1–30; parentheses give its first occurrence.

	Easy				Hard
Task	SFT	AWR	PPO	SAC	SFT	AWR	PPO	SAC
handover_block	10.0	62.0	62.0	96.0	10.0	40.0	56.0	86.0
move_can_pot	4.0	66.0	60.0	86.0	8.0	60.0	58.0	78.0
place_a2b_left	20.0	38.0	34.0	40.0	20.0	34.0	34.0	26.0
place_phone_stand	12.0	64.0	62.0	76.0	2.0	50.0	62.0	62.0
pick_dual_bottles	58.0	96.0	88.0	90.0	60.0	88.0	94.0	86.0

Appendix

Table 8: Per-task SR (%) on RoboTwin Easy and Hard at epoch 30. All runs use

R=8

, with update multipliers of

4\times

for NSL-AWR,

2\times

for NSL-PPO, and

4\times

for NSL-SAC. AWR, PPO, and SAC columns denote the NSL methods. Bold marks the best entry within each difficulty block for each row, including ties.

Method	$U=1\times$	$U=2\times$	$U=4\times$	$U=8\times$
NSL-AWR	63.75	70.75	68.75	68.25
NSL-PPO	69.25	72.25	65.50	50.50
NSL-SAC	63.75	71.50	80.75	77.00

Appendix

Table 9: Update-budget ablation on RoboTwin Easy. Entries are eight-task mean SR (%) at epoch 20 with

R=8

. Bold indicates the best multiplier for each method.

Method	$R=1$	$R=4$	$R=8$	$\Delta$
NSL-AWR	61.00	67.50	68.75	$+7.75$
NSL-PPO	55.50	68.00	72.25	$+16.75$
NSL-SAC	60.25	71.00	80.75	$+20.50$

Appendix

Table 10: Rollout-count ablation on RoboTwin Easy at epoch 20. Entries are eight-task mean SR (%). Update multipliers remain at

4\times

2\times

, and

4\times

for AWR, PPO, and SAC, respectively.

\Delta

is the increase from

R=1

R=8

, in percentage points.

Explore similar work

UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptation
May 11, 2026Junjie Lu, Xinyao Qin, Yuhua Jiang +6RL for Robotics Robot Policy Adaptation
University of Technology Sydney · Work conducted at Microsoft Research. · Tsinghua University +1
PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models
May 11, 2026Xinyu Guo, Bin Xie, Wei Chai +4Robot Policy Adaptation Robot Skill Learning
Institute of Automation, Chinese Academy of Sciences · Dexmal · Nanjing University of Aeronautics and Astronautics +1
RA-VLA: Retrieval-Augmented VLA for Test-Time Adaptation
Aug 26, 2026Sanghwan Jang, Minjin Jeon, Minsoo Kim +3Test-Time Adaptation of VLA Models Vision-Language-Action Models
Department of Computer Science and Engineering, POSTECH, Pohang, South Korea · Graduate School of Artificial Intelligence, POSTECH, Pohang, South Korea

May 11, 2026, cs.ROJ/K move · Enter open · S save

UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptation

Junjie Lu, Xinyao Qin, Yuhua Jiang, Kaixin Wang+5

University of Technology Sydney · Work conducted at Microsoft Research. · Tsinghua University+1

RL for Robotics Robot Policy Adaptation Reinforcement Learning VLM Adaptation Human-in-the-Loop AI Robotic Manipulation+1

Open paper AskPDF arXiv

Diffusion-based vision-language-action (VLA) models have emerged as strong priors for robotic manipulation, yet adapting them to real-world distributions remains challenging. In particular, on-robot reinforcement learning (RL) is expensive and time-consuming, so effective adaptation depends on efficient policy improvement within a limited budget of real-world interactions. Noise-space RL lowers the cost by keeping the pretrained VLA fixed as a denoising generator while updating only a lightweight actor that predicts the noise. However, its performance is still limited due to inefficient autonomous exploration. Human corrective interventions can reduce this exploration burden, but they are naturally provided in action space, whereas noise-space finetuning requires supervision over noise variables. To address these challenges, we propose UniSteer, a Unified Noise Steering framework that combines human corrective guidance with noise-space RL through approximate action-to-noise inversion. Given a human corrective action, UniSteer inverts the frozen flow-matching decoder to recover a noise target, which provides supervised guidance for the same noise actor that is simultaneously optimized via reinforcement learning. Real-world experiments on diverse manipulation tasks show that UniSteer adapts more efficiently than strong noise-space RL and action-space human-in-the-loop baselines, improving the success rate from 20% to 90% in 66 minutes on average across four real-world adaptation tasks.

Abstract

Explore similar work

UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptation

PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models

RA-VLA: Retrieval-Augmented VLA for Test-Time Adaptation