Pretrained vision-language-action (VLA) models encode broad manipulation priors, yet adapting them to specific tasks requires balancing limited robot interaction, computational resources, and safety constraints. Noise space learning (NSL) adapts flow-matching-based VLAs by learning a context-conditioned distribution over noise while keeping the pretrained VLA fixed. Building on prior noise-steering methods, we formulate NSL as an interface for VLA adaptation that accommodates different learning algorithms and instantiate it with PPO, SAC, and AWR. Theoretically, we provide an information-theoretic analysis of how NSL enables efficient adaptation. Empirically, we study these NSL algorithms on LIBERO, RoboTwin, and four real-robot manipulation tasks under interaction, computation, and execution constraints. Across these settings, we develop a suite of effective NSL variants and derive practical recommendations for choosing among them under different resource and deployment regimes.
Figures & tables
Figure 1: Different VLA adaptation approaches for a deterministic generator. NSL learns a context-conditioned noise distribution, generator adaptation updates the noise-to-action map, and residual adaptation modifies generated actions before execution. Orange modules are trainable; blue modules are fixed.
Figure 2: Noise space learning on LIBERO-10 with SmolVLA and π0.5 under Full and One demonstration settings. Curves show success rates over 500 evaluation episodes at each epoch; dashed lines denote the corresponding base-policy references. Values below each panel report the best observed success rate within 60 epochs.
Model / metric
Base policy
Action-AWR
RLT
NSL-AWR
NSL-PPO
NSL-SAC
SmolVLA-Full SR (%)
50.6
93.0
94.2
94.2
95.8
95.8
π0.5 -One SR (%)
40.4
65.8
67.8
64.0
67.4
71.2
Avg. SR (%)
45.5
79.4
81.0
79.1
81.6
83.5
Update time (s/batch)
—
210.20
111.60
1.09
2.34
9.93
Table 1: Performance and update time on two model settings. Adapted-policy SR is the best observed within 60 epochs, with 500 evaluation episodes per epoch. Avg. is the equal-weight mean of the two model settings; Base policy gives the unadapted reference. Time is the gradient-update time per collection batch averaged over both models, excluding collection and evaluation.
Figure 3: Effect of update budget, averaged equally over SmolVLA-Full and π0.5 -One. Top: 1× , 2× , and 4× update budgets within each algorithm. Bottom: algorithms compared at each update budget. The rollout protocol is fixed. Dashed lines show the mean base-policy success rate (45.5%).
Figure 4: Effect of rollout count, averaged equally over SmolVLA-Full and π0.5 -One. Top: one, four, and eight rollouts per task–reset pair within each algorithm. Bottom: algorithms compared at each rollout count. Update ratios remain AWR 1× , PPO 2× , and SAC 4× . Dashed lines show the mean base-policy success rate (45.5%).
Model
Method
Baseline (%)
w/o Noise Limit
w/o Task Balancing
w/o Success/Failure Balancing
All Removed
SmolVLA Full
SAC
93.4
32.0(−61.4)
80.6(−12.8)
85.2(−8.2)
0.0(−93.4)
PPO
92.2
92.2(0.0)
87.2(−5.0)
91.8(−0.4)
84.0(−8.2)
AWR
87.2
87.2(−0.0)
87.2(−0.0)
87.0(−0.2)
90.2(+3.0)
π0.5 -One
SAC
66.6
56.6(−10.0)
60.4(−6.2)
61.8(−4.8)
39.8(−26.8)
PPO
58.2
58.2(0.0)
50.4(−7.8)
58.0(−0.2)
52.4(−5.8)
AWR
59.0
59.0(−0.0)
61.6(+2.6)
58.4(−0.6)
60.2(+1.2)
Table 2: Component ablations. Values are the best success rates (%) within the first 30 epochs, with changes from the corresponding baseline in parentheses (pp). PPO and AWR already use unbounded noise, so their no-limit entries reuse the baseline.
Figure 5: Real-robot tasks. Arrows indicate operations, and numbers denote their order. The inset shows a completed bead-placement goal from a successful demonstration.
Task (SR, %)
Initial policy
Autonomous
Human-guided
Action-AWR
RLT
NSL-AWR
DAgger
UniSteer
NSL-AWR + human
Cup stacking
75.0
100.0
100.0
100.0
100.0
100.0
100.0
Toy fishing
55.0
70.0
75.0
80.0
70.0
90.0
90.0
Sequential insertion
15.0
50.0
75.0
70.0
70.0
85.0
90.0
Bead placement
5.0
15.0
30.0
35.0
25.0
20.0
55.0
Avg.
37.5
58.8
70.0
71.3
66.3
73.8
83.8
Table 3: Real-robot success rates over 40 trials per task and training time. Avg. SR and cumulative model-update time are averaged over the four tasks.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
NSL-AWR
NSL-PPO
NSL-SAC
Actor learning rate
10−5
10−5
10−5
Value/critic learning rate
2×10−5
2×10−5
2×10−5
Training minibatch size
2,048
2,048
2,048
Discount per chunk
0.99
0.99
0.99
Replay success sampling fraction
0.5
0.5
0.5
Random seed
42
42
42
Appendix
Table 4: Hyperparameters for the main LIBERO experiments. SAC learns its entropy temperature, initialized at 0.006 , with Adam at learning rate 3×10−4 . Its target entropy is specified per noise dimension.
Method
Main update multiplier
Actor updates
Value/critic updates
NSL-AWR
1×
100
200
NSL-PPO
2×
200
400
NSL-SAC
4×
400
800
Appendix
Table 5: Scheduled updates for a collection batch containing 20,480 new chunk transitions. Counts scale with the actual number of collected transitions. Each SAC critic update jointly optimizes both Q networks.
Task
Demonstrations
SFT steps
Cup stacking
25
30,000
Toy fishing
35
30,000
Sequential insertion
35
50,000
Bead placement
20
140,000
Appendix
Table 6: Task-specific π0.5 SFT data and training steps. Bead placement uses 20 complete demonstration trajectories.
Figure 6: Mean trajectories collected per adaptation round. Bars separate model and human trajectories; labels above bars give totals, and side labels give human counts. Autonomous denotes the shared schedule for Action-AWR, RLT, and NSL-AWR.
Figure 7: NSL-AWR actor-mean evolution on 500 fixed LIBERO observations. Rows show the four backbone–data settings; colors denote tasks. Each point is a projected 32-dimensional conditional actor mean. Each model uses one PCA basis fitted to all 30,500 means from 61 checkpoints, with fixed axis limits across its panels. Axis percentages give the explained variance over those checkpoints. The 500 initial means overlap and are shown as one black point.
Figure 8: NSL-AWR output statistics on the same fixed observations. Mean shift and epoch-to-epoch drift average per-observation L2 distances of actor means relative to initialization and the preceding checkpoint, respectively. Actor standard deviation is averaged over observations and noise coordinates. Between-task variance is the fraction of total actor-mean squared variation explained by task centroids, computed in the original 32-dimensional space. Curves use all 61 checkpoints without smoothing; the initial drift and between-task fraction are undefined and omitted.
Figure 9: Effect of the noise limit on NSL-SAC with SmolVLA-Full. Left: mean L2 displacement of the underlying Gaussian mean from initialization over 500 fixed contexts per run, measured before any tanh transformation. Right: mean minibatch-maximum Bellman target, weighted by the number of critic updates. The dotted line marks the return upper bound of 1 under terminal binary rewards without entropy backup.
Figure 10: SAC training diagnostics for real-robot bead placement. (a) Mean and maximum absolute deterministic noise output 5tanh(μ) over minibatch contexts and coordinates; the dotted line marks the amplitude limit. (b) Mean absolute twin-critic disagreement for actor-proposed and replay noise. (c) Logged actor-proposal minus replay Q value, using the arithmetic mean of the two critics for both. Thin lines show minibatch statistics; thick lines show trailing 50-update means. The Q difference compares the separately logged actor and replay evaluations.
Setting
Method
SFT
Epoch 30
Δ
Best (epoch)
Easy
NSL-AWR 4×
31.50
73.50
+42.00
73.50 (30)
NSL-PPO 2×
31.50
70.50
+39.00
73.50 (29)
NSL-SAC 4×
31.50
84.25
+52.75
84.25 (30)
Hard
NSL-AWR 4×
32.75
65.00
+32.25
67.00 (29)
NSL-PPO 2×
32.75
69.25
+36.50
72.25 (27)
NSL-SAC 4×
32.75
77.25
+44.50
78.50 (26)
Appendix
Table 7: RoboTwin main results with R=8 . SR (%) is the equal-weight mean over all eight tasks at the same epoch. Δ is epoch-30 SR minus SFT, in percentage points. Best is the highest mean SR within epochs 1–30; parentheses give its first occurrence.
Easy
Hard
Task
SFT
AWR
PPO
SAC
SFT
AWR
PPO
SAC
handover_block
10.0
62.0
62.0
96.0
10.0
40.0
56.0
86.0
move_can_pot
4.0
66.0
60.0
86.0
8.0
60.0
58.0
78.0
place_a2b_left
20.0
38.0
34.0
40.0
20.0
34.0
34.0
26.0
place_phone_stand
12.0
64.0
62.0
76.0
2.0
50.0
62.0
62.0
pick_dual_bottles
58.0
96.0
88.0
90.0
60.0
88.0
94.0
86.0
Appendix
Table 8: Per-task SR (%) on RoboTwin Easy and Hard at epoch 30. All runs use R=8 , with update multipliers of 4× for NSL-AWR, 2× for NSL-PPO, and 4× for NSL-SAC. AWR, PPO, and SAC columns denote the NSL methods. Bold marks the best entry within each difficulty block for each row, including ties.
Method
U=1×
U=2×
U=4×
U=8×
NSL-AWR
63.75
70.75
68.75
68.25
NSL-PPO
69.25
72.25
65.50
50.50
NSL-SAC
63.75
71.50
80.75
77.00
Appendix
Table 9: Update-budget ablation on RoboTwin Easy. Entries are eight-task mean SR (%) at epoch 20 with R=8 . Bold indicates the best multiplier for each method.
Method
R=1
R=4
R=8
Δ
NSL-AWR
61.00
67.50
68.75
+7.75
NSL-PPO
55.50
68.00
72.25
+16.75
NSL-SAC
60.25
71.00
80.75
+20.50
Appendix
Table 10: Rollout-count ablation on RoboTwin Easy at epoch 20. Entries are eight-task mean SR (%). Update multipliers remain at 4× , 2× , and 4× for AWR, PPO, and SAC, respectively. Δ is the increase from R=1 to R=8 , in percentage points.
Diffusion-based vision-language-action (VLA) models have emerged as strong priors for robotic manipulation, yet adapting them to real-world distributions remains challenging. In particular, on-robot reinforcement learning (RL) is expensive and time-consuming, so effective adaptation depends on efficient policy improvement within a limited budget of real-world interactions. Noise-space RL lowers the cost by keeping the pretrained VLA fixed as a denoising generator while updating only a lightweight actor that predicts the noise. However, its performance is still limited due to inefficient autonomous exploration. Human corrective interventions can reduce this exploration burden, but they are naturally provided in action space, whereas noise-space finetuning requires supervision over noise variables. To address these challenges, we propose UniSteer, a Unified Noise Steering framework that combines human corrective guidance with noise-space RL through approximate action-to-noise inversion. Given a human corrective action, UniSteer inverts the frozen flow-matching decoder to recover a noise target, which provides supervised guidance for the same noise actor that is simultaneously optimized via reinforcement learning. Real-world experiments on diverse manipulation tasks show that UniSteer adapts more efficiently than strong noise-space RL and action-space human-in-the-loop baselines, improving the success rate from 20% to 90% in 66 minutes on average across four real-world adaptation tasks.
Junjie Lu, Xinyao Qin, Yuhua Jiang +6
University of Technology Sydney · Work conducted at Microsoft Research. · Tsinghua University +1
Large-scale pretraining has made Vision-Language-Action (VLA) models promising foundations for generalist robot manipulation, yet adapting them to downstream tasks remains necessary. However, the common practice of full fine-tuning treats pretraining as initialization and can shift broad priors toward narrow training-distribution patterns. We propose PriorVLA, a novel framework that preserves pretrained priors and learns to leverage them for effective adaptation. PriorVLA keeps a frozen Prior Expert as a read-only prior source and trains an Adaptation Expert for downstream specialization. Expert Queries capture scene priors from the pretrained VLM and motor priors from the Prior Expert, integrating both into the Adaptation Expert to guide adaptation. Together, PriorVLA updates only 25% of the parameters updated by full fine-tuning. Across RoboTwin 2.0, LIBERO, and real-world tasks, PriorVLA achieves stronger overall performance than full fine-tuning and state-of-the-art VLA baselines, with the largest gains under out-of-distribution (OOD) and few-shot settings. PriorVLA improves over pi0.5 by 11 points on RoboTwin 2.0-Hard and achieves 99.1% average success on LIBERO. Across eight real-world tasks and two embodiments, PriorVLA reaches 81% in-distribution (ID) and 57% OOD success with standard data. With only 10 demonstrations per task, PriorVLA reaches 48% ID and 32% OOD success, surpassing pi0.5 by 24 and 22 points, respectively.
Xinyu Guo, Bin Xie, Wei Chai +4
Institute of Automation, Chinese Academy of Sciences · Dexmal · Nanjing University of Aeronautics and Astronautics +1
Vision-Language-Action (VLA) models provide a versatile foundation for general robotic manipulation, yet they exhibit significant brittleness when confronted with novel task distributions. While In-Context Imitation Learning (ICIL) offers a training-free alternative, existing frameworks suffer from an adaptation bottleneck that hinders the effective translation of expert context to executable actions. This failure originates from superficial retrieval mechanisms and an inherent behavioral inertia that anchors the policy to its pre-trained priors. To address these limitations, we present RA-VLA, a retrieval-augmented VLA framework that integrates behavior-aligned context retrieval with a grounded execution pipeline. By enforcing faithful adherence to functional cues within a scalable architecture, RA-VLA facilitates seamless task adaptation while preserving inference efficiency. Our empirical evaluations across the LIBERO benchmark and a real-world UR5e environment demonstrate that RA-VLA achieves superior success rates and computational efficiency, establishing a robust framework for training-free robotic adaptation.
Sanghwan Jang, Minjin Jeon, Minsoo Kim +3
Department of Computer Science and Engineering, POSTECH, Pohang, South Korea · Graduate School of Artificial Intelligence, POSTECH, Pohang, South Korea