LoRN: Low-Rank Noise Steering for Generative Robot Policies
Organizations: NVIDIA
Abstract
Noise steering improves pretrained generative robot policies by learning to select their initial noise without updating model weights. We find that actions in the evaluated world-action models are more sensitive to video noise than action noise, making video noise an expressive input for steering. However, the noise-steering methods we evaluate, in their published forms, struggle to scale to the high-dimensional noise inputs of large policy classes such as world-action models. We introduce Low-Rank Noise Steering (LoRN), a reinforcement learning method that scales noise steering to world-action models. A small steering actor selects a direction through a fixed low-rank random projection, while our norm-controlled construction keeps the resulting noise near the typical scale of a Gaussian sample. This separates the actor's control dimension from the full noise dimension, enabling steering of both video and action inputs. Across three world-action models and ten model-task pairs, video steering improves average success by 26.2 percentage points over the base policy, compared with 7.1 points for DSRL action steering. For smaller action policies, the same construction achieves comparable final success while generally improving early-training success. Steering actors trained in simulation also transfer to three physical bimanual manipulation tasks across two base policies, without additional real-world training.
Figures & tables
| Task | Policy | Simulation | Robot |
| Cube to bowl | Base policy | 19.2 | 0/10 |
| DSRL | 29.7 10.4 | 4/10 | |
| LoRN | 43.8 3.7 | 6/10 | |
| Marker to cup | Base policy | 3.8 | 0/10 |
| DSRL | 28.2 7.6 | 4/10 | |
| LoRN | 39.6 3.5 | 4/10 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Policy / noise input | Basis | Projection seed(s) | |||
| Cosmos3-Edge video | 506,880 | 32 | Random | 1 | 1234–1236, one per learner seed |
| DreamZero-14B video | 112,640 | 32 | Random | 1 | 1234; action 4321 |
| DreamZero-5B video | 25,600 | 32 | Frame-constant | 1 | 1234; action 4321 |
| LIBERO / ALOHA | 1,600 | 16 | Tiled | 0.5 | 4321, all learner seeds |
| 960 | 32 | Random | 1 | 4321, all learner seeds | |
| Diffusion Policy | 28 | 28 | Random | 0.5 | 4321, all learner seeds |
| Setting | Robomimic | Cosmos / DreamZero-14B | DreamZero-5B | GR00T N1.7 (YAM station) | ||
| Implementation | jaxrl2 SAC | SB3 DSRL-NA | PyTorch SAC | PyTorch SAC | PyTorch SAC | PyTorch SAC |
| Hidden layers | ||||||
| Critics; aggregation | 10; mean / mean | 2; min / min | 10; mean / mean | 10; mean / mean | 4; mean of 2 / min | 4; mean of 2 / min |
| Learning rates | / / | for all | / / | / / | / / | / / – |
| Batch size | 256 | 256 | 768 | 512 | 64 | 64 |
| Replay capacity | 25k / 150k | 1,900 (recent decisions only) | 48,000 | 200,000 | 200,000 |
| Policy | Training budget | Seeds | Reported evaluation per seed |
| Cosmos3-Edge | 1.5M env. steps; Marker 1.0M | 3 | 240 episodes at the specified checkpoint |
| DreamZero-14B | 200k env. steps | 3 | 120 paired episodes near the end of training |
| DreamZero-5B | 614,400 env. steps | 3 | Mean of the last three 25-episode evaluations |
| 500k gradient updates (LIBERO); 3M (ALOHA) | 5 | Mean of the last ten smoothed evaluations ( Section B.5 ) | |
| 1M env. steps | 5 | Mean of the last ten evaluations | |
| Diffusion Policy | 12 h (Lift/Can); 30 h (Square) | 5 | Mean of the last ten 200-episode evaluations |
| Policy | Tasks | Episode limit | Reset randomization | Reward | In-training evaluation |
| Cosmos3-Edge | RoboLab MustardInLeftBin, MarkerInMug, ButterAboveRaisin, StackYellowOnRed | 448, 600, 576, 896 | Object positions uniform within cm, 0.25 m separation floor | on the success transition, 0 otherwise; time limits bootstrap | 24 episodes / 15k env. steps |
| DreamZero-14B | The same four RoboLab tasks | Same | Same | Same | 16–20 episodes / 20–40k env. steps |
| DreamZero-5B | LIBERO-90 cream-cheese box ( LIVING_ROOM_SCENE1 , index 47) and moka pot ( KITCHEN_SCENE3 , index 19) | 600 | LIBERO initial states | Same | 25 episodes / 38,400 env. steps |
| LIBERO-90 cream cheese (57), moka pot (19), cream-cheese box (47), black bowl (9), alphabet soup (55), butter (56); ALOHA Transfer Cube | 400 | LIBERO: the benchmark’s 50 initial states; ALOHA: default environment resets | per decision, 0 on success; only success disables bootstrapping | 10 episodes / 10k gradient updates | |
| YAM Load dishwasher; Pack container | 300; 450 | Dishwasher: recorded scene; Pack: bottles –8 cm, box cm | Terminal: 0 on success, on failure or timeout | 16 episodes / 18k env. steps | |
| Diffusion Policy | Robomimic Lift, Can, Square | 300, 300, 400 | Standard resets | Sparse reward minus one, summed over the chunk; time limits terminal | 200 episodes / 48k env. steps (64k on Square) |
| Policy | Variant | Final success (%) | Reported |
| Release rule in the DSRL recipe: per-coordinate target 0.35, box 5 | 97.0 / 78.0 / 64.7 / 64.5 / 94.1 | yes | |
| Same, target 0.10 | 92.8 (cream cheese only) | ||
| Algorithm 1, on the broadcast row (one seed) | 62 / 92 / 58 / 0 / 85 | ||
| Algorithm 1 with the release’s per-coordinate target, 0.35 per coordinate or 11.2 on the 32-d row (one seed) | 100 / 88 / 54 / 1 / 81 | ||
| Algorithm 1, on the broadcast row | 39.8 / 18.9 | yes | |
| Algorithm 1 with the release’s per-coordinate target, 11.2 on the 32-d row | 13.4 / 3.6 |
| (a) Which noise to steer | |||||
| Method | Input | Mustard | Marker | Butter | Stack |
| Base policy | – | 25.8 | 3.3 | 52.1 | 62.9 |
| DSRL, tiled 8-d latent | Action | 26.5 2.2 | 3.6 0.6 | 55.0 0.8 | 68.6 1.0 |
| DSRL, all 256 entries | Action | 26.7 1.0 | 4.6 1.5 | 58.6 3.5 | 70.4 1.2 |
| LoRN, rank 32 | Action | 21.7 1.5 | 6.1 1.7 | 51.4 1.9 | 67.2 3.4 |
| LoRN, rank 32 | Video | 58.3 1.6 | 43.3 3.6 | 71.4 4.7 | 82.0 1.7 |
| Task | Base | 4 | 8 | 16 | 32 | 256 |
| Mustard | 25.8 | 41.0 5.5 | 47.6 4.9 | 56.7 1.3 | 58.3 1.6 | 57.1 1.3 |
| Marker | 3.3 | 9.3 4.7 | 15.7 5.1 | 32.6 6.5 | 43.3 3.6 | 38.5 1.8 |
| Butter | 52.1 | 63.3 2.1 | 68.7 3.2 | 68.1 2.9 | 71.4 4.7 | 71.5 1.3 |
| Stack | 62.9 | 72.6 2.4 | 75.6 1.8 | 75.1 1.5 | 82.0 1.7 | 81.1 0.5 |
| Method | Input | Mustard | Marker | Butter | Stack |
| Base policy | – | 65.8 | 25.0 | 41.7 | 49.2 |
| DSRL, tiled 32-d latent | Action | 69.4 0.3 | 31.4 1.1 | 46.1 2.0 | 54.7 5.4 |
| LoRN, rank 32 | Action | 66.4 2.0 | 32.5 1.0 | 56.7 3.8 | 50.6 1.5 |
| LoRN, rank 32 | Video | 84.2 1.9 | 70.0 6.6 | 83.1 4.3 | 71.9 3.1 |
| Method | Input | Cream-cheese box | Moka pot |
| Base policy | – | 48.0 | 56.0 |
| DSRL | Action | 69.8 4.9 | 68.9 3.6 |
| LoRN, tiled rank 16 | Action | 59.1 3.6 | 60.0 5.6 |
| LoRN, rank 32 | Video | 79.1 3.6 | 48.4 1.9 |
| Early-window mean (%) | Final success (%) | |||||||
| Policy | Task | Base | DSRL | LP-DS | LoRN | DSRL | LP-DS | LoRN |
| Cream cheese | 29.3 | 81.1 2.0 | 88.2 1.8 | 88.2 1.0 | 98.8 0.8 | 97.0 1.1 | 98.8 0.6 | |
| Moka pot | 80.7 | 63.6 2.7 | 66.7 3.1 | 70.1 2.4 | 91.1 3.1 | 78.0 5.1 | 89.7 2.4 | |
| Cream-cheese box | 60.7 | 55.3 5.1 | 51.6 4.6 | 66.1 1.6 | 84.3 2.0 | 64.7 5.4 | 87.2 1.9 | |
| Black bowl | 0.4 | 44.7 4.8 | 28.7 5.1 | 53.0 4.3 | 93.8 3.0 | 64.5 5.6 | 92.4 1.5 | |
| Alphabet soup | 51.1 | 89.2 0.9 | 86.6 1.8 | 79.9 2.9 | 98.4 0.5 | 94.2 1.1 | 96.4 0.6 | |
Robot Policy Learning
Robot policy learning develops controllers that map sensory observations, task goals, and interaction history to actions for embodied robots.
See all 405 papers