You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vector

Omkar Patil; Ondrej Biza; Thomas Weng; Karl Schmeckpeper; Wil Thomason; Xiaohan Zhang; Kausik Sivakumar; Robin Walters

cs.ROMar 16, 2026

You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vector

Authors: Omkar Patil, Ondrej Biza, Thomas Weng, Karl Schmeckpeper, Wil Thomason, Xiaohan Zhang, Kausik Sivakumar, Robin Walters,

Organizations: Robotics and AI Institute (RAI) · Arizona State University (ASU) · Northeastern University

Abstract

Generative robot policies trained on demonstrations using behavior cloning often learn actions that are sub-optimal or misaligned with respect to the downstream task. Policy improvement approaches aim to bridge this gap and improve the cumulative reward with minimal interventions on the pre-trained policy. However, an impediment to their practical deployment is the requirement of cumbersome hyperparameter tuning specific to model or task families. In this work, we present an episodic, derivative-free latent policy improvement approach that works with little to no changes across model scales from MLPs to VLAs. We demonstrate that the performance of a pretrained, frozen diffusion or flow matching policy can be improved with respect to a downstream reward by swapping the sampling of initial noise from the prior distribution (typically isotropic Gaussian) with a well-chosen, constant initial noise input---a golden ticket. We show the prevalence of golden tickets by improving the policy performance of $46$ out of $51$ tasks across manipulation benchmarks, with absolute improvements in success rate by up to $79\%$ for simulated tasks, and $28\%$ within $60$ search episodes for real-world tasks. Further, we find that the versatility of our approach opens up new avenues such as simultaneous policy improvement for multiple downstream rewards. Project webpage: https://lottery-tickets.rai-inst.com/.

Figures & tables

Fig. 1 : Model design challenges for derivative-based methods: Two plots showing only DSRL curves with different hyperparameters. Actor-critic MLP used for Can task $128\!\times\!3$ is not optimal on LIBERO#57 . (a) $\pi_{0}$ ’s larger action space needs $10$ critics $n_{Q}$ and noise space tiling. (b) Actor-critic MLP $64\!\times\!2$ beats the reported result (black) but $512\!\times\!2$ fails. Can policy improvement be made simpler and robust?

Fig. 2 : Overview of standard diffusion policy inference (left) versus our proposed inference approach of using golden tickets (right). Given a frozen, pretrained diffusion or flow matching policy $\pi_{\text{pre}}$ , rather than sampling from a Gaussian every time an action needs to be computed, we use a constant, well-chosen initial noise vector $w$ , called a golden ticket. We find golden tickets improve policy performance across a range of observation inputs, model architectures, and embodiments.

Fig. 3 : D3IL task Avoiding-M2 : (a) Demonstrations the policy was trained on covering blue and red modes; (b) Learned policies often generate diverse behaviors (gray and green); (c) Initial noises (300 here) provide a broad mode coverage for us to choose a noise sample that optimizes the downstream reward; (d) DSRL, optimized for the blue mode. See Section IV-A .

Panel of Figure 4 — Fig. 4 : Rollouts from diffusion policies sampling with Gaussian noise that we use in our hardware experiments. (top) An example successful rollout; (bottom) A failed rollout.

Fig. 5 : Existence and performance of golden tickets ( gold , right) relative to the base policy ( blue , left) on simulated benchmarks. Simply optimizing the initial noise vector can lead to strong performance gains over the base policy across benchmarks and policies. We report mean and standard deviation of success rates (details in Section V-B ). Our open-source repository will contain code and golden tickets for the first three benchmarks: franka_sim, robomimic, and LIBERO.

Fig. 6 : Sample efficiency comparison on Can with an MLP base policy (a) and three LIBERO-90 tasks with

\pi_{0}

(b–d). Simply searching for constant noise vectors (GT) becomes comparable in sample efficiency to actor-critic methods at the high-dimensional noise spaces of VLA models, where DSRL on the full

1600

D noise (dashed) fails to improve on the base policy.

Fig. 7 : Various tickets (pink) for the franka_sim pick policy, evaluated according to success rate and speed (mean length of successful episodes). Right is higher success rate, down is faster time to success. Because lottery tickets exhibit extreme differences in policy performance, a Pareto frontier is defined by tickets that are further right/down than others (represented with golden ticket icons).

Fig. 8 : (a) Effect of hyperparameter changes on GT search for cream cheese in tray . (b) GT ablations: searching a tiled space leads to worse performance due to a constrained search space. Search in latent policy parameter space (akin to DSRL actor) also leads to poor performance.

Method	Lift ( $1$ M)	Can ( $2$ M)
Random search	$15$ ( $23\times$ )	$32$ ( $26\times$ )
Plain CEM	$23$ ( $15\times$ )	$42$ ( $20\times$ )
CEM+racing	$67$ ( $5\times$ )	$149$ ( $6\times$ )
DSRL	$345$	$827$

TABLE I : Wall-clock minutes to run

1

M env steps on Lift and

2

M on Can (speed-up over DSRL in parentheses)

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Policy / checkpoint	Obs / Action	Tasks
franka_sim [ 28 ]	Flow-matching MLP (open-sourced)	state / EE-pos + gripper	pick_cube
robomimic [ 31 ]	DPPO diffusion github.com/irom-princeton/dppo	state / EE-pose	Lift , Can , Square , Transport
LIBERO [ 29 ]	SmolVLA [ 30 ] huggingface.co/HuggingFaceVLA/smolvla_libero	$2\times$ RGB + state + lang / $32\!\times\!50$ chunk	$30$ tasks across Object, Goal, Spatial ( $10$ each)
SimplerEnv (WidowX) [ 27 ]	GR00T N1.5 [ 26 ] github.com/NVIDIA/Isaac-GR00T/tree/n1.5-release	RGB + lang / EE	close_drawer , open_drawer , put_eggplant_in_basket , put_eggplant_in_sink , stack_cube , carrot_on_plate , spoon_on_towel
DexMimicGen [ 32 ]	U-Net diffusion (ours)	$3\times$ RGB + proprio / bimanual EE	Drawer Cleanup , Tray Lift , Threading , Box Cleanup , Piece Assembly
Franka HW (RGB)	U-Net diffusion (ours)	$3\times$ RGB + proprio / EE	cube_pick , piston_assembly

Appendix

TABLE II : Summary of experimental benchmarks. “Ours” indicates a policy trained internally. EE = end-effector, proprio = proprioception, PCD = point cloud.

Fig. 9 : Images from some of our simulated benchmarks: (1-3) LIBERO Object, Spatial, and Goal task suites, (4-6) DexMimicGen Tray Lift , Threading , and Piece Assembly tasks, and (7-9) robomimic Lift , Can and Square .

Task	Base	Avg. Gold	Gold	$N$
close_drawer	$65$	$99.2$	$99.0/98.7/100.0$	$11/16/6$
put_eggplant_in_basket	$63$	$91.7$	$96.0/93.3/85.7$	$16/32/14$
put_eggplant_in_sink	$21$	$69.3$	$67.7/64.0/76.3$	$270/27/100$
stack_cube	$54$	$71.3$	$71.7/73.7/68.7$	$230/69/27$
carrot_on_plate	$72$	$74.3$	$78.7/51.3/93.0$	$150/106/81$
spoon_on_towel	$82$	$79.9$	$77.3/72.3/90.0$	$230/53/66$

Appendix

TABLE III : SimplerEnv (WidowX) results over three search seeds. Gold : best held-out success rate (%,

300

episodes) in each seed’s CEM run;

N

: CEM candidates evaluated in each run.

Benchmark	Task	Base	Golden	Search
franka_sim	Avg. over $4$ ckpts	$38.5\pm 27.9$	$96.0\pm 2.4$	${\approx}250\times 25$ (RS)
robomimic	Lift	$78.4\pm 3.9$	$96.2\pm 1.4$	$5000\times 100$ (RS/CEM)
	Can	$42.8\pm 4.4$	$95.0\pm 2.1$
	Square	$52.2\pm 3.6$	$42.6\pm 5.7$
	Transport	$10.7\pm 2.2$	$18.8\pm 3.6$
	Avg.	$46.0$	$63.1$

Appendix

TABLE IV : Per-task summary of the golden-ticket vs. Gaussian comparison across all

51

task–policy pairs. Base and Golden are held-out success rates (%); Search combines the per-task budget (tickets

\times

search-envs per ticket) with the search method (RS = random search, CEM = cross-entropy method). Shaded Avg. rows give the per-benchmark mean across the listed tasks.

Panel of Figure 14 — Fig. 10 : DSRL ( blue ) vs. Golden Ticket via Random Search ( yellow ) on DexMimicGen across five tasks (columns), two DDIM step settings (top row of each panel: DDIM- $2$ ; bottom row: DDIM- $8$ ), and two search-episode budgets ( $5$ k in (a), $10$ k in (b)). The dashed horizontal line in each panel marks the base-policy success rate at that DDIM setting.

LIBERO-Goal
#Ticket	T0	T1	T2	T3	T4	T5	T6	T7	T8	T9	Avg
Base	72	94	86	52	92	78	80	100	94	68	81.6
#03c2 $\star$	84	100	92	40	100	64	94	100	98	18	79.0
#2cee	78	84	100	2	98	76	22	100	86	0	64.6
#1ff8	98	64	86	14	100	0	38	96	100	2	59.8
#4d97	2	96	16	68	98	92	30	54	100	4	56.0

Appendix

TABLE VI : Success rates (%) for base policy and lottery tickets on LIBERO-Goal, LIBERO-Object, and LIBERO-Spatial with SmolVLA. T0 - T9 represent different tasks. Top-1 ticket per task shown (unique set) plus best average ticket (

\star

). Underline:

\geq

base policy performance.

Task pair	Ticket	Search idx	Task $1$	Task $2$	Avg.
Drawer OD , CD	Base policy	—	$84$	$65$	$74.5$
	ticket# $1$	$201$	$90.0$	$\mathbf{98.3}$	$94.2$
	ticket# $2$	$225$	$\mathbf{97.3}$	$\mathbf{98.3}$	$\mathbf{97.8}$
Eggplant PES , PEB	Base policy	—	$21$	$63$	$42.0$
	ticket# $1$	$382$	$60.7$	$\mathbf{88.3}$	$\mathbf{74.5}$
	ticket# $2$	$410$	$\mathbf{64.0}$	$82.3$	$73.2$

Appendix

TABLE VII : Multi-task golden ticket search on two SimplerEnv (WidowX) task pairs. Cells are held-out success rates (%,

300

episodes per task); Avg. is the pair average.

Robot Policy Adaptation

Robot policy adaptation changes or steers learned robot behaviors to handle new tasks, environments, hardware, and user preferences.

See all 173 papers

Jun 16, 2026

Where Should Action Generation Begin? A Learnable Source Prior for Generative Robot Policies

Generative robot policies typically begin action generation from an observation-independent standard Gaussian distribution, leaving the choice of source distribution underexplored. This work asks a simple question: where should action generation begin? We propose LeaP, a Learnable source Prior that replaces the standard Gaussian with a proprioception-conditioned diagonal Gaussian over action chunks. Parameterized by a lightweight MLP, LeaP jointly predicts the mean and state-adaptive variance of the source distribution, while keeping the downstream generator architecture and inference solver unchanged. This design provides an observation-informed yet stochastic initialization, allowing the generator to focus on precise action refinement rather than transporting samples from an uninformed noise source. On 15 RoboTwin manipulation tasks, LeaP achieves an average success rate of 81.6%, outperforming four representative baselines -- including deterministic-source methods, a no-prior counterpart, and a diffusion-bridge policy -- by 6.5 to 25.5 percentage points. The same prior consistently improves both flow-matching and diffusion-bridge generators, while using fewer parameters and converging faster. The advantage carries over to real-world deployment, where LeaP attains the best performance. These results suggest that the source distribution is an independent and reusable design axis for generative robot policies, complementary to the choice of generative dynamics.

Meipo Dai, Qiyuan Zhuang, He-Yang Xu +4

Southeast University · The Chinese University of Hong Kong

Robot Policy Learning Robotic Manipulation Policy Learning Conditional Generative Modeling

Ask PDF

Sep 26, 2026

RoboFFT: Finetuning generative robot policy via online reinforcement learning with forward process

Generative models, such as diffusion and flow-based models, have shown strong promise for robot policy learning by capturing complex and multimodal action distributions from demonstrations. However, policies trained solely with imitation learning often suffer from imperfect demonstrations and distributional shifts, while further improvement typically requires additional expert data. Reinforcement learning offers a natural solution through environment interaction, but effectively finetuning generative robot policies remains challenging due to the intractability of likelihood estimation. In this work, we propose RoboFFT, a forward-process reinforcement learning framework for finetuning generative robot policies, which applies forward noising to sampled actions and uses the weighted score / flow matching loss to construct a surrogate policy ratio for PPO-style updates. We evaluate RoboFFT with popular generative robot policies on representative simulation benchmarks, including long-horizon planning and sparse reward settings. Extensive experiments and analysis demonstrate that RoboFFT consistently improves performance while achieving better stability and training efficiency. We further integrate RoboFFT into a real world RL framework and demonstrate its effectiveness in real world tasks. Project website: https://student-of-holmes.github.io/RoboFFT/.

Yu Li, Shenghe Hu, Yuhan Wang +4

Institute for Artificial Intelligence, Peking University · Zhongguancun Academy · PKU-PsiBot Joint Lab +1

Robot Policy Learning RL for Robotics Robotic RL RL Fine-Tuning RL for Generative Models

Ask PDF

Jul 9, 2026

FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space

Pretrained generative robot policies based on flow matching and diffusion have achieved impressive results across a wide range of manipulation tasks. Yet real-world deployments routinely expose failure modes outside the pretraining distribution. Closing these gaps typically requires large-scale data collection or online reinforcement learning on physical hardware, which is impractical for rapid and safe adaptation. We present FlowDAgger, a sample- and compute-efficient method for adapting frozen generative robot policies from human interventions in latent space. Our key idea is action inversion: each human expert action is mapped to the noise that would have produced it under the frozen base policy, using reverse-time integration followed by local refinement. The resulting inverted noise provides supervision for a lightweight latent policy that steers the base model at deployment time, enabling rapid skill acquisition while preserving its behavioral priors. We evaluate FlowDAgger in simulation and on real-world bimanual and single-arm manipulation, adapting both action-head VLAs and world-action models from a handful of interventions. FlowDAgger outperforms supervised fine-tuning and latent-space RL baselines and preserves pretrained skills on held-out tasks, offering a practical path for adapting robot foundation models in the real world. Website: https://microsoft.github.io/FlowDAgger

Michael Murray, Daphne Chen, Simran Bagaria +7

1Microsoft Research · University of Washington · 2Microsoft +1

Latent Action Learning Robot Policy Learning Robot Policy Adaptation Robot Skill Learning Robot Foundation Models Data-Efficient Robot Learning+2

Ask PDF

Abstract

Robot Policy Adaptation

Explore similar work

Where Should Action Generation Begin? A Learnable Source Prior for Generative Robot Policies

RoboFFT: Finetuning generative robot policy via online reinforcement learning with forward process

FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space