Generative robot policies can represent diverse, multimodal behaviors, but adapting pretrained policies to deployment-time constraints such as collision avoidance and orientation maintenance remains challenging. Existing inference-time steering methods typically apply gradient guidance through iterative diffusion or flow processes, which can be computationally expensive for real-time control. We propose INSPO, which formulates inference-time steering of one-step generative policies as trajectory optimization in the policy's input noise space. By optimizing the input noise while evaluating constraints on the induced state trajectory, INSPO searches the policy-induced behavior space without directly modifying generated actions. The optimization includes a regularization term that encourages solutions to remain consistent with the policy's input distribution and is solved online using population-based particle optimization. We evaluate INSPO on state- and image-based task-specific policies and generalist vision-language-action policies across Push-T, Can pick-and-place, and LIBERO-Spatial. INSPO improves task success and constraint satisfaction over best-of-N sampling and action projection, while comparing favorably with gradient-guided generation at lower runtime.
Figures & tables
Figure 1: Inference-time safety steering of a one-step generative policy. The policy πθ maps noise samples in the noise space (left) to actions in the action space (right) with a single function evaluation. The red region C in the action space defines actions that violate the deployment-time constraint. Satisfying and violating actions are respectively shown in green and red, with their corresponding pre-images in the noise space colored accordingly. At inference time, we optimize the sampled noise to steer violating actions toward satisfying ones while regularizing the noise to remain close to its original distribution. Arrows show four such steered samples: each links a violating sample to its optimized counterpart, drawn in both spaces.
Figure 2: Comparison of trajectories generated by different methods on the Push-T task. Each row shows the progression of one method over time. Red borders indicate collisions, while green borders indicate successful task completion without collision. Corresponding videos are provided in the supplementary materials.
Figure 3: Comparison of trajectories generated by different methods on the Can pick-and-place task. Each row shows the progression of one method over time. Red borders indicate constraint violations, while green borders indicate successful task completion without constraint violations.
Figure 4: Comparison of trajectories generated by different methods on the LIBERO-Spatial task. Each row shows the progression of one method over time. Red borders indicate collisions, while green borders indicate successful task completion without collision.
Method
# Success ↑
# Collision ↓
Time [s] ↓
Drifting Policy
2
8
0.01 ± 0.01
Best-of- N
4
6
0.02 ± 0.01
INSPO (ours)
8
2
0.06 ± 0.10
Table III : Obstacle Avoidance results in real-world lifting experiments.
Figure 5: Validation on a real-world lifting task with an obstacle (dark red sphere) introduced only at deployment. Each row from left to right is one method over time. Top: without steering, the robot collides with the obstacle (red borders). Bottom: our method steers the robot around the obstacle and completes the lift (green border). The obstacle is re-colored from white for clarity of presentation. Corresponding videos are provided in the supplementary materials.
Real-time robot control demands fast action generation. Diffusion and flow matching policies for robot control require multi-step sampling, limiting their deployment in real-time scenarios. Natively reducing the sampling steps to one sacrifices representation quality and task performance, creating a trilemma among speed, fidelity, and performance. We present One-Step Generative Policy Optimization (OGPO), a systematic framework to resolve this trilemma. OGPO first pairs a lightweight architecture with the interval velocity principle for distillation-free one-step inference, while representation spreading prevents representation quality degradation. It then performs on-policy reinforcement learning (RL) fine-tuning on this fast, stable policy to break the imitation learning ceiling. Experiments on RoboMimic and OpenAI Gym benchmarks show that OGPO matches or exceeds multi-step baselines while achieving a 5-20 times inference speedup and over 120Hz control frequency. Physical deployment on a Franka-Emika-Panda robot validates real-world applicability. Project page: https://ogpo-project.github.io/
Guowei Zou, Haitao Wang, Hejun Wu +3
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, China
Diffusion and flow-based generative policies provide a powerful policy class for reinforcement learning by inducing rich stochastic exploration through iterative action generation. However, the stochasticity of diffusion policies is not suitable for stable and precise control in high-dimensional robotic systems, where small action variations can accumulate into inconsistent motion and reduced robustness. To address this issue, we propose SteerGenPO, a latent-space reinforcement learning framework that steers a trained generative policy into a robust deterministic robotic controller. The key idea is to replace stochastic latent sampling of the trained generative policy with a learned latent actor that predicts a state-dependent latent input for the generative policies. This separates exploration and control: stochastic generative sampling provides diverse action proposals during policy learning, while deterministic latent steering provides stable and adaptive control at deployment. We evaluate SteerGenPO on six Isaac Lab benchmarks and a Unitree G1 locomotion task. The results show SteerGenPO improves over both classical RL and generative RL baselines, while its deterministic latent steering produces more stable inference-time behaviors and more reliable command responses.
Inference-time guidance is essential for steering generative robot policies toward dynamic objectives without retraining, yet existing methods are largely confined to chunk-based architectures that exhibit high latency and lack the reactivity needed for test-time preference alignment or obstacle avoidance. In this work, we formally derive the optimal guidance term for Stochastic Interpolants (SI) by analyzing the value function's time evolution via the Backward Kolmogorov Equation, establishing a modified drift that theoretically guarantees sampling from a target distribution. We apply this framework to real-time control through the Streaming Stochastic Interpolant Policy (SSIP), which generalizes the deterministic Streaming Flow Policy (SFP). Unifying this guidance law with the streaming architecture enables fast and reactive control. To support diverse deployment needs, we propose two complementary mechanisms: training-free Stochastic Trajectory Ensemble Guidance (STEG) that computes gradients on-the-fly for zero-shot adaptation, and training-based Conditional Critic Guidance (CCG) for amortized inference. Empirical evaluations demonstrate that our guided streaming approach significantly outperforms conventional chunk-based policies in reactivity and provides superior, physically valid guidance for dynamic, unstructured environments.
Puming Jiang, Meiyi Wang, Kelvin Lin +2
School of Computing, National University of Singapore