cs.ROSep 30, 2026

PrefPI: Preference-Guided Steering into Out-of-Distribution Behaviors

Authors: Seungeun Rho, Wontaek Kim, Danfei Xu, Sehoon Ha

Organizations: School of Interactive Computing Georgia Institute of Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We present PrefPI (Preference-Guided Policy Iteration), an iterative framework for steering pretrained generative robot policies using only relative preferences over self-generated trajectories. Unlike prior preference-learning methods that primarily sharpen modes already represented by the policy, we study steering beyond the initial effective support, where desired behaviors are rarely or never observed under the initial policy. Our key idea is to formulate preference learning as preference-conditioned generative modeling: preferred trajectories define a conditional distribution, whose density ratio with the broader behavior prior provides an implicit preference signal amplified by classifier-free guidance (CFG). Repeating this preference-conditioned modeling and guidance step yields a form of preference-guided policy iteration, turning incremental improvements toward previously inaccessible behaviors. Across diffusion policies and the PI0.5 flow- matching VLA in simulation and the real world, PrefPI produces substantial behavioral shifts with limited feedback. In particular, PrefPI increases object transport height from 10.7 cm to 19.8 cm on real hardware with only 150 preference-labeled trajectories.

Figures & tables

Figure 1: With only 150 preference-labeled trajectories (50 per loop over 3 loops), PrefPI increased the object’s maximum height from 10.7 cm to 19.8cm on real hardware, enabling obstacle clearance.
Figure 2: Overview of PrefPI. Each iteration trains two branches, one conditioned on the top- mm rollouts and the other on all successful rollouts, and CFG combines the two branches to produce the next policy.
Figure 3: PrefPI progressively steers the policy toward preferred behaviors. Results are averaged over three seeds. Top: preference metric; bottom: success rate.
Figure 4: Trajectories before and after training are highly disjoint, indicating that the blue trajectories lie beyond the initial policy’s effective support.
Figure 5: Real-world steering on three different preferences. PrefPI improves the preference metric by 1.83×1.83\times ( speed ), 1.9×1.9\times ( detour ), and 4.5×4.5\times ( throw ), while success rates remain above 90%90\% .
Figure 6: PrefPI completes the task nearly twice as fast as the initial policy using only 150 preference-labeled trajectories (30 per iteration over 5 iterations).
Figure 7: Comparison with baselines, averaged over five seeds. PrefPI achieves (a) the strongest OOD steering overall, (b) while matching the best baseline on in-distribution refinement.
Figure 8: PrefPI benefits from a high-diversity prior.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

 Dk={τi}i=1N,τi∼pk(τ)\mathcal{D}_{k}=\{\tau_{i}\}_{i=1}^{N},\qquad\tau_{i}\sim p_{k}(\tau) 
Appendix
Algorithm 1 PrefPI: Preference-Guided Policy Iteration
Figure 9: Example throw rollouts. The initial policy barely throws the object, while by iteration 14, PrefPI achieves a much longer throwing distance. A towel absorbs impact in the landing area.
HyperparameterValue
Base policyPI0.5-LIBERO ( pi05_libero )
VLM backboneSigLIP So400m/14 + Gemma 2B
Action expertGemma 300M
Action dim. / horizon32 (7 used) / 10
Max. prompt length200 tokens
Compute dtypebfloat16
Appendix
Table 1: Model and fine-tuning configuration for PI0.5.
HyperparameterValue
OptimizerAdamW
Adam β1,β2\beta_{1},\beta_{2}0.9,0.950.9,0.95
Peak learning rate5×10−55\times 10^{-5}
LR schedule20-step warmup, then constant
Weight decay10−1010^{-10}
Gradient clippingGlobal norm 1.0
Appendix
Table 2: Optimization and online adaptation settings.

Explore similar work

CardsList

  1. Steering Generative Robot Policies with Lexicographic Preferences

    Sep 14, 2026Yixuan Jia, Jonathan P. HowRobot PoliciesRobot Systems

    Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, Cambridge, MA 02319 USA

  2. PreferenceFlow: Test-Time Guidance of Flow-Matching Robot Policies from Human Interventions

    Sep 29, 2026Yiqi Tang, Diyuan Shi, Runze Li +1Flow PoliciesRobot Policies

    Westlake University

  3. Freeform Preference Learning for Robotic Manipulation

    Jun 30, 2026Marcel Torne, Anubha Mahajan, Abhijnya Bhat +1Robot PoliciesRobotic Manipulation

    Stanford University