Progress-Guided Preference Generation for Aligning Generative Control Policies

Tung M. Luu; Tri Ton; Duc M. Nguyen; Xuan Thanh Nguyen; Nguyen Gia Binh; Quang Tan Nguyen; Vu Duong; Daniel Sonntag

cs.LGOct 19, 2026

Progress-Guided Preference Generation for Aligning Generative Control Policies

Authors: Tung M. Luu, Tri Ton, Duc M. Nguyen, Xuan Thanh Nguyen, Nguyen Gia Binh, Quang Tan Nguyen, Vu Duong, Daniel Sonntag,

Organizations: KAIST · Center for AI Research & CECS, VinUniversity · VinRobotics · DFKI ·

Abstract

Generative policies have achieved strong performance in imitation learning tasks, but their success is often constrained by the limited availability of expert demonstrations. Although preference-based fine-tuning can leverage suboptimal rollouts to improve performance, existing approaches heavily rely on expensive human annotations. In this work, we introduce Progress-Guided Preference Generation (PGPG), an unsupervised preference alignment framework that eliminates the need for human-in-the-loop supervision. Our method leverages ordered coverage alignment to measure sequential progress relative to a small set of expert demonstrations, enabling the automatic extraction of pseudo-preference labels from autonomously collected rollouts. Furthermore, we introduce key enhancements to improve the reliability of these progress-guided preferences, enabling effective fine-tuning of pretrained generative policies without requiring manually designed reward functions. Extensive experiments across a suite of challenging long-horizon manipulation tasks demonstrate that PGPG outperforms standard supervised fine-tuning baselines. Moreover, our conceptually simple framework is broadly applicable across multiple classes of generative policies, demonstrating strong versatility when integrated with both diffusion and flow-matching policies.

Figures & tables

Figure 1: Overview of PGPG . Autonomously collected rollouts ( $\mathcal{D}_{\text{rollout}}$ ) are scored against expert references ( $\mathcal{D}_{\text{demo}}$ ) using an alignment pipeline consisting of Ordered Coverage Alignment, Marginal Progress Scoring, and Multi-Demonstration Aggregation. The aggregated segment scores are then used to derive the preference labels. The resulting dataset $\mathcal{D}_{\text{pref}}$ provides structured, dense pseudo-preferences to stably fine-tune generative policies $\pi_{\theta}(a|s)$ without manual reward engineering.

Panel of Figure 2 — Figure 2: Success rates of methods on five Robomimic manipulation tasks. Error bars denote standard deviation across seeds. DPPO is applied only to the diffusion policy.

Method	Lift	Can	Square	Transp.	ToolHang
Temporal OT	20.7 $\pm$ 1.2	58.7 $\pm$ 5.8	62.7 $\pm$ 3.1	36.7 $\pm$ 6.1	32.0 $\pm$ 5.3
Temporal OT + MPS	82.0 $\pm$ 2.0	72.0 $\pm$ 5.3	51.3 $\pm$ 10.1	43.3 $\pm$ 5.0	41.3 $\pm$ 8.3
DTW	90.7 $\pm$ 3.1	70.0 $\pm$ 5.3	72.7 $\pm$ 1.2	45.3 $\pm$ 11.0	30.0 $\pm$ 8.7
DTW + MPS	80.0 $\pm$ 5.3	78.0 $\pm$ 7.2	62.7 $\pm$ 7.0	38.7 $\pm$ 2.3	31.3 $\pm$ 7.0
ORCA	32.0 $\pm$ 2.0	70.7 $\pm$ 5.0	42.0 $\pm$ 3.5	47.3 $\pm$ 4.6	41.3 $\pm$ 3.1
ORCA + MPS (Ours)	96.7 $\pm$ 1.2	89.3 $\pm$ 4.6	72.0 $\pm$ 7.2	58.7 $\pm$ 6.1	56.0 $\pm$ 8.7

Table 1: Comparison of fine-tuning performance using preference labels generated by different sequence-matching metrics within our framework. Best results and those within one standard deviation are bolded.

Panel of Figure 3 — Figure 3: Ablation study across different design choices.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Number of rollouts	Lift	Square
500	96.7 $\pm$ 1.2	72.0 $\pm$ 7.2
375	94.7 $\pm$ 3.2	71.7 $\pm$ 2.5
250	96.0 $\pm$ 3.0	69.7 $\pm$ 6.1
125	92.3 $\pm$ 4.5	65.7 $\pm$ 3.1

Appendix

Table 2: PGPG’s performance using different numbers of rollouts ( i.e. ,

|\mathcal{D}_{\text{rollout}}|

Number of demos	Lift	Can	Square
Base policy	44.0 $\pm$ 7.2	42.7 $\pm$ 1.2	39.3 $\pm$ 3.1
0 (pure rollouts)	32.7 $\pm$ 4.6	64.7 $\pm$ 6.4	51.3 $\pm$ 5.0
40	95.3 $\pm$ 1.2	88.0 $\pm$ 4.0	56.7 $\pm$ 6.1
120	96.7 $\pm$ 1.2	89.3 $\pm$ 4.6	72.0 $\pm$ 7.2

Appendix

Table 3: PGPG’s performance using different numbers of demonstrations ( i.e. ,

|\mathcal{D}_{\text{demo}}|

Encoder	Lift	Can	Square
Base policy	44.0 $\pm$ 7.2	42.7 $\pm$ 1.2	39.3 $\pm$ 3.1
PGPG w/ VC-1	96.0 $\pm$ 4.0	83.3 $\pm$ 4.6	52.7 $\pm$ 3.1
PGPG w/ SigLIP2	68.7 $\pm$ 7.0	58.0 $\pm$ 8.7	42.7 $\pm$ 7.0
PGPG w/ R3M (Ours)	96.7 $\pm$ 1.2	89.3 $\pm$ 4.6	72.0 $\pm$ 7.2

Appendix

Table 4: PGPG performance with different visual encoders.

$L$	100	150	200
Success Rate	54.3 $\pm$ 7.6	71.3 $\pm$ 7.1	72.0 $\pm$ 7.2

Appendix

Table 5: Performance of PGPG with different segment lengths

L

on Square task.

Aggregation	Lift	Square
Mean	90.7 $\pm$ 2.3	52.7 $\pm$ 2.1
Max	96.7 $\pm$ 1.2	72.0 $\pm$ 7.2

Appendix

Table 6: Performance of PGPG with different aggregation operators across demonstrations.

Method	Lift	Can	Square	Transp.	ToolHang
CLIP	36.0 $\pm$ 7.2	62.0 $\pm$ 6.0	58.0 $\pm$ 8.7	21.3 $\pm$ 4.2	30.7 $\pm$ 9.0
S3D	53.3 $\pm$ 3.1	60.0 $\pm$ 3.5	62.8 $\pm$ 4.6	26.7 $\pm$ 4.2	54.0 $\pm$ 8.7
Robometer	38.7 $\pm$ 8.1	85.3 $\pm$ 3.1	40.0 $\pm$ 3.5	28.0 $\pm$ 6.9	31.3 $\pm$ 6.1
PGPG (Ours)	96.7 $\pm$ 1.2	89.3 $\pm$ 4.6	72.0 $\pm$ 7.2	58.7 $\pm$ 6.1	56.0 $\pm$ 8.7

Appendix

Table 7: Comparison with Vision-Language and Progress Reward Baselines. Best results and those within one standard deviation are bolded.

Figure 4: Average wall-clock time required to compute progress rewards per trajectory, evaluated over 50 rollouts. By leveraging the ORCA progress metric, PGPG is nearly $31\times$ faster than the Robometer [ 29 ] baseline, successfully eliminating the high inference latency typically associated with progress reward models.

Figure 5: LIBERO-10 tasks: Each row of frames shows one of the five available expert demonstrations for a task, progressing from the initial state (left) to task completion (right).

Task	Instruction	Pretrained	Filtered BC	PGPG
T1	put both the alphabet soup and the tomato sauce in the basket	18.0 $\pm$ 5.3	37.3 $\pm$ 3.1	64.7 $\pm$ 7.6
T2	put both the cream cheese box and the butter in the basket	58.0 $\pm$ 2.0	76.7 $\pm$ 1.2	94.7 $\pm$ 3.1
T3	turn on the stove and put the moka pot on it	80.0 $\pm$ 3.5	76.7 $\pm$ 1.2	87.3 $\pm$ 3.1
T4	put the black bowl in the bottom drawer of the cabinet and close it	86.7 $\pm$ 4.2	96.0 $\pm$ 2.0	100.0 $\pm$ 0.0
T5	put the white mug on the left plate and put the yellow and white mug on the right plate	64.0 $\pm$ 5.3	81.3 $\pm$ 1.2	78.7 $\pm$ 1.2
T6	pick up the book and place it in the back compartment of the caddy	53.3 $\pm$ 8.1	72.7 $\pm$ 4.2	58.7 $\pm$ 1.2

Appendix

Table 8: Success rate (%) of methods on LIBERO-10. Mean

\pm

standard deviation over three seeds with 50 episodes per task each. The best result for each task is shown in bold.

Hyperparameter	Value
Preference Generation (PGPG)
Reference Demonstrations ( $M$ )	10-40
Discrimination Threshold ( $\epsilon_{\sigma}$ )	$\{0.3,0.4,0.6\}$
Visual Feature Encoder	R3M [ 44 ]
Distance Metric	Cosine Distance
Segment Length ( $L$ )	100 (Lift, Can), 200 (Square), 300 (others)

Appendix

Table 9: Comprehensive list of hyperparameters for PGPG.

Preference-Based RL

Preference-based reinforcement learning learns policies from comparisons or rankings of behaviors instead of requiring hand-designed numerical rewards.

See all 118 papers

PrefPI: Preference-Guided Steering into Out-of-Distribution Behaviors
Sep 30, 2026Seungeun Rho, Wontaek Kim, Danfei Xu +1Robot Policy Learning Robot Policy Adaptation
School of Interactive Computing Georgia Institute of Technology
Pave-GRPO: Beyond Instantaneous Guidance through Principled Average Velocity Decomposition
Jun 1, 2026Pengyang Ling, Jiazi Bu, Yujie Zhou +6Diffusion Model Alignment Group Relative Policy Optimization
University of Science and Technology of China, China · Shanghai Jiao Tong University, China · Fudan University, China +3
OGPO: Sample Efficient Full-Finetuning of Generative Control Policies
May 4, 2026Sarvesh Patil, Mitsuhiko Nakamoto, Manan Agarwal +14RL for Robotics RL for Generative Models
Carnegie Mellon University · University of California, Berkeley · University of Washington +1

Sep 30, 2026J/K move · Enter open · S save

PrefPI: Preference-Guided Steering into Out-of-Distribution Behaviors

Seungeun Rho, Wontaek Kim, Danfei Xu, Sehoon Ha

School of Interactive Computing Georgia Institute of Technology

Robot Policy Learning Robot Policy Adaptation Preference-Based RL

Open paper AskPDF arXiv

We present PrefPI (Preference-Guided Policy Iteration), an iterative framework for steering pretrained generative robot policies using only relative preferences over self-generated trajectories. Unlike prior preference-learning methods that primarily sharpen modes already represented by the policy, we study steering beyond the initial effective support, where desired behaviors are rarely or never observed under the initial policy. Our key idea is to formulate preference learning as preference-conditioned generative modeling: preferred trajectories define a conditional distribution, whose density ratio with the broader behavior prior provides an implicit preference signal amplified by classifier-free guidance (CFG). Repeating this preference-conditioned modeling and guidance step yields a form of preference-guided policy iteration, turning incremental improvements toward previously inaccessible behaviors. Across diffusion policies and the PI0.5 flow- matching VLA in simulation and the real world, PrefPI produces substantial behavioral shifts with limited feedback. In particular, PrefPI increases object transport height from 10.7 cm to 19.8 cm on real hardware with only 150 preference-labeled trajectories.

Abstract

Preference-Based RL

Explore similar work

PrefPI: Preference-Guided Steering into Out-of-Distribution Behaviors

Pave-GRPO: Beyond Instantaneous Guidance through Principled Average Velocity Decomposition

OGPO: Sample Efficient Full-Finetuning of Generative Control Policies