Organizations: University of Southern California, Los Angeles, CA, USA. · Beijing University of Chemical Technology, Beijing, China. · George Mason University, Fairfax, VA, USA. · Indiana University Bloomington, Bloomington, IN, USA.
Preference-based reinforcement learning (PbRL) has emerged as a promising paradigm for learning robot behaviors aligned with human intent. However, its scalability is fundamentally limited by the need to collect expensive human preference labels for every new task. In this work, we present PrefO2O, an offline-to-online latent preference transfer framework that enables zero-shot PbRL on unseen target tasks without requiring any human feedback. The core idea is to learn a task-invariant latent preference space from a small amount of offline preference data on source tasks, where transferable preference structures are disentangled from task-specific variations. This learned manifold allows PrefO2O to transfer preference structures and reward priors to new tasks under distribution shift, while confidence-weighted optimal transport alignment generates reliable pseudo-preference supervision for online reward adaptation. Extensive experiments on robotic manipulation and locomotion benchmarks demonstrate that PrefO2O substantially outperforms existing preference-transfer baselines and matches or exceeds PbRL methods trained with real human feedback, while generalizing across multi-to-one and one-to-one transfer settings as well as across different robot embodiments.
Figures & tables
Fig. 1: Conceptual illustration of PrefO2O. Offline preference data from multiple source tasks are used to learn a task-invariant latent preference space that captures shared preference structures while absorbing task-specific variations. When deployed on an unseen target task, this learned space enables both transferable reward priors and OOD-aware pseudo-preference supervision for zero-shot online PbRL without target-task human feedback.
Fig. 2: Overview of PrefO2O. The offline phase jointly learns a trajectory encoder, a preference VAE, and a meta reward model from source-task preference data. In the online phase, the pretrained encoder and reward model warm-start target-task PbRL, and the frozen preference VAE serves as a stable reference for OT-based alignment. Source labels are propagated to target trajectory pairs through the transport matrix, producing pseudo-preference labels with OOD-aware confidence weights for online reward learning and policy optimization.
Fig. 3: Learning curves of all methods on six tasks under two transfer protocols. The left panel shows intra-embodiment transfer, while the right panel shows cross-embodiment transfer. Learning curves for Sweep Into and Walker Walk are provided in Appendix B .
Fig. 4: Pseudo-label accuracy (%) across three benchmarks. Bars show the mean over five random seeds; error bars indicate one standard deviation.
Fig. 5: Results of one-to-one transfer with intra- and cross-embodiment settings. Different colors indicate different single source tasks used for transfer. The dashed line reports the default multi-to-one performance as a reference.
Fig. 6: Results of ablation study on four target tasks under default intra-embodiment, multi-to-one transfer settings.
Fig. 7: t-SNE visualization of preference representations with Window Close as the target task, comparing the raw space against the learned latent preference space under intra-embodiment (left) and cross-embodiment (right) transfer. Each point denotes either a raw trajectory-pair difference or its corresponding latent preference vector epref . The upper row colors points by task domain and the lower row by preference label Υ . The learned latent space exhibits strong cross-task mixing and clear separation across preference labels, indicating a task-invariant yet preference-discriminative structure that the raw space lacks.
Fig. 8: Attribute alignment along the learned preference axis with Window Close as the target task, evaluated under intra-embodiment (top) and cross-embodiment (bottom) transfer. Each point denotes a latent preference vector epref , colored by preference label Υ . The horizontal axis is the preference-axis projection score, obtained by projecting epref onto the direction between the centroids of samples with Υ=0 and Υ=1 . The vertical axis reports the attribute value difference for three task-relevant behavioral attributes: approach progress, path directness, and motion smoothness. The dark curve shows the attribute trend along the preference axis, computed as the binned mean attribute value. The Spearman correlation coefficient ρ in each panel quantifies the monotonic association between the preference-axis projection score and the corresponding attribute value difference. The attributes increase consistently along the learned preference axis, indicating that the latent preference space captures a structured behavioral manifold aligned with preference-relevant properties.
Fig. 9: Case study of OT-based preference alignment in the learned latent space, with Sweep Into as the target task. Left: a chord diagram where outer bars denote individual preference pairs grouped by source task, colored by preference label Υ ; each chord connects a target pair to a source pair, with width proportional to the transport mass Tij∗ . Every target pair draws correspondences from all source tasks, but with markedly different mass, showing that alignment is driven by preference structure rather than task identity. Right: for two representative target pairs (Sweep Into #4 and #53), the source pair receiving the highest transport mass, visualized as trajectory rollouts. Despite belonging to entirely different tasks with distinct object interactions, the matched source and target pairs share the same underlying behavioral quality contrast, confirming that the learned latent space aligns preferences along transferable behavioral features rather than task-specific motions.
Fig. 10: PrefO2O successfully completes real-world Cube Stacking and Pick-and-Place tasks on a Kinova Jaco robot.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Fig. 11: Learning curves on Sweep Into from MetaWorld and Walker Walk from DeepMind Control Suite under the same-embodiment transfer protocol.
Preference-based reinforcement learning learns policies from comparisons or rankings of behaviors instead of requiring hand-designed numerical rewards.
Preference-based reinforcement learning (PbRL) avoids explicit reward engineering by learning from pairwise human preference feedback. Existing offline PbRL methods typically follow a two-stage pipeline, first learning a reward or preference model from labeled preferences and then performing offline RL on unlabeled data. We revisit offline PbRL through the lens of reward-free representation learning (RFRL) from the zero-shot RL literature, and propose a new training framework that first learns latent successor-measure representations from reward-free offline data, followed by contrastive search and fine-tuning using preference data. Through extensive experiments and ablations, we show that our method achieves superior preference efficiency over offline PbRL baselines. This work is the first to connect RFRL with PbRL, highlighting its potential as a feedback-efficient solution. Our code is publicly available at https://github.com/rl-bandits-lab/FB-PbRL.
Jun-Jie Yang, Chia-Heng Hsu, Kui-Yuan Chen +1
Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan
Conveying complex objectives to reinforcement learning (RL) agents often requires meticulous reward engineering. Preference-based RL (PbRL) offers a promising alternative by learning reward functions from human feedback, but its scalability is hindered by high labeling costs. Inspired by advances in Video Foundation Models (ViFMs), we present Video-based Optimal Transport Preference (VOTP), a semi-supervised framework that learns effective reward functions from only a handful of labels. By leveraging optimal transport to align visual trajectories within the rich representation space of ViFMs, VOTP effectively generates high-fidelity pseudo-labels for large amounts of unlabeled data, substantially reducing human supervision. Extensive experiments across locomotion and manipulation benchmarks demonstrate the superiority of VOTP, which outperforms state-of-the-art offline PbRL methods under limited feedback budgets. We also showcase the robustness of VOTP in the presence of visual distractors and validate its utility on real robotic tasks, where it learns meaningful rewards with minimal human input.
Tung M. Luu, Hwanhee Kim, Younghwan Lee +1
Korea Advanced Institute of Science and Technology (KAIST).
Preference-based Reinforcement Learning (PbRL) entails a variety of approaches for aligning models with human intent to alleviate the burden of reward engineering. However, most previous PbRL work has not investigated the robustness to labeler errors, inevitable with labelers who are non-experts or operate under time constraints. We introduce Similarity as Reward Alignment (SARA), a simple contrastive framework that is both resilient to noisy labels and adaptable to diverse feedback formats. SARA learns a latent representation of preferred samples and computes rewards as similarities to the learned latent. On preference data with varying realistic noise rates, we demonstrate competitive and more stable performance on continuous control offline RL benchmarks, with statistically significant improvements over baselines (Wilcoxon signed-rank, p < 0.01). We also compute correlation to the environment rewards as a proxy for measuring alignment to the underlying preference criteria. We show that the SARA computed rewards display higher correlation across noise rates compared to baselines.
Sara Rajaram, R. James Cotton, Fabian H. Sinz
Institute for Computer Science and Campus Institute for Data Science, University of Göttingen, Göttingen, Germany · University Hospital Tübingen, Tübingen, Germany · International Max Planck Research School for Intelligent Systems, Tübingen, Germany +2