Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learning
Authors: Srivalli Katkuri, Maxwell Kawada, Juan Wachs
Organizations: School of Mechanical Engineering Purdue University West Lafayette, USA · Weldon School of Biomedical Engineering Purdue University West Lafayette, USA · Edwardson School of Industrial Engineering Purdue University West Lafayette, USA
Vision-language models (VLMs) have emerged as a powerful source of supervision for reinforcement learning, enabling agents to leverage rich semantic knowledge during training. Inspired by the success of preference-based reward learning (PbRL) in reinforcement learning from human feedback (RLHF), vision-language model generated image-based preferences provide an effective source for learning reward functions. This can be done by visually comparing two outcomes through the Bradley-Terry (BT) model. However, this pairwise formulation utilizes only two observations at a time, despite VLMs being capable of ranking multiple candidates. The Plackett-Luce (PL) formulation can shape a reward model with listwise rankings as opposed to pairwise preferences, allowing for a more suited use of a VLM based ranking. In this work, to our knowledge, we introduce the first framework that combines VLM-generated preferences with the Plackett-Luce model for reward learning. We evaluate our approach on Meta-World manipulation tasks and show that Plackett-Luce (PL) reward models can train robotic policies from VLM-generated rankings as effectively as pairwise Bradley-Terry, K-wise Bradley-Terry, and RL-VLM-F baselines. Across all environments, at least one PL ranking size (K∈{3,4,5}) consistently performs with or outperforms other methods in mean success rate. Unlike pairwise methods, which are restricted to K=2, PL supports different ranking sizes and can therefore be adapted to the environment and desired feedback format. Our best PL configuration achieves an 86% mean final success rate and matches the Oracle baseline on Drawer Open. Overall, these results demonstrate that listwise VLM preference supervision is a competitive and flexible approach to reward learning for reinforcement learning.
Figures & tables
Fig. 1: System overview for each iteration. Our framework for testing different reward models for VLM preference-based reward learning starts with a simulated robotic arm attempting a task. Scenes from these task attempts are saved into a large buffer, which are then sampled in pairs or groups (pairwise or listwise) for the VLM to generate preference labels. After preference labels are generated, the reward model is updated using either the Plackett-Luce or Bradley-Terry formulation. This reward model is then used to update a Soft Actor-Critic (SAC) policy which dictates how the robotic arm agent acts in the next iteration
Fig. 2: Visual description of each sampling method to obtain preference labels. From the buffer of images collected throughout iterations, either two observations or K observations are sampled at the same time. K refers to the number of observations that the VLM is simultaneously queried with for listwise methods. Pairwise samples are evaluated by the VLM using either the RL-VLM-F or BT-Pairwise method. The RL-VLM-F method uses two VLM queries, whereas BT-Pairwise does the same comparison with one VLM query. Listwise samples are evaluated by the VLM using either the Plackett-Luce or BT-Kwise method. Both of these methods require a single VLM query to obtain preference labels.
Fig. 3: Initial position of Sawyer robot arm for each Meta-World task.
OS
Ubuntu 24.04.3 LTS
GPU
NVIDIA GeForce RTX 3090 (24 GB)
NVIDIA Driver
580.95.05
NVCC Version
V12.0.140
CPU
AMD Ryzen 9 7950X
Python
3.12.13
TABLE I: Hardware and Software Versions
Fig. 4: Deterministic evaluation success rates for all primary methods and oracle baseline. For the Drawer Open task, RL-VLM-F was able to score the highest final success rate at 92%, with BT-Pairwise and BT-Kwise ( K=5 ) tied for 2nd best at 89%. Both of these methods outperformed the oracle baseline, which tied with PL ( K=4 ) at 86%. For the Door Close task, PL ( K=5 ) was the best performing method, followed by BT-Kwise ( K=4 ) at 52%. However, the oracle method for door close outperformed all methods with 100% success rate. In the Button Press task, BT-Kwise K=4 performed the best with a mean 45% success rate, followed by PL K=4 with a mean of 41%. Once again, the oracle baseline outperformed all methods in this environment.
Method
Drawer Open
Door Close
Button Press
PL ( K=3 )
74±10.7
40±16.4
22±5.6
PL ( K=4 )
86±3.7
48±19.3
41±18.4
PL ( K=5 )
81±10.2
54±13.2
26±6.2
BT-Kwise ( K=3 )
83±3.4
44±18.7
36±13.2
BT-Kwise ( K=4 )
84±9.1
52±16.0
45±8.2
BT-Kwise ( K=5 )
89±7.1
47±22.0
7±4.4
TABLE II: Primary Success Rates (%, Mean ± SEM across five seeds)
Fig. 5: Ablation aggregate success rate across five seeds ± standard error of the mean (SEM). Overall, Bradley-Terry had a higher average success rate after 100,000 environment steps, with a final success rate of 55% compared to 49% with Plackett-Luce.
Seed
PL Success-Rate
BT-Kwise Success-Rate
0
0%
0%
1
65%
95%
2
15%
100%
3
85%
0%
4
80%
80%
Mean
49%
55%
TABLE III: Ablation Final Success Rates per Run
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 6: Number of OpenAI API queries per run method
Fig. 7: Number of input tokens used per run method
Fig. 8: Estimated cost of each run method with M=4 . Estimations come directly from OpenAI’s GPT-5.6 Luna pricing [ 21 ] .
Method
Seed 0
Seed 1
Seed 2
Seed 3
Seed 4
Mean ± SEM
PL ( K=3 )
100
75
80
35
80
74.0±10.7
PL ( K=4 )
85
85
80
80
100
86.0±3.7
PL ( K=5 )
85
75
45
100
100
81.0±10.2
BT-Kwise ( K=3 )
75
90
85
75
90
83.0±3.4
BT-Kwise ( K=4 )
85
85
100
100
50
84.0±9.1
BT-Kwise ( K=5 )
65
100
100
80
100
89.0±7.1
Appendix
TABLE IV: Drawer Open Final Success Rates by Seed (%).
Method
Seed 0
Seed 1
Seed 2
Seed 3
Seed 4
Mean ± SEM
PL ( K=3 )
90
5
40
60
5
40.0±16.4
PL ( K=4 )
50
10
100
80
0
48.0±19.3
PL ( K=5 )
60
35
15
70
90
54.0±13.2
BT-Kwise ( K=3 )
60
55
0
100
5
44.0±18.7
BT-Kwise ( K=4 )
45
75
100
30
10
52.0±16.0
BT-Kwise ( K=5 )
10
25
100
100
0
47.0±22.0
Appendix
TABLE V: Door Close Final Success Rates by Seed (%).
Method
Seed 0
Seed 1
Seed 2
Seed 3
Seed 4
Mean ± SEM
PL ( K=3 )
30
25
25
30
0
22.0±5.6
PL ( K=4 )
35
95
0
5
70
41.0±18.4
PL ( K=5 )
15
45
30
10
30
26.0±6.2
BT-Kwise ( K=3 )
40
85
20
25
10
36.0±13.2
BT-Kwise ( K=4 )
60
45
60
15
45
45.0±8.2
BT-Kwise ( K=5 )
0
0
20
0
15
7.0±4.4
Appendix
TABLE VI: Button Press Final Success Rates by Seed (%).
Seed
Uniform
State-Diverse
0
0%
80%
1
65%
100%
2
15%
100%
Mean ± SEM
27%±20%
93%±7%
Appendix
TABLE VII: Effect of State-Diverse Sampling on Drawer Open
Setting
M=1
M=5
Ranking size, K
3
3
Sampling strategy
State-diverse
State-diverse
Total rankings
182
910
Seed 0 success
65%
75%
Seed 1 success
0%
100%
Seed 2 success
10%
10%
Appendix
TABLE VIII: Effect of Increasing Feedback Groups on Door Close
Preference-based reinforcement learning learns policies from comparisons or rankings of behaviors instead of requiring hand-designed numerical rewards.
Sparse rewards are inherently challenging for reinforcement learning agents as they lack intermediate feedback to guide exploration and to correctly attribute the sparse success rewards to relevant parts of the trajectory. Naive reward shaping can induce reward hacking, yielding policies that exploit auxiliary signals instead of solving the intended task. Potential-based reward shaping (PBRS) guarantees preservation of the optimal policy set, but requires the definition of a heuristic potential function over the state space. In this work, we introduce the VLM-guided PBRS framework VLM-PBRS that learns the potential function directly from vision language model (VLM) feedback. We query a lightweight VLM to obtain preferences over image pairs and train a model of the potential function using these preferences. As this approach is based on potential-based reward shaping, it preserves the original optimal policies, and removes the need for expert-designed reward shaping terms. Because large VLMs are prohibitively expensive to invoke repeatedly during policy learning, we employ smaller, more computationally efficient VLMs. Although the resulting preference labels are less accurate, empirical evidence shows that the preference labels can still be used to accelerate learning. We validate our method empirically in the Meta-World and Franka Kitchen environments and highlight the connection between VLM preference label accuracy and sample efficiency improvements. Our contributions are threefold: (1) the first application of VLM preference-based learning to synthesize a potential function for PBRS, (2) a principled, low-cost solution that leverages small VLMs, and (3) extensive empirical demonstration of improved sample efficiency and robustness to reward hacking.
Henrik Müller, Daniel Kudenko
L3S Research Center Leibniz University Hannover Hannover, 30167, Germany
Designing effective reward functions remains a major challenge in reinforcement learning (RL), particularly in open-ended environments where task goals are abstract and difficult to quantify. In this work, we present VLM-AR3L, a framework that leverages Vision-Language Models (VLMs) to provide both absolute and relative rewards for RL. VLM-AR3L interprets an agent's visual observations in the context of a natural language task goal, and learns both absolute and relative rewards from VLM-generated preference labels. The absolute reward model predicts scalar evaluations for individual states, while the relative reward model compares consecutive observations to infer progress or regression toward the task goal. Their integration combines the stability of state-based evaluation with the robustness of comparative supervision. We evaluate VLM-AR3L across benchmarks spanning classic control, manipulation, and open-world embodied tasks, with a particular focus on Minecraft given its visual complexity and long-horizon decision-making requirements. Experimental results show that VLM-AR3L consistently outperforms prior VLM-based reward learning methods.
Kuan-Chen Chen, Winston Chen, Wei-Fang Sun +1
Department of Computer Science, National Tsing Hua University · NVIDIA AI Technology Center (NVAITC)
Preference-based reinforcement learning can learn effective reward functions from comparisons, but its scalability is constrained by the high cost of oracle feedback. Lightweight vision-language embedding (VLE) models provide a cheaper alternative, but their noisy outputs limit their effectiveness as standalone reward generators. To address this challenge, we propose ROVED, a hybrid framework that combines VLE-based supervision with targeted oracle feedback. Our method uses the VLE to generate segment-level preferences and defers to an oracle only for samples with high uncertainty, identified through a filtering mechanism. In addition, we introduce a parameter-efficient fine-tuning method that adapts the VLE with the obtained oracle feedback in order to improve the model over time in a synergistic fashion. This ensures the retention of the scalability of embeddings and the accuracy of oracles, while avoiding their inefficiencies. Across multiple robotic manipulation tasks, ROVED matches or surpasses prior preference-based methods while reducing oracle queries by up to 80%. Remarkably, the adapted VLE generalizes across tasks, yielding cumulative annotation savings of up to 90%, highlighting the practicality of combining scalable embeddings with precise oracle supervision for preference-based RL.
Udita Ghosh, Dripta S. Raychaudhuri, Jiachen Li +2
Univeristy of California, Riverside · AWS AI Labs (Work done outside AWS)