Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learning

Srivalli Katkuri; Maxwell Kawada; Juan Wachs

cs.LGAug 26, 2026

Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learning

Authors: Srivalli Katkuri, Maxwell Kawada, Juan Wachs

Organizations: School of Mechanical Engineering Purdue University West Lafayette, USA · Weldon School of Biomedical Engineering Purdue University West Lafayette, USA · Edwardson School of Industrial Engineering Purdue University West Lafayette, USA

Abstract

Vision-language models (VLMs) have emerged as a powerful source of supervision for reinforcement learning, enabling agents to leverage rich semantic knowledge during training. Inspired by the success of preference-based reward learning (PbRL) in reinforcement learning from human feedback (RLHF), vision-language model generated image-based preferences provide an effective source for learning reward functions. This can be done by visually comparing two outcomes through the Bradley-Terry (BT) model. However, this pairwise formulation utilizes only two observations at a time, despite VLMs being capable of ranking multiple candidates. The Plackett-Luce (PL) formulation can shape a reward model with listwise rankings as opposed to pairwise preferences, allowing for a more suited use of a VLM based ranking. In this work, to our knowledge, we introduce the first framework that combines VLM-generated preferences with the Plackett-Luce model for reward learning. We evaluate our approach on Meta-World manipulation tasks and show that Plackett-Luce (PL) reward models can train robotic policies from VLM-generated rankings as effectively as pairwise Bradley-Terry, $K$ -wise Bradley-Terry, and RL-VLM-F baselines. Across all environments, at least one PL ranking size ( $K \in \{3,4,5\}$ ) consistently performs with or outperforms other methods in mean success rate. Unlike pairwise methods, which are restricted to $K=2$ , PL supports different ranking sizes and can therefore be adapted to the environment and desired feedback format. Our best PL configuration achieves an 86% mean final success rate and matches the Oracle baseline on Drawer Open. Overall, these results demonstrate that listwise VLM preference supervision is a competitive and flexible approach to reward learning for reinforcement learning.

Figures & tables

Fig. 1: System overview for each iteration. Our framework for testing different reward models for VLM preference-based reward learning starts with a simulated robotic arm attempting a task. Scenes from these task attempts are saved into a large buffer, which are then sampled in pairs or groups (pairwise or listwise) for the VLM to generate preference labels. After preference labels are generated, the reward model is updated using either the Plackett-Luce or Bradley-Terry formulation. This reward model is then used to update a Soft Actor-Critic (SAC) policy which dictates how the robotic arm agent acts in the next iteration

Fig. 2: Visual description of each sampling method to obtain preference labels. From the buffer of images collected throughout iterations, either two observations or K observations are sampled at the same time. K refers to the number of observations that the VLM is simultaneously queried with for listwise methods. Pairwise samples are evaluated by the VLM using either the RL-VLM-F or BT-Pairwise method. The RL-VLM-F method uses two VLM queries, whereas BT-Pairwise does the same comparison with one VLM query. Listwise samples are evaluated by the VLM using either the Plackett-Luce or BT-Kwise method. Both of these methods require a single VLM query to obtain preference labels.

Fig. 3: Initial position of Sawyer robot arm for each Meta-World task.

OS	Ubuntu 24.04.3 LTS
GPU	NVIDIA GeForce RTX 3090 (24 GB)
NVIDIA Driver	580.95.05
NVCC Version	V12.0.140
CPU	AMD Ryzen 9 7950X
Python	3.12.13

TABLE I: Hardware and Software Versions

Fig. 4: Deterministic evaluation success rates for all primary methods and oracle baseline. For the Drawer Open task, RL-VLM-F was able to score the highest final success rate at 92%, with BT-Pairwise and BT-Kwise ( $K=5$ ) tied for 2nd best at 89%. Both of these methods outperformed the oracle baseline, which tied with PL ( $K=4$ ) at 86%. For the Door Close task, PL ( $K=5$ ) was the best performing method, followed by BT-Kwise ( $K=4$ ) at 52%. However, the oracle method for door close outperformed all methods with 100% success rate. In the Button Press task, BT-Kwise $K=4$ performed the best with a mean 45% success rate, followed by PL $K=4$ with a mean of 41%. Once again, the oracle baseline outperformed all methods in this environment.

Method	Drawer Open	Door Close	Button Press
PL ( $K=3$ )	$74\pm 10.7$	$40\pm 16.4$	$22\pm 5.6$
PL ( $K=4$ )	$86\pm 3.7$	$48\pm 19.3$	$41\pm 18.4$
PL ( $K=5$ )	$81\pm 10.2$	$\mathbf{54\pm 13.2}$	$26\pm 6.2$
BT-Kwise ( $K=3$ )	$83\pm 3.4$	$44\pm 18.7$	$36\pm 13.2$
BT-Kwise ( $K=4$ )	$84\pm 9.1$	$52\pm 16.0$	$\mathbf{45\pm 8.2}$
BT-Kwise ( $K=5$ )	$89\pm 7.1$	$47\pm 22.0$	$7\pm 4.4$

TABLE II: Primary Success Rates (%, Mean

\pm

SEM across five seeds)

Fig. 5: Ablation aggregate success rate across five seeds $\pm$ standard error of the mean (SEM). Overall, Bradley-Terry had a higher average success rate after 100,000 environment steps, with a final success rate of 55% compared to 49% with Plackett-Luce.

Seed	PL Success-Rate	BT-Kwise Success-Rate
0	0%	0%
1	65%	95%
2	15%	100%
3	85%	0%
4	80%	80%
Mean	49%	55%

TABLE III: Ablation Final Success Rates per Run

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 6: Number of OpenAI API queries per run method

Fig. 7: Number of input tokens used per run method

Fig. 8: Estimated cost of each run method with $M=4$ . Estimations come directly from OpenAI’s GPT-5.6 Luna pricing [ 21 ] .

Method	Seed 0	Seed 1	Seed 2	Seed 3	Seed 4	Mean $\pm$ SEM
PL ( $K=3$ )	100	75	80	35	80	$74.0\pm 10.7$
PL ( $K=4$ )	85	85	80	80	100	$86.0\pm 3.7$
PL ( $K=5$ )	85	75	45	100	100	$81.0\pm 10.2$
BT-Kwise ( $K=3$ )	75	90	85	75	90	$83.0\pm 3.4$
BT-Kwise ( $K=4$ )	85	85	100	100	50	$84.0\pm 9.1$
BT-Kwise ( $K=5$ )	65	100	100	80	100	$89.0\pm 7.1$

Appendix

TABLE IV: Drawer Open Final Success Rates by Seed (%).

Method	Seed 0	Seed 1	Seed 2	Seed 3	Seed 4	Mean $\pm$ SEM
PL ( $K=3$ )	90	5	40	60	5	$40.0\pm 16.4$
PL ( $K=4$ )	50	10	100	80	0	$48.0\pm 19.3$
PL ( $K=5$ )	60	35	15	70	90	$\mathbf{54.0\pm 13.2}$
BT-Kwise ( $K=3$ )	60	55	0	100	5	$44.0\pm 18.7$
BT-Kwise ( $K=4$ )	45	75	100	30	10	$52.0\pm 16.0$
BT-Kwise ( $K=5$ )	10	25	100	100	0	$47.0\pm 22.0$

Appendix

TABLE V: Door Close Final Success Rates by Seed (%).

Method	Seed 0	Seed 1	Seed 2	Seed 3	Seed 4	Mean $\pm$ SEM
PL ( $K=3$ )	30	25	25	30	0	$22.0\pm 5.6$
PL ( $K=4$ )	35	95	0	5	70	$41.0\pm 18.4$
PL ( $K=5$ )	15	45	30	10	30	$26.0\pm 6.2$
BT-Kwise ( $K=3$ )	40	85	20	25	10	$36.0\pm 13.2$
BT-Kwise ( $K=4$ )	60	45	60	15	45	$\mathbf{45.0\pm 8.2}$
BT-Kwise ( $K=5$ )	0	0	20	0	15	$7.0\pm 4.4$

Appendix

TABLE VI: Button Press Final Success Rates by Seed (%).

Seed	Uniform	State-Diverse
0	0%	80%
1	65%	100%
2	15%	100%
Mean $\pm$ SEM	$\mathbf{27\%\pm 20\%}$	$\mathbf{93\%\pm 7\%}$

Appendix

TABLE VII: Effect of State-Diverse Sampling on Drawer Open

Setting	$M=1$	$M=5$
Ranking size, $K$	3	3
Sampling strategy	State-diverse	State-diverse
Total rankings	182	910
Seed 0 success	65%	75%
Seed 1 success	0%	100%
Seed 2 success	10%	10%

Appendix

TABLE VIII: Effect of Increasing Feedback Groups on Door Close

OS	Ubuntu 24.04.3 LTS
GPU	NVIDIA GeForce RTX 3090 (24 GB)
NVIDIA Driver	580.173.02
NVCC Version	V12.0.140
CPU	11th Gen Intel(R) Core(TM) i9-11900KF
Python	3.12.13

Appendix

TABLE IX: Alternate Configuration

Fig. 9: Plackett-Luce CartPole success rate

Fig. 10: Bradley-Terry-Kwise CartPole success rate

Parameter	Value
Vision-Language Model
VLM backend	OpenAI
VLM model	GPT-5.6 Luna
Image detail	Original
Reasoning effort	None
VLM timeout	180 s

Appendix

TABLE X: Primary Training and Evaluation Hyperparameters

Preference-Based RL

Preference-based reinforcement learning learns policies from comparisons or rankings of behaviors instead of requiring hand-designed numerical rewards.

See all 126 papers

Automating Potential-based Reward Shaping with Vision Language Model Guidance
Jun 25, 2026Henrik Müller, Daniel KudenkoReward Shaping Sparse-Reward RL
L3S Research Center Leibniz University Hannover Hannover, 30167, Germany
VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learning
Jul 1, 2026Kuan-Chen Chen, Winston Chen, Wei-Fang Sun +1Multimodal Reward Modeling Reward Design for RL
Department of Computer Science, National Tsing Hua University · NVIDIA AI Technology Center (NVAITC)
Reducing Oracle Feedback with Vision-Language Embeddings for Preference-Based RL
Mar 30, 2026Udita Ghosh, Dripta S. Raychaudhuri, Jiachen Li +2Preference-Based RL Data-Efficient Robot Learning
Univeristy of California, Riverside · AWS AI Labs (Work done outside AWS)

Jun 25, 2026J/K move · Enter open · S save

Automating Potential-based Reward Shaping with Vision Language Model Guidance

Henrik Müller, Daniel Kudenko

L3S Research Center Leibniz University Hannover Hannover, 30167, Germany

Reward Shaping Sparse-Reward RL Policy Learning Reward Design for RL

Open paper AskPDF arXiv

Sparse rewards are inherently challenging for reinforcement learning agents as they lack intermediate feedback to guide exploration and to correctly attribute the sparse success rewards to relevant parts of the trajectory. Naive reward shaping can induce reward hacking, yielding policies that exploit auxiliary signals instead of solving the intended task. Potential-based reward shaping (PBRS) guarantees preservation of the optimal policy set, but requires the definition of a heuristic potential function over the state space. In this work, we introduce the VLM-guided PBRS framework VLM-PBRS that learns the potential function directly from vision language model (VLM) feedback. We query a lightweight VLM to obtain preferences over image pairs and train a model of the potential function using these preferences. As this approach is based on potential-based reward shaping, it preserves the original optimal policies, and removes the need for expert-designed reward shaping terms. Because large VLMs are prohibitively expensive to invoke repeatedly during policy learning, we employ smaller, more computationally efficient VLMs. Although the resulting preference labels are less accurate, empirical evidence shows that the preference labels can still be used to accelerate learning. We validate our method empirically in the Meta-World and Franka Kitchen environments and highlight the connection between VLM preference label accuracy and sample efficiency improvements. Our contributions are threefold: (1) the first application of VLM preference-based learning to synthesize a potential function for PBRS, (2) a principled, low-cost solution that leverages small VLMs, and (3) extensive empirical demonstration of improved sample efficiency and robustness to reward hacking.

Abstract

Preference-Based RL

Explore similar work

Automating Potential-based Reward Shaping with Vision Language Model Guidance

VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learning

Reducing Oracle Feedback with Vision-Language Embeddings for Preference-Based RL