SPEAR: Selective Preference Elicitation with Adaptive Rewards for VLM-Guided Robotic Manipulation

Zewen Wu; Ruizhe Liu; Pei Zhou; Feng Chen; Yanchao Yang

cs.ROOct 15, 2026

SPEAR: Selective Preference Elicitation with Adaptive Rewards for VLM-Guided Robotic Manipulation

Authors: Zewen Wu, Ruizhe Liu, Pei Zhou, Feng Chen, Yanchao Yang

Organizations: HKU Musketeers Foundation Institute of Data Science, The University of Hong Kong, Hong Kong SAR, China. · Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China.

Abstract

Recent Vision-Language Models (VLMs) enable automated preference labeling for robotic manipulation, yet their deployment remains impractical since each query incurs substantial inference cost, while VLMs frequently abstain on visually challenging comparisons. We propose SPEAR, a framework that transforms VLMs into efficient preference oracles via two synergistic mechanisms. First, we design distinguishability-aware query selection to reduce abstention rates by targeting only informative and distinguishable comparisons. Second, we deploy an adaptive reward transition that bootstraps exploration with VLM-synthesized guidance before preference-based refinement. Experiments on the manipulation benchmarks demonstrate that SPEAR achieves 88.9% average success, outperforming baselines by 23.3 percentage points, while significantly reducing abstention and wall-clock training time on contact-rich tasks. SPEAR learns three Mobile ALOHA manipulation tasks fully online on real hardware, demonstrating end-to-end real-world applicability. By addressing query efficiency and exploration challenges, SPEAR establishes VLMs as practical preference oracles for scalable robot learning without manual reward engineering.

Figures & tables

Fig. 1: Overview of SPEAR . (1) Distinguishability-aware query selection (top, blue) predicts VLM response likelihood via multi-scale features and weights information gain accordingly, selecting only answerable and informative trajectory pairs for querying; (2) Adaptive reward transition (bottom, green) bootstraps exploration using VLM-synthesized warm-up rewards before transitioning to refined preference-based learning. Together, these mechanisms enable efficient online learning by ensuring both high-quality query selection and sufficient behavioral diversity for meaningful preference elicitation.

Method	Drawer-Open	Drawer-Close	Button-Press-Topdown	Soccer	Door-Close	Window-Open	Overall
VLM-RM [ 5 ]	0.0 $\pm$ 0.0%	0.2 $\pm$ 0.5%	0.0 $\pm$ 0.0%	15.7 $\pm$ 15.3%	7.5 $\pm$ 10.5%	29.8 $\pm$ 25.9%	8.9 $\pm$ 8.7%
Text2Reward [ 6 ]	17.0 $\pm$ 30.4%	79.2 $\pm$ 6.4%	22.3 $\pm$ 17.7%	6.5 $\pm$ 5.9%	99.5 $\pm$ 0.9%	54.1 $\pm$ 24.0%	46.4 $\pm$ 14.2%
Pebble [ 19 ]	1.5 $\pm$ 2.0%	87.1 $\pm$ 5.5%	40.6 $\pm$ 9.5%	70.5 $\pm$ 26.1%	91.0 $\pm$ 6.8%	7.8 $\pm$ 5.5%	49.8 $\pm$ 9.2%
RL-VLM-F [ 3 ]	11.3 $\pm$ 11.2%	98.2 $\pm$ 3.6%	12.9 $\pm$ 13.4%	88.2 $\pm$ 7.9%	100.0 $\pm$ 0.0%	81.1 $\pm$ 2.7%	65.3 $\pm$ 6.5%
SPEAR (ours)	87.0 $\pm$ 15.9%	100.0 $\pm$ 0.0%	58.2 $\pm$ 16.0%	89.9 $\pm$ 11.7%	100.0 $\pm$ 0.0%	94.3 $\pm$ 4.3%	88.2 $\pm$ 8.0%

TABLE I: Success rates (mean

\pm

std) across tasks. Middle = 45% - 55% steps.

Method	Drawer-Open	Drawer-Close	Button-Press-Topdown	Soccer	Door-Close	Window-Open	Overall
VLM-RM [ 5 ]	0.0 $\pm$ 0.0%	0.0 $\pm$ 0.0%	0.0 $\pm$ 0.0%	14.1 $\pm$ 22.3%	3.7 $\pm$ 5.1%	71.6 $\pm$ 14.1%	14.9 $\pm$ 6.9%
Text2Reward [ 6 ]	34.8 $\pm$ 32.9%	95.8 $\pm$ 4.8%	53.7 $\pm$ 27.5%	8.9 $\pm$ 5.4%	99.8 $\pm$ 0.5%	87.7 $\pm$ 12.7%	63.4 $\pm$ 14.0%
Pebble [ 19 ]	10.0 $\pm$ 11.6%	88.1 $\pm$ 3.2%	41.2 $\pm$ 7.7%	91.0 $\pm$ 13.3%	73.0 $\pm$ 11.5%	5.3 $\pm$ 4.1%	51.4 $\pm$ 8.6%
RL-VLM-F [ 3 ]	33.9 $\pm$ 21.4%	98.8 $\pm$ 3.1%	17.9 $\pm$ 16.6%	75.4 $\pm$ 11.6%	95.7 $\pm$ 7.6%	69.2 $\pm$ 5.3%	65.1 $\pm$ 10.9%
SPEAR (ours)	86.4 $\pm$ 12.7%	100.0 $\pm$ 0.0%	67.0 $\pm$ 17.8%	95.1 $\pm$ 3.9%	100.0 $\pm$ 0.0%	84.9 $\pm$ 5.7%	88.9 $\pm$ 6.7%

TABLE II: Success rates (mean

\pm

std) across tasks. Final = last 10% steps.

Fig. 2: Learning curves across six Meta-World tasks. Our method (green) consistently outperforms baselines in both mid-range and final performance. Shaded regions indicate standard deviation over 5 seeds.

Method	Query Efficiency			Time Consumption
	Total Q	Abst. Rate	Effective Q	Aver. Q (s)	DistTrain (s)	Total (h)
RL-VLM-F	1640	0.555	729	29.87	–	13.6
SPEAR (ours)	960	0.296	676	31.42	114.5	9.1

TABLE III: Query efficiency and wall-clock cost at 30% success. SPEAR effectively reduces the abstention rate and lowers the training wall-clock time overhead.

Frame Gap	Metric (%)	Drawer-Open	Door-Close	Soccer	Button-Press	Avg
1–5	Abstain	61.2	88.8	77.2	76.6	75.9
1–5	Acc	61.3	75.0	53.7	56.0	61.5
6–10	Abstain	61.6	87.5	64.6	68.1	70.5
6–10	Acc	79.6	100.0	91.3	68.6	84.9
11–20	Abstain	49.2	82.0	72.7	65.6	67.4
11–20	Acc	91.3	100.0	100.0	97.0	97.1

TABLE IV: Diagnostics of averaged 5 VLM backbones with conservative prompt. Distinctiveness increases

\Rightarrow

abstention decreases and accuracy increases.

Fig. 3: Average fit probability of 5 VLM backbone behaviors. As visual distinguishability increases, the abstention rate decreases and the accuracy increases.

Method	Mid (450k–550k)	End (900k–1000k)
Random	1.1 $\pm$ 1.8 %	14.0 $\pm$ 15.9 %
Entropy-only	3.3 $\pm$ 4.2 %	21.0 $\pm$ 19.0 %
Distinguishability-only	21.9 $\pm$ 19.8 %	69.7 $\pm$ 24.3 %
Integrated (ours)	58.4 $\pm$ 23.8 %	80.7 $\pm$ 13.2 %

TABLE V: Ablation on query acquisition mechanisms. Integrated weighting (ours) achieves the best performance.

Method	Mid (450k–550k)	End (900k–1000k)
Baseline	1.7 $\pm$ 2.5 %	24.8 $\pm$ 16.2 %
+ Relay Exploration	26.1 $\pm$ 21.6 %	73.6 $\pm$ 19.5 %
+ Reward Transition	39.8 $\pm$ 17.8 %	67.0 $\pm$ 21.9 %
Full Exploration (ours)	92.7 $\pm$ 8.9 %	88.3 $\pm$ 11.7 %

TABLE VI: Ablation on exploration mechanisms. Combining two mechanisms achieves the best performance.

Configuration	MSE $\downarrow$	MAE $\downarrow$
Global only ( $\phi_{\text{rel}}$ )	0.1542	0.2973
Local only ( $\phi_{\text{loc}}$ )	0.1662	0.3380
Combined (ours)	0.1508	0.2950
w/o calibration	0.1610	0.3232
w/ calibration (ours)	0.1515	0.2978

TABLE VII: Feature/calibration ablation. Multi-scale features and temperature scaling improve prediction reliability.

Fig. 4: Real-robot experiment task settings. (a) drawer-open . (b) window-open . (c) soccer .

Active Preference Learning

See all 16 papers

Scaling Vision-Language Reward Learning for Robot Manipulation in Parallel Simulation
Sep 18, 2026Lobna Joualy, Eric Demeester, Nikolaos TsiogkasRobotic RL Preference-Based RL
Department of Computer Science and Department of Mechanical Engineering, KU Leuven, 3590 Diepenbeek, Belgium.
SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation
Jun 9, 2026Qianzhong Chen, Hau Zheng, Justin Yu +8RL for Robotics Reward Modeling
Stanford University · UC Berkeley · Shanghai Jiao Tong University
Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents
Apr 15, 2026Angelo Moroncelli, Roberto Zanetti, Marco Maccarini +1RL for Robotics Data-Efficient Robot Learning
University of Applied Science and Arts of Southern Switzerland, Department of Innovative Technologies, IDSIA-SUPSI, Lugano, Switzerland · Università della Svizzera Italiana, Faculty of Informatics, Lugano, Switzerland · Politecnico di Milano, Mechanical Department, Milano, Italy

Sep 18, 2026J/K move · Enter open · S save

Scaling Vision-Language Reward Learning for Robot Manipulation in Parallel Simulation

Lobna Joualy, Eric Demeester, Nikolaos Tsiogkas

Department of Computer Science and Department of Mechanical Engineering, KU Leuven, 3590 Diepenbeek, Belgium.

Robotic RL Preference-Based RL Reinforcement Learning Data-Efficient Robot Learning VLMs for Robotics

Open paper AskPDF arXiv

Vision-language models (VLMs) can replace human annotators in preference-based reward learning, but sequential API requests and single-environment data collection make training slow and costly. We present RAPID (Reward learning with Adaptive Parallel Image Diversity), a system that couples GPU-parallel rollout with data-aware policy updates, single-request preference labeling, automatic reward stabilization, and representative image sampling. We evaluate these components on five Franka Panda manipulation tasks in IsaacLab. Parallel rollout and adaptive updates provide the first substantial reduction in training time: under matched two-stage prompting, mean runtime falls from 9.18 to 3.13 hours. With all RAPID components enabled, training completes in 1.15 hours using 896 rather than 19,840 API calls per run, and aggregate final success rises from 86.3% to 98.7%. This represents an 8.0

\times

end-to-end speedup and a 95.5% reduction in API usage. An offline evaluation with Gemma~3 12B and GPT-4.1 mini demonstrates that single-request prompting reduces labeling latency and cost across both models. Code is available at: https://github.com/rapid-vlm/rapid-vlm-rl.

Abstract

Active Preference Learning

Explore similar work

Scaling Vision-Language Reward Learning for Robot Manipulation in Parallel Simulation

SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation

Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents