Scaling Vision-Language Reward Learning for Robot Manipulation in Parallel Simulation

Lobna Joualy; Eric Demeester; Nikolaos Tsiogkas

cs.ROSep 18, 2026

Scaling Vision-Language Reward Learning for Robot Manipulation in Parallel Simulation

Authors: Lobna Joualy, Eric Demeester, Nikolaos Tsiogkas

Organizations: Department of Computer Science and Department of Mechanical Engineering, KU Leuven, 3590 Diepenbeek, Belgium.

Abstract

Vision-language models (VLMs) can replace human annotators in preference-based reward learning, but sequential API requests and single-environment data collection make training slow and costly. We present RAPID (Reward learning with Adaptive Parallel Image Diversity), a system that couples GPU-parallel rollout with data-aware policy updates, single-request preference labeling, automatic reward stabilization, and representative image sampling. We evaluate these components on five Franka Panda manipulation tasks in IsaacLab. Parallel rollout and adaptive updates provide the first substantial reduction in training time: under matched two-stage prompting, mean runtime falls from 9.18 to 3.13 hours. With all RAPID components enabled, training completes in 1.15 hours using 896 rather than 19,840 API calls per run, and aggregate final success rises from 86.3% to 98.7%. This represents an 8.0 $\times$ end-to-end speedup and a 95.5% reduction in API usage. An offline evaluation with Gemma~3 12B and GPT-4.1 mini demonstrates that single-request prompting reduces labeling latency and cost across both models. Code is available at: https://github.com/rapid-vlm/rapid-vlm-rl.

Figures & tables

Panel of Figure 1 — Fig. 1 : System overview. (a) Active phase: Parallel environments collect experience and render observations via a tiled camera. Representative sampling selects diverse images for the VLM buffer. Periodically, image pairs are sampled and labeled by a vision-language model using single-request queries. The reward model ensemble trains on preferences and provides learning signals to the policy. (b) Frozen phase: When correlation-based monitoring meets the fixed stability criteria, preference sampling, reward updates, and VLM queries stop. Policy training continues with the frozen reward model.

Panel of Figure 2 — Fig. 2 : Manipulation task environments. Left to right: Open Drawer, Open Window, Push Button (top-down), Soccer (ball-into-goal), Sweep Into (cube-into-hole).

Fig. 3 : Success versus wall-clock time (mean $\pm$ sample standard deviation, three seeds and linear interpolation at five-minute intervals over the shared observed time range).

Task	C1	C2	C3	C4	C5
Open Drawer	100.0 $\pm$ 0.0	93.3 $\pm$ 5.8	96.3 $\pm$ 6.4	98.7 $\pm$ 0.6	95.0 $\pm$ 3.6
Open Window	65.3 $\pm$ 11.0	44.7 $\pm$ 27.9	95.3 $\pm$ 8.1	96.7 $\pm$ 3.5	100.0 $\pm$ 0.0
Push Button	84.7 $\pm$ 18.6	57.0 $\pm$ 25.2	99.3 $\pm$ 1.2	99.3 $\pm$ 1.2	100.0 $\pm$ 0.0
Soccer	84.7 $\pm$ 18.0	97.0 $\pm$ 2.6	99.0 $\pm$ 1.0	98.7 $\pm$ 1.2	100.0 $\pm$ 0.0
Sweep Into	96.7 $\pm$ 1.5	93.0 $\pm$ 5.0	98.0 $\pm$ 1.7	95.3 $\pm$ 3.8	98.3 $\pm$ 0.6
Aggregate	86.3 $\pm$ 16.5	77.0 $\pm$ 26.8	97.6 $\pm$ 4.3	97.7 $\pm$ 2.6	98.7 $\pm$ 2.4

TABLE I : Final success (%, mean

\pm

sample standard deviation over three seeds), where each seed is averaged over its last 10 evaluations.

Config.	Envs	Prompt	Stability	Representative	Success (%)	Time (h)	API calls
C1	1	two	no	no	86.3 $\pm$ 16.5	9.18 $\pm$ 0.73	19,840 $\pm$ 0
C2	1	one	no	no	77.0 $\pm$ 26.8	7.75 $\pm$ 0.36	9,920 $\pm$ 0
C3	50	one	no	no	97.6 $\pm$ 4.3	2.74 $\pm$ 0.21	9,920 $\pm$ 0
C4	50	one	yes	no	97.7 $\pm$ 2.6	1.31 $\pm$ 0.47	1,888 $\pm$ 2,537
C5	50	one	yes	yes	98.7 $\pm$ 2.4	1.15 $\pm$ 0.08	896 $\pm$ 656
50-env. two-stage	50	two	no	no	87.1 $\pm$ 25.7	3.13 $\pm$ 0.27	19,840 $\pm$ 0

TABLE II : Component ablation and prompt-format comparison. C1–C5 form the sequential ablation; the final row is the additional 50-environment variant with two-stage prompting. Success, time, and API calls are mean

\pm

sample standard deviation over 15 task–seed runs.

Fig. 4 : Offline comparison of Gemma 3 12B and GPT-4.1 mini under single-request and sequential prompting. Outcomes use 750 clear pairs, while latency and cost cover the full 1,000-pair benchmark.

Representation	Success (%)	Time (h)	API calls
16 $\times$ 16 pixels	98.7 $\pm$ 2.4	1.15	896
32 $\times$ 32 pixels	96.0 $\pm$ 6.5	1.36	2,373
64 $\times$ 64 pixels	87.9 $\pm$ 20.6	1.53	3,128
Reward encoder	89.7 $\pm$ 15.2	1.65	3,771

TABLE III : Sampling-feature ablation; each row summarizes 15 task–seed runs. Success is mean

\pm

sample standard deviation; other columns are means.

Robotic RL

Robotic reinforcement learning uses trial-and-error learning to help robots control, navigate, and manipulate objects.

See all 287 papers

Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models
Mar 17, 2026Yanru Wu, Weiduo Yuan, Esteban Martinez Licon +4Long-Horizon Robotic Manipulation Robot Skill Learning
USC Physical Superintelligence Lab
Automating Potential-based Reward Shaping with Vision Language Model Guidance
Jun 25, 2026Henrik Müller, Daniel KudenkoReward Shaping Sparse-Reward RL
L3S Research Center Leibniz University Hannover Hannover, 30167, Germany
Reducing Temporal Redundancy for Efficient Vision-Language-Action Inference
Jul 14, 2026Yuzhou Wu, Yuxin Zheng, Muchun Niu +6Efficient VLA Model Inference Efficient VLA Models
1Tianji KernalMind co ltd · 2Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · 3Shanghai Jiao Tong University, Shanghai, China +2

Mar 17, 2026J/K move · Enter open · S save

Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models

Yanru Wu, Weiduo Yuan, Esteban Martinez Licon, Ang Qi+3

USC Physical Superintelligence Lab

Long-Horizon Robotic Manipulation Robot Skill Learning Robotic RL Multimodal Reward Modeling Visual RL VLMs for Robotics+1

Open paper AskPDF arXiv

Reinforcement Learning (RL) has shown strong potential for improving robotic manipulation policies, yet its practical use remains bottlenecked by the difficulty of specifying reward functions that are both semantically meaningful and reusable across tasks. In this paper, we propose Large Reward Models (LRMs), a framework that adapts foundation VLMs into frame-level reward generators for robot policy refinement. We specialize a state-of-the-art VLM on a multi-source dataset spanning real-world robot trajectories, human-object interactions, and simulated manipulation environments. Unlike prior approaches that mainly evaluate trajectories post-hoc, LRMs expose multiple reward interfaces from visual observations: progress estimation, task completion, and temporal contrastive comparison. Starting from an imitation-learned policy, we use these VLM-derived rewards to guide PPO refinement on held-out long-horizon manipulation tasks. Our experiments show that LRM progress rewards provide the strongest non-privileged online refinement signal, improving the IL baseline and narrowing the gap to privileged environment rewards. We further deploy progress rewards for progress-weighted behavioral cloning on four real-world manipulation tasks spanning two robot platforms, improving over SFT on all four tasks. These results suggest that modality-specific specialization of foundation VLMs can provide practical visual reward signals for both simulated policy refinement and physical robot self-improvement without hand-coded task rewards.

Abstract

Robotic RL

Explore similar work

Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models

Automating Potential-based Reward Shaping with Vision Language Model Guidance

Reducing Temporal Redundancy for Efficient Vision-Language-Action Inference