cs.LGOct 11, 2026

Harnessing VLM in Embodied RL: From Reward Regressor to Success Detector via Exploration-Augmented Key-Observation Gating

Authors: Xin Liu, Yutong Jiang, Jian Su, Ziheng Wang, Chuang Wang, Haoran Li, Dongbin Zhao

Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of the Chinese Academy of Sciences · AGIBOT

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

VLMs provide promising solutions to embodied reward design. Most current methods that train VLMs as reward regressors not only require extensive annotations but are also primarily designed for finetuning pretrained policies, and perform less well in the more challenging from-scratch RL setting. This issue stems from a fundamental mismatch: regressing precise rewards is misaligned with what VLMs naturally excel at. We posit that it is more aligned with VLMs' inherent multi-modal VQA capabilities for the models to function as success detectors, and propose VLM success reward via Exploration-Augmented Key-Observation Gating (VLMeg) to well harness VLM for embodied reward design. As the core of VLMeg, the proposed exploration-augmented key-observation gating first determines whether the current RL episode is judgment-worthy based on its similarity to a generated success trajectory. Then, the key observation with the lowest confusion from the judgment-worthy episode will be selected for VLM evaluation as episodic reward. This gating mechanism not only avoids redundant calls but also enables higher detection accuracy of VLM. Considering the sparsity of judgment-worthy episodes, the advanced exploration incentives based on the generated trajectory are introduced, enabling more efficient exploration toward judgment-worthy trajectories for more VLM reward triggers and thus gaining higher RL efficiency. Extensive experiments in MetaWorld demonstrate VLMeg achieves state-of-the-art from-scratch RL, outperforming advanced VLM-based baselines on 12 challenging reward-free tasks. Furthermore, VLMeg also enables effective RL without any ground-truth reward in 3 challenging real-world experiments. Code will be open after acceptance at https://github.com/Raina915/VLMeg.

Figures & tables

Figure 1: Regressing precise rewards is misaligned with what VLMs naturally excel at. By contrast, serving as success detectors is more suitable for VLMs because it fully leverages their innate visual-question answering (VQA) capabilities. VLMeg is correspondingly proposed to well harness VLM in reward design, enabling more accurate embodied reward and more efficient RL.
Figure 2: Overview of VLMeg. First, successful frames are extracted from the generated expert videos. During rollout, each observation oto_{t} is compared with the success frames. If the maximum similarity exceeds the threshold τsim\tau_{\mathrm{sim}} , the trajectory is considered worthy of VLM verification; otherwise, rollout continues until the horizon TT , and trajectories that never satisfy the threshold skip VLM evaluation. For qualified trajectories, the observation most similar to the success-frame set is selected as the key observation. The initial frame, success reference, and selected key observation are then sent to the frozen VLM for success detection, and the resulting episodic success reward is combined with exploration incentives to improve RL efficiency.
VLM inputAcc.
w/ Reference Frames100%
w/o Reference Frames42%
Table 1: Motivation study for context-aware VLM success detection on faucet-close . The dataset used is described in Appendix C.
DatasetAcc.(successful data)Acc.(failing data)
Deviated20%80%
Near100%100%
Table 2: We construct two datasets: Deviated and Near. Deviated contains images where the robot arm states significantly deviate from the expert frames, while Near contains images where the robot arm states are close to those in the expert frames. More details about the datasets are described in Appendix C.
Figure 3: Simulation results on the 12 MetaWorld reward-free manipulation tasks. VLMeg is compared with representative VLM-based reward baselines under from-scratch RL using the same task prompts and expert-video budget.
Figure 4: Smoothed training curves on real-world Franka manipulation tasks. VLMeg enables better RL efficiency and lower intervention rates than Robo-Reward, demonstrating its effectiveness in physical manipulation RL.
Figure 5: Ablation results on four MetaWorld tasks. The comparison isolates the contribution of dense incentives, gated key-observation selection, and episodic VLM success verification.
Figure 6: Comparison of cumulative exploration incentives and VLM rewards obtained by different trajectories.
Figure 7: The left figure shows task success rate and VLM call rate during training. The right figure shows the number of VLM calls and task success rate before policy convergence (150k steps). VLMeg is our proposed method, while VLMeg-w/o-gating directly uses the last frame of the trajectory as the key observation.
Figure 8: Ablation results on six MetaWorld tasks. The comparison isolates the contribution of key-observation Gating, episodic VLM success verification, and exploration incentives.
Figure 9: Hyper-parameter sensitivity of the success-reward weight λ\lambda on two MetaWorld tasks. The curves compare training success rates with λ=1\lambda=1 , λ=10\lambda=10 , and λ=50\lambda=50 . A small λ\lambda weakens the episodic VLM success signal, while an overly large λ\lambda may overemphasize sparse success verification and reduce robustness on some tasks. The setting λ=10\lambda=10 provides the most consistent performance across the evaluated tasks.
Figure 10: Comparison between different VLM backbones in VLMeg on four MetaWorld tasks. GPT-5.5 and GPT-4.1 represent closed-source large VLMs, while Cosmos denotes the open-source smaller model Cosmos-Reason2-8B. The closed-source large VLMs provide substantially stronger success-detection signals than Cosmos-Reason2-8B, leading to better RL performance. GPT-5.5 is generally more stable and robust across tasks, while GPT-4.1 can also achieve strong performance on several tasks.
Figure 11: Effectiveness of key-observation gating on faucet-close , faucet-open , and plate-slide-side . The number of training steps shown in each task title denotes the point at which VLMeg converges on that task; all VLM calls, VLM judgment accuracy, and task success rates are accumulated or evaluated up to this convergence point.
Hyper-parameterSetting
Video fondation ModelWan2.1-I2V-14B
Number of finetuning videos3-5
Video frames250250
Frame shape256×256×3256\times 256\times 3
Frame stack11
Action repeat22
Table 3: The hyper-parameters of VLMeg.
Figure 12: Examples in the Near dataset used in the reference-frame motivation study on faucet-open . Top: successful examples close to the expert success frame. Bottom: failing examples whose robot-arm states are highly similar to the expert success frame.
Figure 13: Examples in the Deviated dataset. Top: successful examples. Bottom: failing examples. In both subsets, the robot-arm states deviate from the expert success frame, and no occlusion is present.
More in this topic

Reward Design for RL

Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.

See all 261 papers

Explore similar work

CardsList
Feb 24, 2026

How Foundational Skills Influence VLM-based Embodied Agents:A Native Perspective

Recent advances in vision-language models (VLMs) have shown promise for human-level embodied intelligence. However, existing benchmarks for VLM-driven embodied agents often rely on high-level commands or discretized action spaces, which are non-native settings that differ markedly from real-world control. In addition, current benchmarks focus primarily on high-level tasks and lack joint evaluation and analysis at both low and high levels. To address these limitations, we present NativeEmbodied, a challenging benchmark for VLM-driven embodied agents that uses a unified, native low-level action space. Built on diverse simulated scenes, NativeEmbodied includes three representative high-level tasks in complex scenarios to evaluate overall performance. For more detailed analysis, we further decouple the skills required by complex tasks and construct four types of low-level tasks, each targeting a fundamental embodied skill. This joint evaluation across task and skill granularities enables fine-grained assessment of embodied agents. Experiments with state-of-the-art VLMs reveal clear deficiencies in several fundamental embodied skills, and further analysis shows that these bottlenecks significantly limit performance on high-level tasks. NativeEmbodied highlights key challenges for current VLM-driven embodied agents and provides insights to guide future research.

Bo Peng, Pi Bu, Keyu Pan +7

University of Science and Technology of China · Alibaba Group · Independent Researcher +1

Embodied Agent BenchmarksEmbodied AIVision-Language AgentsVLM EvaluationRobotic Control
AskPDF
Jul 1, 2026

VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learning

Designing effective reward functions remains a major challenge in reinforcement learning (RL), particularly in open-ended environments where task goals are abstract and difficult to quantify. In this work, we present VLM-AR3L, a framework that leverages Vision-Language Models (VLMs) to provide both absolute and relative rewards for RL. VLM-AR3L interprets an agent's visual observations in the context of a natural language task goal, and learns both absolute and relative rewards from VLM-generated preference labels. The absolute reward model predicts scalar evaluations for individual states, while the relative reward model compares consecutive observations to infer progress or regression toward the task goal. Their integration combines the stability of state-based evaluation with the robustness of comparative supervision. We evaluate VLM-AR3L across benchmarks spanning classic control, manipulation, and open-world embodied tasks, with a particular focus on Minecraft given its visual complexity and long-horizon decision-making requirements. Experimental results show that VLM-AR3L consistently outperforms prior VLM-based reward learning methods.

Kuan-Chen Chen, Winston Chen, Wei-Fang Sun +1

Department of Computer Science, National Tsing Hua University · NVIDIA AI Technology Center (NVAITC)

Multimodal Reward ModelingReward Design for RLPreference-Based RL
AskPDF
Oct 3, 2026

ROOT: Discovering Rewards for User-Specified Embodied Behaviors

Reinforcement learning for embodied control remains constrained by the difficulty of reward specification. Although recent large language model (LLM)-based methods can synthesize reward functions from natural-language descriptions, they often fail to capture subtle behavioral properties that humans care about, such as natural gait, posture, and movement style. This limitation arises because many desired behaviors are easier to recognize visually than to encode in a reward function. We introduce Reward Optimization via Observable Trees (ROOT), a framework for discovering reward functions that align learned policies with user-specified embodied behaviors. Rather than relying solely on scalar training statistics, ROOT casts reward design as an observation-guided search over a persistent experiment tree that stores reward programs, trained policies, and rollout observations, together with behavioral insights distilled by a video-language model, to diagnose behavioral failures and guide subsequent reward refinements. We evaluate ROOT on seven tasks across four embodiments: simulated Hopper, HalfCheetah, Ant, Unitree Go2, and as well as the real-world Unitree Go2. ROOT produces behaviors that better align with user intent than those generated by existing LLM-based reward-generation methods, achieving up to 86.8% locomotion-completeness accuracy and improving Vid-LLM behavioral alignment from 3.56/5 to 4.14/5, a 16.5% improvement over baselines. Human evaluations further support these results, with ROOT preferred in 51-63% of pairwise comparisons.

Eren Sadikoglu, Aditya Taparia, Xinyuan Liu +1

Robotics and Autonomous Systems, Arizona State University, Tempe, Arizona, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA

Reward Design for RLRobotic RLLLM-Guided RLRL for Legged Locomotion
AskPDF