cs.ROFeb 22, 2026

Seeing Farther and Smarter: Value-Guided Multi-Path Reflection for VLM Policy Optimization

Authors: Yanting Yang, Shenyuan Gao, Qingwen Bu, Li Chen, Dimitris N. Metaxas

Organizations: Rutgers University · The Hong Kong University of Science and Technology · The University of Hong Kong

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Solving complex, long-horizon robotic manipulation tasks requires a deep understanding of physical interactions, reasoning about their long-term consequences, and precise high-level planning. Vision-Language Models (VLMs) offer a general perceive-reason-act framework for this goal. However, previous approaches using reflective planning to guide VLMs in correcting actions encounter significant limitations. These methods rely on inefficient and often inaccurate implicit learning of state-values from noisy foresight predictions, evaluate only a single greedy future, and suffer from substantial inference latency. To address these limitations, we propose a novel test-time computation framework that decouples state evaluation from action generation. This provides a more direct and fine-grained supervisory signal for robust decision-making. Our method explicitly models the advantage of an action plan, quantified by its reduction in distance to the goal, and uses a scalable critic to estimate. To address the stochastic nature of single-trajectory evaluation, we employ beam search to explore multiple future paths and aggregate them during decoding to model their expected long-term returns, leading to more robust action generation. Additionally, we introduce a lightweight, confidence-based trigger that allows for early exit when direct predictions are reliable, invoking reflection only when necessary. Extensive experiments on diverse, unseen multi-stage robotic manipulation tasks demonstrate a 24.6% improvement in success rate over state-of-the-art baselines, while significantly reducing inference time by 56.5%.

Figures & tables

Fig. 1 : Overview of our value-guided multi-path reflection framework . Initially, the VLM policy proposes a candidate action list conditioned on the current and goal images ( \lx@sectionsign\lx@sectionsign III-B ). The action with the highest probability (“pick up purple”) is treated as a preliminary plan and assessed by a confidence-based trigger ( \lx@sectionsign\lx@sectionsign III-D ). If reflection is needed, a diffusion dynamics model imagines multiple future trajectories via beam search, guided by the candidate action list. A critic evaluates each of these imagined trajectories by estimating their distance reduction to the goal state, and this evaluation is incorporated as feedback into multiple distinct contexts. These inputs are then aggregated during the decoding stage, allowing the agent to reflect on the potential futures, revise its initial plan, and generate a more proper action (“pick up yellow”) ( \lx@sectionsign\lx@sectionsign III-C ).
Fig. 2 : (a) Overview of the multi-path reflection mechanism during decoding . Output distributions from multiple inputs based on distinct imagined futures are dynamically combined using complementary or contrastive decoding based on their Jensen-Shannon Divergence to enhance or correct output. (b) A substantial 78.9% of optimal actions are produced in the initial proposal stage, while the reflection stage is also essential for more challenging decisions, leading to an 18% increase in overall success.
Fig. 3 : Task examples.
Fig. 4 : Comparison of success rates on 100 unseen long-horizon manipulation tasks. Our value-guided reflective planning framework, in both its simulation-based (sim) and diffusion-based variants, significantly outperforms all baselines, including traditional methods like MCTS and Behavioral Cloning (BC) and the prior state-of-the-art, ReflectVLM. For a fair comparison, both our method and ReflectVLM are evaluated under only a single round of post-training. The “Ours w/ oracle value” results demonstrate the upper-bound performance of our framework with perfect value estimation.
Fig. 5 : (a) Inference time cost comparison . Our method significantly outperforms ReflectVLM [ 19 ] , reducing inference time by over 45% via the confidence-based early-exit strategy. (b) Ablation study on multi-path aggregation strategies . Our aggregation mechanism during decoding outperforms both the single-path baseline and traditional post-hoc selection methods, such as Best-of-N and Majority Voting. (c) Qualitative analysis of the advantage distribution for revised actions. Our method’s reflection is highly precise, primarily targeting low-advantage actions (peaking near 0). In contrast, the baseline’s reflection is indiscriminate and less likely to trigger necessary revisions.
More in this topic

VLMs for Robotics

Vision-language models for robotics connect visual and linguistic understanding to grounded navigation, manipulation, and robot control.

See all 657 papers

Explore similar work

CardsList
Mar 5, 2026

Critic in the Loop: A Tri-System VLA Framework for Robust Long-Horizon Manipulation

Balancing high-level semantic reasoning with low-level reactive control remains a core challenge in visual robotic manipulation. While Vision-Language Models (VLMs) excel at cognitive planning, their inference latency precludes real-time execution. Conversely, fast Vision-Language-Action (VLA) models often lack the semantic depth required for complex, long-horizon tasks. To bridge this gap, we introduce Critic in the Loop, an adaptive hierarchical framework driven by dynamic VLM-Expert scheduling. At its core is a bionic Tri-System architecture comprising a VLM brain for global reasoning, a VLA cerebellum for reactive execution, and a lightweight visual Critic. By continuously monitoring the workspace, the Critic dynamically routes control authority. It sustains rapid closed-loop execution via the VLA for routine subtasks, and adaptively triggers the VLM for replanning upon detecting execution anomalies such as task stagnation or failures. Furthermore, our architecture seamlessly integrates human-inspired rules to intuitively break infinite retry loops. This visually-grounded scheduling minimizes expensive VLM queries, while substantially enhancing system robustness and autonomy in out-of-distribution (OOD) scenarios. Comprehensive experiments on challenging, long-horizon manipulation benchmarks reveal that our approach achieves state-of-the-art performance.

Pengfei Yi, Yingjie Ma, Wenjiang Xu +4

Institute of Automation, Chinese Academy of Sciences · the School of Artificial Intelligence, University of Chinese Academy of Sciences

Long-Horizon Robotic ManipulationVLMs for RoboticsRobustness of VLA ModelsAdaptive Model RoutingEfficient VLA Model Inference
AskPDF
Apr 15, 2026

Goal2Skill: Long-Horizon Manipulation with Adaptive Planning and Reflection

Recent vision-language-action (VLA) systems have demonstrated strong capabilities in embodied manipulation. However, most existing VLA policies rely on limited observation windows and end-to-end action prediction, which makes them brittle in long-horizon, memory-dependent tasks with partial observability, occlusions, and multi-stage dependencies. Such tasks require not only precise visuomotor control, but also persistent memory, adaptive task decomposition, and explicit recovery from execution failures. To address these limitations, we propose a dual-system framework for long-horizon embodied manipulation. Our framework explicitly separates high-level semantic reasoning from low-level motor execution. A high-level planner, implemented as a VLM-based agentic module, maintains structured task memory and performs goal decomposition, outcome verification, and error-driven correction. A low-level executor, instantiated as a VLA-based visuomotor controller, carries out each sub-task through diffusion-based action generation conditioned on geometry-preserving filtered observations. Together, the two systems form a closed loop between planning and execution, enabling memory-aware reasoning, adaptive replanning, and robust online recovery. Experiments on representative RMBench tasks show that the proposed framework substantially outperforms representative baselines, achieving a 32.4% average success rate compared with 9.8% for the strongest baseline. Ablation studies further confirm the importance of structured memory and closed-loop recovery for long-horizon manipulation.

Zhen Liu, Xinyu Ning, Zhe Hu +8

Beijing University of Posts and Telecommunications · InspireOmni AI · Tsinghua University

Long-Horizon Robotic ManipulationRobot Task PlanningVLMs for RoboticsVisuomotor Policy Learning
AskPDF
Jun 26, 2026

S2^2-VLA: State-Space Guided Vision-Language-Action Models for Long-Horizon Manipulation

Vision-Language-Action (VLA) models have demonstrated strong capabilities in robotic manipulation, but their performance degrades significantly in long-horizon tasks due to cumulative error propagation. This limitation largely arises from static feature fusion mechanisms that rely on fixed weights to combine visual, language, and action representations, preventing the model from adapting to different phases of task execution. To address this limitation, we propose S2^2-VLA, a framework that introduces a State-Space Guided Adaptive Attention (SSGAA) mechanism. SSGAA maintains a belief state that tracks task progression and generates dynamic gating weights to adaptively fuse information from three complementary sources visual features for spatial perception, task intents for high-level task planning, and temporal action sequences for execution consistency. This adaptive fusion allows the model to shift its focus throughout task execution, aligning with the evolving requirements of different task stages. Despite its compact 2B parameter size, S2^2-VLA consistently outperforms larger 7B-scale models and achieves state-of-the-art performance on long-horizon manipulation benchmarks, including LIBERO and SimplerEnv. highlighting the importance of adaptive feature fusion for long-horizon robotic manipulation.

Zhipeng Xie, Zongyi Han, Xiangyi Wei +3

School of Computer Science and Technology, East China Normal University, Shanghai 200062, China · 2State Key Laboratory of Submarine Geoscience, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai 200240, China

Long-Horizon Robotic ManipulationVision-Language-Action ModelsMultimodal Fusion
AskPDF