cs.CVNov 18, 2026

ViStep-RL: Stepwise Reinforcement Learning for Long-Horizon Visual Agents

Authors: Junyang Chen, Yao Qin, Xiao Li, Ningchao Wang, Yiguo Qiao, Wenbin Li, Xin Geng

Organizations: School of Computer Science and Engineering, Southeast University, Nanjing 211189, China · Key Laboratory of Computer Network and Information Integration (Ministry of Education), Southeast University, Nanjing 211189, China · University of Science and Technology of China, Hefei, China · University of Bath, Bath, United Kingdom

Abstract

Long-horizon visual agents must do more than inspect images: they must turn observations into evidence that remains usable as a task unfolds. Existing active-perception methods teach agents where and how to look, while a final-answer reward gives little structure for deciding which observations satisfy dependent subgoals or should be retained for later use. We present ViStep-RL (Visual Stepwise Reinforcement Learning), which models execution as a dependency-linked graph of evidence-bearing subgoals. A verifier tracks which subgoals are supported by valid evidence, and changes in graph potential provide a net-progress learning signal; an active memory protocol preserves evidence across turns and links retrieval to downstream use. On ViInteract, ViStep-RL reaches 60.0% strict completion with Qwen2.5-VL-7B, compared with 10.0% for the base model and 46.5% after supervised fine-tuning. Controlled studies show that incremental task-state rewards outperform cumulative rewards by 3.2 points, while active evidence writing improves strict completion by 7.75 points over automatic writing. Under a common interaction framework, the task-trained ViStep-RL policy achieves 59.5% answer accuracy, 24.0 points above the strongest of six general-purpose reference models. Held-out episodes confirm the gain at both Qwen scales, and gains extend to three external visual reasoning benchmarks.

Figures & tables

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. StructRL: Online Structured Reinforcement Learning for Long-Horizon Vision-Language-Action Tasks

    Sep 28, 2026Ziyi Yin, Sangmin Woo, Kang Zhou +4Reward ShapingLong-Horizon Robotic Manipulation

  2. RL Teaches VLMs Where to Attend but Not How to Perceive

    Oct 14, 2026Ziqian Zhang, Junjie Liao, Zhining Zhang +1RL for VLMsVisual Evidence Sensitivity in VLMs

  3. Recursive Belief Vision Language Action Models

    Feb 24, 2026Vaidehi Bagaria, Bijo Sebastian, Nirav Kumar PatelVision-Language-Action ModelsLong-Horizon Robotic Manipulation