RL for VLMs

RL: Reinforcement Learning · VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

36 papers in the last four weeks, up 227% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 287

All topics

Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigation
May 26, 2026Heng Qu, Yike Liu, Renren Jin +4VLM Evaluation Vision-Language Navigation
1Wuhan University · 2Xiaomi
DynFrame: Adaptive Reasoning-Driven Multimodal Framework with Dynamic Frame Augmentation for Complex Video Understanding
May 26, 2026Peng Zhang, Guanghao Zhang, Wanggui He +10Multimodal Large Language Models Video Reasoning
Zhejiang University · Alibaba Group
InterSketch: An Interleaved Reasoning Model with Self-correcting Visual Sketch and Stepwise Reward
May 26, 2026Zhiwei Ning, Wenwen Tong, Xiangli Kong +12Multimodal Reasoning VLM Reasoning
Shanghai Jiao Tong University · SenseTime Research · Shandong Normal University
ProSR: Process-Shaped Spatial Reasoning for Reliable Chain-of-Thought in VLMs
May 25, 2026Jiangyang Li, Cong Wan, Changjie Wu +8Visual Spatial Reasoning VLM Reasoning
1Xi’an Jiaotong University · 2Amap, Alibaba Group · 4Shenzhen University +1
DUEL: Adversarial Self-Play for Multimodal Reasoning
May 24, 2026Lin Qiu, Hanqing Zeng, Yao Liu +3VLM Robustness VLM Reasoning
Meta AI
B-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation
May 22, 2026Mario Markov, Stefan Maria Ailuro, Mohammad Mahdi +2Image Segmentation Reinforcement Learning
INSAIT, Sofia University "St. Kliment Ohridski" · INSAIT, Sofia University “St. Kliment Ohridski”
Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention
May 21, 2026Changyuan Tian, Zhicong Lu, Huaxing Liu +7Multimodal Large Language Models Multimodal Reasoning
AMAP, Alibaba Group · University of Chinese Academy of Sciences · Tsinghua University +1
Learning Spatiotemporal Sensitivity in Video LLMs via Counterfactual Reinforcement Learning
May 21, 2026Dazhao Du, Jian Liu, Jialong Qin +7Counterfactual Evaluation Video-Language Models
Hong Kong University of Science and Technology · Tencent
EvoVid: Temporal-Centric Self-Evolution for Video Large Language Models
May 21, 2026Shiqi Huang, Ziyue Wang, Zhongrong Zuo +3Vision-Language Models Video Reasoning
School of Electrical and Electronic Engineering, Nanyang Technological University · ByteDance
Look-Closer-Then-Diagnose: Confidence-Aware Ultrasound VQA via Active Zooming
May 20, 2026Yue Zhou, Erxuan Wu, Yikang Sun +5Visual Question Answering Uncertainty Estimation for VLA Models
Computer Aided Medical Procedures (CAMP), TU Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Zhongshan Hospital, Fudan University, China +1
IndusAgent: Reinforcing Open-Vocabulary Industrial Anomaly Detection with Agentic Tools
May 20, 2026Rongbin Tan, Fangfang Lin, Zhenlong Yuan +10Industrial Anomaly Detection Tool-Using Vision-Language Agents
State Key Lab of Processors, Institute of Computing Technology, CAS
PAPO-VLA: Planning-Aware Policy Optimization for Vision-Language-Action Models
May 19, 2026Peizheng Guo, Jingyao Wang, Changwen Zheng +1Robot Policy Learning Credit Assignment in RL
Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences
ClaimDiff-RL: Fine-Grained Caption Reinforcement Learning through Visual Claim Comparison
May 19, 2026Tianle Li, Xuyang Shen, Yan Ma +7VLM Evaluation Reward Modeling
1The Chinese University of Hong Kong · 2MiniMax
Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysis
May 19, 2026Tianwei Lin, Zhongwei Qiu, Jie Cao +7Radiology Report Generation Radiology VLMs
Zhejiang University · DAMO Academy, Alibaba Group · Hupan Lab +1
Leveraging Latent Visual Reasoning in Silence
May 18, 2026Dongyao Zhu, Zhen Wang, Xi Xiao +7Visual Spatial Reasoning Latent Visual Reasoning
North Carolina State University · UC, San Diego · University of Alabama at Birmingham +4
Self-Evolving Spatial Reasoning in Vision Language Models via Geometric Logic Consistency
May 18, 2026Junming Liu, Yuqi Li, Yifei Sun +6VLM Robustness Vision-Language Models
Shanghai Artificial Intelligence Laboratory · The City University of New York · The Chinese University of Hong Kong +3
IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learning
May 18, 2026Chenghao Li, Fusheng Hao, Xikai Zhang +5Multimodal Grounding Object Hallucination in VLMs
Hangzhou International Innovation Institute, Beihang University · Shenzhen Institute of Advanced Integration Technology, Shenzhen · Kuaishou Technology +1
AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agents
May 18, 2026Pan Wang, Yihao Hu, Xiujin Liu +3Memory-Augmented VLMs Reward Shaping
Ant Group · University of Science and Technology of China · Westlake University +2
An Efficient Streaming Video Understanding Framework with Agentic Control
May 18, 2026Jinming Liu, Jianguo Huang, Zhaoyang Jia +7Streaming Video Understanding Adaptive Model Routing
1Shanghai Jiao Tong University · 2Eastern Institute of Technology, Ningbo, China · 3Microsoft Research Asia
GROW: Aligning GRPO with State-Action Modeling for Open-World VLM Agents
May 18, 2026Xiongbin Wu, Zhihao Luo, Shanzhe Lei +7Reinforcement Learning Group Relative Policy Optimization
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · East China Normal University +2
Ranking-Aware Calibration for Reliable Multimodal Reinforcement Learning
May 16, 2026Peng Cui, Boyao Yang, Jun ZhuMultimodal Robustness VLM Reasoning
Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, Tsinghua-Bosch Joint ML Center, THBI Lab, Tsinghua University, Beijing 100084, China · Dept. of Automation, Tsinghua University, Beijing 100084, China
OmniVL-Guard Pro: A Tool-Augmented Agent for Omnibus Vision-Language Forensics
May 16, 2026Jinjie Shen, Zheng Huang, Yuchen Zhang +7Tool Use in VLMs Image Forgery Detection
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China · Wuhan University, Wuhan, China · Xi’an Jiaotong University +2
Learn Where Outcomes Diverge: Efficient VLA RL via Probabilistic Chunk Masking
May 15, 2026Vaidehi Bagaria, Nikshep Grampurohit, Pulkit VermaReinforcement Learning Neural Network Optimization
Indian Institute of Technology Madras
WorldVLN: Autoregressive World Action Model for Aerial Vision-Language Navigation
May 15, 2026Baining Zhao, Jiacheng Xu, Weicheng Feng +13World Model Learning Vision-Language Navigation
Tsinghua University · Shandong University · Manifold AI +2
From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Grounding
May 15, 2026Yuyuan Liu, Yiping Ji, Anjie Le +6Vision-Language Grounding Visual Grounding
Department of Engineering Science, University of Oxford · Australian Institute for Machine Learning, Adelaide University · Technical University of Munich +2
REC-RL: Referring expression counting via Gaussian and range-based reward optimization
May 15, 2026Hui Liu, Yunlai Teng, Kunlong Bai +4VLM Reasoning RL for VLMs
JIUTIAN Research, Beijing, China
ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both
May 14, 2026Ziyu Guo, Rain Liu, Xinyan Chen +1Visual Reasoning Large Vision-Language Models
Meta AI · The Chinese University of Hong Kong
EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Grounding
May 14, 2026Yuejiao Su, Xinshen Zhang, Zhen Ye +3Egocentric Video Understanding Vision-Language Grounding
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR. · Division of Emerging Interdisciplinary Areas (EMIA), The Hong Kong University of Science and Technology, Hong Kong SAR.
Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning
May 13, 2026Haozhe Wang, Qixin Xu, Changpeng Wang +4Multimodal Reasoning VLM Reasoning
1Hong Kong University of Science and Technology (HKUST) · 2Tsinghua University · 3Zhejiang University +2
CAVE: A Structured Credit Assignment Approach for Fragmented Visual Evidence Reasoning
May 13, 2026Tengda Guo, Jie Leng, Hanlei Li +6Process Reward Models Visual Reasoning
Tsinghua University · Peking University · Zhejiang University of Technology

May 26, 2026, cs.AIJ/K move · Enter open · S save

Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigation

Heng Qu, Yike Liu, Renren Jin, Wenzong Zhang+3

1Wuhan University · 2Xiaomi

VLM Evaluation Vision-Language Navigation VLM Reasoning RL for VLMs

Open paper AskPDF arXiv

Vision-Language Models (VLMs) have shown rapid progress in mobile GUI navigation. This paper presents a systematic study of data scaling, benchmarking, and reasoning for VLM-based agents in this domain. To facilitate rigorous evaluation, we introduce HyperTrack, a large-scale dataset with over 16000 real-world tasks across more than 650 Chinese mobile applications, along with GUIEvalKit, an open-source toolkit for unified benchmarking of VLMs on offline GUI navigation tasks. Using HyperTrack, we analyze the effects of training data scale on both supervised and reinforcement-based finetuning. Our results show that reinforcement-based finetuning consistently outperforms supervised finetuning, particularly in out-of-domain settings, highlighting the synergy between data scaling and reinforcement learning. Leveraging GUIEvalKit, we further benchmark state-of-the-art (SOTA) VLMs and analyze how interaction history and reasoning capabilities influence task completion. Together, HyperTrack and GUIEvalKit provide a comprehensive platform for developing and evaluating VLM agents in mobile GUI navigation tasks.

Momentum

Related topics

Latest papers 287