RL for VLMs

RL: Reinforcement Learning · VLM: Vision-Language Model

This field studies how reinforcement learning can improve models that interpret images or video alongside language, and sometimes take actions in visual environments. Common approaches include group-relative policy optimization, PPO, and reinforcement learning from verifiable rewards, with feedback based on answer correctness, visual evidence, intermediate reasoning, preferences, or tool use. Applications range from image and video question answering, spatial reasoning, and medical analysis to navigation and robotic manipulation. A central challenge is ensuring that better scores reflect genuine use of visual evidence rather than language shortcuts, while designing reliable rewards and efficient training that generalize beyond the training setting.

Latest papers

476 papers68 papers in the last four weeks, up 94% on the four weeks before. 0.4% of all new papers.

Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs
Mar 21, 2026Huan Zheng, Yucheng Zhou, Tianyi Yan +6Endoscopy Medical Diagnosis
SKL-IOTSC, CIS, University of Macau · 2Shanghai Jiao Tong University · 3COWARobot Co. Ltd. +1
Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models
Mar 21, 2026Zhilong Zhang, Haoxiang Ren, Yihao Sun +6RL for VLMs Vision-Language-Action Models
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · Mila - Quebec AI Institute +1
Chain-of-Adaptation: Surgical Vision-Language Adaptation with Reinforcement Learning
Mar 20, 2026Jiajie Li, Chenhui Xu, Meihuan Liu +1VLM Adaptation Medical VLMs
University at Buffalo Buffalo, NY, USA
Scaling Sim-to-Real VLA Reinforcement Learning with Generative 3D Worlds
Mar 19, 2026Andrew Choi, Xinjie Wang, Zhizhong Su +1Sim-to-Real Transfer Vision-Language-Action Models
Horizon Robotics
AcceRL: A Distributed Asynchronous Reinforcement Learning and World Model Framework for Vision-Language-Action Models
Mar 19, 2026Chengxuan Lu, Shukuan Wang, Yanjie Li +10World Models for VLA Models Asynchronous RL
IROOTECH TECHNOLOGY Wolf 1069 b Lab, Sany Group
Insight-V++: Towards Advanced Long-Chain Visual Reasoning with Multimodal Large Language Models
Mar 18, 2026Yuhao Dong, Zuyan Liu, Shulin Tian +2VLM Reasoning Multimodal Large Language Models
S-Lab, Nanyang Technological University, Tencent Hunyuan, and Tsinghua University
SaFeR-Steer: Evolving Multi-Turn MLLMs via Synthetic Bootstrapping and Feedback Dynamics
Mar 18, 2026Haolong Hu, Hanyu Li, Tiancheng He +6Language Model Steering Multimodal Large Language Models
Huazhong University of Science and Technology · Beijing University of Posts and Telecommunications · West China Biomedical Big Data Center, Sichuan University +2
Ruyi2.5 Technical Report
Mar 18, 2026Huan Song, Shuyu Tian, Qingfei Zhao +5Edge-Cloud Collaborative Inference Multimodal Foundation Models
Institute of Artificial Intelligence (TeleAI), China Telecom
HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning
Mar 17, 2026Shenzhi Wang, Shixuan Liu, Jing Zhou +8Reasoning for VLMs VLM Reasoning
Qwen Team, Alibaba Inc. · LeapLab, Tsinghua University
DreamPlan: Efficient Reinforcement Fine-Tuning of Vision-Language Planners via Video World Models
Mar 17, 2026Emily Yue-Ting Jia, Weiduo Yuan, Tianheng Shi +3VLMs for Robotics RL for VLMs
USC Physical Superintelligence Lab · Toyota Research Institute
Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Models
Mar 17, 2026Weijie Qiu, Dai Guan, Junxin Wang +6Multimodal Reward Modeling VLM Evaluation
Qwen Large Model Application Team, Alibaba · Institute of Automation, Chinese Academy of Sciences
From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation
Mar 16, 2026Yibin Liu, Yaxing Lyu, Daqi Gao +5Robot Task Progress Estimation Robotics
Shanghai Jiao Tong University · Northeastern University · The University of Hong Kong +2
Gym-V: A Unified Vision Environment System for Agentic Vision Research
Mar 16, 2026Fanqing Meng, Lingxiao Du, Jiawei Gu +9AI Agent Benchmarks RL for VLMs
National University of Singapore. · University of Washington. · The Hong Kong Polytechnic University. +1
Molecular Identifier Visual Prompt and Verifiable Reinforcement Learning for Chemical Reaction Diagram Parsing
Mar 16, 2026Jiahe Song, Chuang Wang, Yinfan Wang +11Visual Prompt Learning Reinforcement Learning with Verifiable Rewards
Shanghai Jiao Tong University · Shanghai AI Laboratory · Beihang University +2
Ego to World: Collaborative Spatial Reasoning in Embodied Systems via Reinforcement Learning
Mar 16, 2026Heng Zhou, Li Kang, Yiran Qin +12Spatial Reasoning VLMs for Robotics
University of Science and Technology of China · Shanghai AI Laboratory · Shanghai Jiao Tong University +4
SmoothVLA: Aligning Vision-Language-Action Models with Physical Constraints via Intrinsic Smoothness Optimization
Mar 14, 2026Jiashun Li, Xiaoyu Shi, Hong Xie +2VLMs for Robotics RL for VLMs
Chongqing University of Posts and Telecommunications · Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences · The First Affiliated Hospital, University of Science and Technology of China
Locatability-Guided Adaptive Reasoning for Image Geo-Localization with Vision-Language Models
Mar 13, 2026Bo Yu, Fengze Yang, Yiming Liu +6VLM Reasoning Cross-View Geo-Localization
The University of Utah · The University of Oklahoma · Worcester Polytechnic Institute +2
Visual-ERM: Reward Modeling for Visual Equivalence
Mar 13, 2026Ziyu Liu, Shengyuan Ding, Xinyu Fang +7Reward Modeling RL for VLMs
Shanghai Jiao Tong University · Shanghai AI Laboratory · Fudan University +1
Linking Perception, Confidence and Accuracy in MLLMs
Mar 12, 2026Yuetian Du, Yucheng Wang, Rongyu Zhang +5Multimodal Large Language Models VLM Calibration
Zhejiang University · University of Michigan · Alibaba Group +1
Multi-Task Reinforcement Learning for Enhanced Multimodal LLM-as-a-Judge
Mar 12, 2026Junjie Wu, Xuan Kan, Zihao He +3Vision-Language Model-as-a-Judge Multimodal Model Evaluation
Meta AI · Hong Kong University of Science and Technology · Emory University
Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning
Mar 12, 2026Jiaheng Hu, Jay Shim, Chen Tang +4Continual Learning for VLA Models VLA Model Adaptation
UT Austin · UCLA · NTU +1
Are Video Reasoning Models Ready to Go Outside?
Mar 11, 2026Yangfan He, Changgyu Boo, Jaehong YoonVLM Robustness Video Reasoning
NTU Singapore · Korea University
Language-Guided Token Compression with Reinforcement Learning in Large Vision-Language Models
Mar 11, 2026Sihan Cao, Jianwei Zhang, Pengcheng Zheng +7Efficient VLM Inference Visual Token Pruning
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Robotics and Advanced Manufacture, Harbin Institute of Technology, Shenzhen, China · College of Information and Control Engineering, Xi’an University of Architecture and Technology, Xi’an, China
Let's Reward Step-by-Step: Step-Aware Contrastive Alignment for Vision-Language Navigation in Continuous Environments
Mar 10, 2026Haoyuan Li, Rui Liu, Hehe Fan +1Vision-Language Navigation RL for VLMs
College of Computer Science and Technology, Zhejiang University, Hangzhou, China
GeoSolver: Scaling Test-Time Reasoning in Remote Sensing with Fine-Grained Process Supervision
Mar 10, 2026Lang Sun, Ronghao Fu, Zhuoran Duan +3VLM Reasoning Process Reward Models
College of Computer Science and Technology, Jilin University, Changchun 130012, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University
OddGridBench: Exposing the Lack of Fine-Grained Visual Discrepancy Sensitivity in Multimodal Large Language Models
Mar 10, 2026Tengjin Weng, Wenhao Jiang, Jingyi Wang +3Fine-Grained Visual Perception Multimodal Large Language Models
College of Computer Science and Software Engineering, Shenzhen University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Tsinghua Shenzhen International Graduate School +2
MM-Zero: Self-Evolving Multi-Model Vision Language Models From Zero Data
Mar 10, 2026Zongxia Li, Hongyang Du, Chengsong Huang +8Vision-Language Models VLM Reasoning
University of Maryland · Brown University · Washington University in St. Louis +4
RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning
Mar 10, 2026Tzu-Heng Huang, Sirajul Salekin, Javier Movellan +2Image Captioning RL for VLMs
Work done during an internship at Apple · University of Wisconsin—Madison · Apple
Chart-RL: Generalized Chart Comprehension via Reinforcement Learning with Verifiable Rewards
Mar 7, 2026Xin Zhang, Xingyu Li, Rongguang Wang +4Chart QA RL for VLMs
Oracle AI
EgoReasoner: Learning Egocentric 4D Reasoning via Task-Adaptive Structured Thinking
Mar 6, 2026Fangrui Zhu, Yunfeng Xi, Jianmo Ni +9Egocentric Video Understanding 4D Scene Understanding
Northeastern University · Google Research · Google DeepMind

Mar 21, 2026J/K move · Enter open · S save

Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs

Huan Zheng, Yucheng Zhou, Tianyi Yan, Dubing Chen+5

SKL-IOTSC, CIS, University of Macau · 2Shanghai Jiao Tong University · 3COWARobot Co. Ltd.+1

Endoscopy Medical Diagnosis Medical VLMs Visual Grounding RL for VLMs

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have demonstrated remarkable potential in medical image analysis. However, their application in gastrointestinal endoscopy is currently hindered by two critical limitations: the misalignment between general model reasoning and standardized clinical cognitive pathways, and the lack of causal association between visual features and diagnostic outcomes. In this paper, we propose a novel Clinical-Cognitive-Aligned (CogAlign) framework to address these challenges. First, we endow the model with rigorous clinical analytical capabilities by constructing the hierarchical clinical cognition dataset and employing Supervised Fine-Tuning (SFT). Unlike conventional approaches, this strategy internalizes the hierarchical diagnostic logic of experts, ranging from anatomical localization and morphological evaluation to microvascular analysis, directly into the model. Second, to eliminate visual bias, we provide a theoretical analysis demonstrating that standard supervised tuning inevitably converges to spurious background correlations. Guided by this insight, we propose a counterfactual-driven reinforcement learning strategy to enforce causal rectification. By generating counterfactual normal samples via lesion masking and optimizing through clinical-cognition-centric rewards, we constrain the model to strictly ground its diagnosis in causal lesion features. Extensive experiments demonstrate that our approach achieves State-of-the-Art (SoTA) performance across multiple benchmarks, significantly enhancing diagnostic accuracy in complex clinical scenarios.