Vision-Language Agents

Follow this topic Ask about this topic

Vision-language agents study how models can use images, video, and language to act in settings such as websites, mobile interfaces, games, robots, and social environments. Common approaches combine visual grounding with planning and action, often adding tools, reusable skills, persistent memory, multi-agent coordination, or reinforcement learning. Benchmarks test both isolated abilities and complete interactive tasks, including GUI navigation, embodied control, spatial reasoning, and real-time response. A central challenge is turning strong local perception and reasoning into reliable long-horizon behavior: agents still struggle with changing scenes, spatial state tracking, memory, and completing all parts of a task.

Latest papers

45 papers4 papers in the last four weeks, up 33% on the four weeks before. 0.0% of all new papers.

StreamVibe: Proactive Video Agents for Evolving User Intent
Oct 26, 2026Chengzhi Liu, Yuxiang Lai, Yuzhe Yang +13Vision-Language Agents Multimodal Intent Recognition
Google · University of California, Santa Barbara · Emory University +4
Learning How to Remember: Memory Modality Selection for GUI Agents
Oct 13, 2026Taeyun Roh, Suhyeong Park, Gijin Kim +2Memory-Augmented Agents GUI Agents
Korea University · Myongji University · AIGEN Sciences
ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills
Oct 8, 2026Hongxing Li, Dingming Li, Yixin Li +6Agent Skill Learning Vision-Language Agents
Zhejiang University
MeshQuery: Agentic Seam Planning for UV Parametrization
Oct 1, 2026Marco Schouten, Arthur Roullier, Elie Michel +3Vision-Language Agents
Adobe Research, France
Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena
Oct 1, 2026Haojian Huang, Pukun Zhao, Zexi Li +7VLM Evaluation Visual Reasoning
HKUST (GZ) · Knowin AI · The Chinese University of Hong Kong
Show-Harness: Just a VLM Agent Can Play Robots
Sep 9, 2026Yanzhe Chen, Zechen Bai, Zhijun Cao +7Cross-Embodiment Robot Learning Language-Conditioned Robot Control
Show Lab, National University of Singapore
Cross-Domain Tracker Adaptation Without Target-Domain Labels via Vision-Language Agents
Sep 4, 2026Daniel Davila, Ravikumar Balakrishnan, Mike CochranMulti-Object Tracking Unsupervised Domain Adaptation
Cisco Systems, Inc. San Jose, CA
Lies We Can See: Joint Verbal and Non-Verbal Deception by VLM Agents in Embodied Social Interactions
Aug 31, 2026Jaewoo Ahn, Junseo Kim, Hyunseo Kim +4Social Deduction Games AI Agent Safety Benchmarks
Seoul National University · Inha University · KAIST +1
Minimal Local Simulation Foundations for LLM- and VLM-Driven Agents in 2D and 3D Environments
Aug 24, 2026Ryuki HyodoAgent-Based Modeling Large Language Model-Based Social Simulation
SpaceData Inc., 1-17-1 Toranomon, Minato-ku, Tokyo 105-6490, Japan · Graduate School of Artificial Intelligence and Science, Rikkyo University, 3-34-1 Nishi-Ikebukuro, Toshima-ku, Tokyo 171-8501, Japan · Earth-Life Science Institute, Institute of Science Tokyo, 2-12-1 Ookayama, Meguro-ku, Tokyo, 152-8550, Japan +1
MOSS-VL Technical Report
Aug 15, 2026Pengyu Wang, Chenkun Tan, Shaojun Zhou +29Vision-Language Models Video-Language Models
OpenMOSS Team
SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation
Aug 11, 2026Zhou Liu, Ligang Huang, Zeli Su +5GUI Agents LLM Agent Skill Retrieval
Peking University · ByteDance · Shanghai Jiao Tong University
ReflectVLN: Training Vision-Language Navigation Agents with Reflective Reasoning
Jul 14, 2026Jiahang Wang, Yirong Yang, Yanqing Zhu +4Hierarchical Planning Vision-Language Navigation
Amap, Alibaba Group, China · Beihang University
Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing
Jul 9, 2026Feng Wang, Canmiao Fu, Zhipeng Huang +3Multimodal Memory Episodic Memory
Peking University · WeChat Vision, Tencent Inc.
Dynamo: Dynamic Skill-Tool Evolution for Vision-Language Agents
Jun 29, 2026Yutao Sun, Yanting Miao, Hao-Xuan Ma +8Tool Use in VLMs VLM Adaptation
1Qwen Large Model Application Team, Alibaba · 2Zhejiang University · University of Waterloo +3
The Latent Bridge: A Continuous Slow-Fast Channel for Real-Time Game Agents
Jun 23, 2026Bojie Li, Noah ShiGame-Playing Agents VLM Reasoning
Pine AI · University of Washington
LandslideAgent with Multimodal LandslideBench: A Domain-Rule-Augmented Agent for Autonomous Landslide Identification and Analysis
Jun 17, 2026Chengfu Liu, Dongyang Hou, Junwu Xiang +5Semantic Segmentation Vision-Language Agents
School of Geosciences and Info-Physics, Central South University, 932 South Lushan Road, Changsha, 410083, Hunan, China.
GUITrans2Act: Understanding User Operational Behaviors from Mobile GUI Interactions with Vision-Language Models
Jun 11, 2026Yudong Zhang, Lei Hu, Daoyang Liu +4Mobile GUI Automation Vision-Language Agents
Honor Device Co., Ltd · The Chinese University of Hong Kong, Hong Kong, China
JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence
Jun 10, 2026Dingyu Yao, Junhao Zhou, Chenxu Yang +17Human-AI Interaction Vision-Language Agents
Guide Me Out: A Framework to Benchmark VLM Operators Communication in Crisis Scenarios
Jun 8, 2026Giacomo Gonella, Stefano Menini, Marco GueriniVLM Evaluation Disaster Response
Fondazione Bruno Kessler, Italy · University of Trento, Italy
MIRAGE: Mobile Agents with Implicit Reasoning and Generative World Models
Jun 3, 2026Zhichao Yang, Yuanze Hu, Haojie Hao +7Mobile GUI Automation Latent Visual Reasoning
Beihang University · Northwestern Polytechnical University · Institute of Software, Chinese Academy of Sciences +2
Multi-Turn Multi-Agent Dialogue for Collaborative Reconstruction Improves VLM Performance on Spatial Reasoning, But Only Barely
May 29, 2026Chalamalasetti Kranti, Sherzod Hakimov, David SchlangenVisual Spatial Reasoning Multi-Agent Collaboration
Computational Linguistics, Department of Linguistics University of Potsdam, Germany · German Research Center for Artificial Intelligence (DFKI), Berlin, Germany
SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes
May 29, 2026Tianhui Liu, Jie Feng, Zhiheng Zheng +6Vision-Language Agents Visual Spatial Reasoning
1The Hong Kong University of Science and Technology (Guangzhou) · 2Zhongguancun Academy · 3Tsinghua University +1
DiTTo: Scalable Order-aware All-in-One Image Restoration Agent
May 29, 2026Seungho Choi, Jihyong OhAgent-Based Image Restoration All-in-One Image Restoration
CMLab, Chung-Ang University
Completion at the Boundary (CaB): Deployable Switching with Completion-Aware Control under Limited Calibration
May 29, 2026Yusuke Sano, Takeshi ItogaVision-Language-Action Models Embodied AI
Intelligent Systems Laboratory, SECOM Co., Ltd.
DepthAgent: Towards Better Universal Depth Estimation via Sample-wise Expert Selection
May 22, 2026Jie Zhu, Girish Chandar Ganesan, Xiaoming LiuMetric Depth Estimation Monocular Depth Estimation
Michigan State University · University of North Carolina at Chapel Hill
GROW: Aligning GRPO with State-Action Modeling for Open-World VLM Agents
May 18, 2026Xiongbin Wu, Zhihao Luo, Shanzhe Lei +7Reinforcement Learning Group Relative Policy Optimization
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · East China Normal University +2
ViSRA: A Video-based Spatial Reasoning Agent for Multi-modal Large Language Models
May 11, 2026Tingshu Mou, Jiabo He, Renying Wang +5Visual Spatial Reasoning 3D Spatial Reasoning
Fudan University · BCAI · Bosch Center for Artificial Intelligence (BCAI) +1
Mirror, Mirror on the Wall: Can VLM Agents Tell Who They Are at All?
May 9, 2026Filippo Ziliotto, Ciro Beneduce, Bruno Lepri +3AI Agent Evaluation Vision-Language Grounding
University of Padova · FondazioneLucianoBrunoSerafiniKessler · Fondazione Bruno Kessler
PRISM: Perception Reasoning Interleaved for Sequential Decision Making
May 6, 2026Mohamed Salim Aissi, Clemence Grislain, Clement Romac +4Active Perception VLM Reasoning
Sorbonne Universite, CNRS, ISIR, F-75005 Paris, France · Hugging Face · Inria (Flowers), University of Bordeaux, France +2
What You Think is What You See: Driving Exploration in VLM Agents via Visual-Linguistic Curiosity
May 5, 2026Haoxi Li, Qinglin Hou, Jianfei Ma +6Intrinsic Motivation RL Exploration
Department of Computer Science and Engineering, Hong Kong University of Science and Technology (HKUST), Hong Kong, China. · Department of Computer Science, University of Southern California, Los Angeles, USA · School of Computing, National University of Singapore, Singapore. +1

Oct 26, 2026J/K move · Enter open · S save

StreamVibe: Proactive Video Agents for Evolving User Intent

Chengzhi Liu, Yuxiang Lai, Yuzhe Yang, Yanqing Liu+12

Google · University of California, Santa Barbara · Emory University+4

Vision-Language Agents Multimodal Intent Recognition Streaming Video Understanding Memory-Augmented Video Understanding

Open paper AskPDF arXiv

Vision-language models are shifting from offline passive understanding toward online proactive interaction. However, prevailing methods largely rely on explicit requests while miss the implicit user intent (e.g., "where should I click next?" while a user navigate a website), meanwhile fail to update the state simultaneously as user requests might update continuously. These limitations make them less suited to intent-aware interaction in practical. To address these challenges, we introduce StreamVibe-Bench, a new benchmark serving as testbed for user intent interaction, with 4.1K evaluation samples and 85K training samples spanning explicit, implicit, and dynamic intents. Building on this benchmark, we develop StreamVibe-VL, a streaming proactive model that infers implicit user intents from continuous observations and dynamically tailors its interactions as these intents evolve. We further introduce StreamVibe-Harness, which maintains user intents as persistent states with a fleixble lifecycles via a hierarchical memory to track the latest user intent and environmental state. Our experiments reveal three key findings: (i) StreamVibe-VL preserves strong general video understanding while improving response timing and answer quality by 10.7% over other baselines; (ii) existing proactive models remain limited in understanding user intent and deciding when to respond, especially under implicit and dynamic intents; and (iii) StreamVibe-Harness further improves response consistency and adaptation over sustained interaction, resulting in a 12.3% gain in proactive interaction.