Vision-Language Models

Also known as VLM

Vision-language models study how systems can connect images or videos with language to recognize, describe, retrieve, and reason about visual content. Common approaches include contrastive image-text encoders such as CLIP, generative models that connect vision encoders to language models, and instruction tuning, sometimes combined with retrieval or tools. Researchers evaluate them on image-text retrieval, visual question answering, OCR and document understanding, visual grounding, and video benchmarks. A central challenge is that fluent answers do not guarantee visual grounding: hallucinations, reliance on textual shortcuts, weak compositional and temporal reasoning, and the costs of efficient deployment remain important concerns.

Latest papers

738 papers77 papers in the last four weeks, up 20% on the four weeks before. 0.4% of all new papers.

Vision-Language Memory for Spatial Reasoning
Nov 25, 2025Zuntao Liu, Yi Du, Taimeng Fu +3Memory-Augmented VLMs Vision-Language Models
Spatial AI & Robotics (SAIR) Lab, University at Buffalo, USA · Stanford University, USA
T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs
Nov 20, 2025Shao-Jun Xia, Huixin Zhang, Zhengzhong TuVision-Language Models In-Context Learning
Duke University · Texas A&M University
GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving
Nov 14, 2025Fabian Schmidt, Markus Enzweiler, Abhinav ValadaVision-Language Models VLMs for Autonomous Driving
Esslingen University of Applied Sciences · University of Freiburg
Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts
Nov 6, 2025Ellis Brown, Jihan Yang, Shusheng Yang +2Benchmark Auditing VLM Evaluation
New York University
TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Models
Sep 29, 2025Zhifang Zhang, Qiqi Tao, Jiaqi Lv +3VLM Robustness Adversarial Attacks on VLMs
School of Computer Science and Engineering, Southeast University, China · School of Electrical Engineering and Computer Science, University of Queensland, Australia · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore +2
HERMAN: Hierarchical Representation Matching for CLIP-based Class-Incremental Learning
Sep 26, 2025Zhen-Hao Xie, Yan Wang, Lan Li +3Class-Incremental Learning Vision-Language Models
School of Artificial Intelligence and the State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China
Image Recognition with Vision and Language Embeddings of VLMs
Sep 11, 2025Illia Volkov, Nikita Kisel, Klara Janouskova +1Image Classification VLM Evaluation
Visual Recognition Group, Faculty of Electrical Engineering, Czech Technical University in Prague Prague, Czechia
Prompting with Sign Parameters for Low-resource Sign Language Instruction Generation
Aug 22, 2025Md Tariquzzaman, Md Farhan Ishmam, Saiyma Sittul Muna +2VLM Evaluation Vision-Language Models
Department of Computer Science and Engineering Islamic University of Technology, Bangladesh
TAR: Temporal Anchor-Constrained Reasoning for Video Temporal Grounding
Aug 11, 2025Chaohong Guo, Xun Mo, Yongwei Nie +3Temporal Video Grounding Vision-Language Models
South China University of Technology · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Bytedance Inc.
SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models
Aug 5, 2025Yufei Xue, Yushi Huang, Lunjie Zhu +2Vision-Language Models VLM Quantization
Institute of Artificial Intelligence (TeleAI), China Telecom · The Hong Kong University of Science and Technology
ID-Align: RoPE-Conscious Position Remapping for Dynamic High-Resolution Adaptation in Vision-Language Models
May 27, 2025Bozhou Li, Wentao ZhangRotary Positional Embeddings Vision-Language Models
Peking University Beijing, China
LaViDa: A Large Diffusion Language Model for Multimodal Understanding
May 22, 2025Shufan Li, Konstantinos Kallidromitis, Hritik Bansal +7Multimodal Understanding Vision-Language Models
1UCLA · 2Panasonic AI Research · 4Salesforce Research +1
Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models
May 18, 2025Kai Tang, Jinhao You, Yichen Guo +8Vision-Language Models Hallucination in Language Models
Peking University · University of Pennsylvania · Nanyang Technological University +4
Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMs
Mar 17, 2025Ruichuan An, Kai Zeng, Ming Lu +5Synthetic Data Augmentation Vision-Language Models
1Peking University · 2CUHK MMLab · 3Zhongguancun Academy +1
Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs
Feb 23, 2025Jen-Tse Huang, Dasen Dai, Jen-Yuan Huang +7VLM Evaluation Vision-Language Models
CUHK · PKU · CUHKSZ +3
REVEAL: Robust Evolution of Vision-Language Models for Explainable AI-Video Detection
Feb 20, 2025Yun-Yun Tsai, Qingyuan Liu, Ruijian Zha +4Tool Use in VLMs Vision-Language Models
Columbia University · Rutgers University
Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Benchmarking
Dec 30, 2024Sangyun Chung, Youngjoon Yu, Se Yeon Kim +2VLM Evaluation Vision-Language Models
Integrated Vision Language Lab., School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), 291 Daehak-ro, Yuseong-gu, Daejeon, 34141, Republic of Korea
VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wild
Oct 1, 2024Junlin Fang, Wenyu Chen, Reshmi Ghosh +7VLM Robustness Vision-Language Models
College of Computing and Data Science Nanyang Technological University · School of Physical and Mathematical Sciences Nanyang Technological University · Microsoft Corp. +1

Nov 25, 2025J/K move · Enter open · S save

Vision-Language Memory for Spatial Reasoning

Zuntao Liu, Yi Du, Taimeng Fu, Shaoshu Su+2

Spatial AI & Robotics (SAIR) Lab, University at Buffalo, USA · Stanford University, USA

Memory-Augmented VLMs Vision-Language Models Visual Spatial Reasoning 3D Spatial Reasoning VLM Reasoning

Open paper AskPDF arXiv

Spatial reasoning is a critical capability for intelligent robots, yet current vision-language models (VLMs) still fall short of human-level performance in video-based spatial reasoning. This gap mainly stems from two challenges: a semantic-geometric misalignment that prevents consistent 3D understanding, and the absence of persistent memory to retain 3D representation and understanding across frames. To address these limitations, we present VLM

^2

, a Vision-Language Model with persistent Memory for spatial reasoning with a view-consistent, 3D-aware representation purely from 2D videos. Specifically, we incorporate a dual-memory module consisting of a working memory that operates as a sliding window to focus on immediate context, and an episodic memory that consolidates and stores critical information across frames. This design enables bounded and efficient spatial reasoning under a fixed computational cost. Extensive experiments on multiple benchmarks show that VLM

^2

achieves state-of-the-art performance among video-based models, significantly advancing the frontier of visual-spatial intelligence.