Vision-Language Grounding

Follow this topic Ask about this topic

Vision-language grounding studies how models connect language to specific visual evidence, enabling them to locate a described object, segment a region, track an entity through video, or act on an instruction. Common approaches align text with image regions using contrastive learning, detection, segmentation, and point or box prediction; newer multimodal models also use attention, structured scene representations, visual search, and step-by-step evidence grounding. Evaluation spans referring-expression benchmarks, high-resolution and video localization, document understanding, and embodied tasks such as navigation and robotic manipulation. A central challenge is that correct answers do not prove models used the image: language priors, ambiguity, small or changing targets, and long-horizon context can all undermine faithful and robust grounding.

Latest papers

420 papers49 papers in the last four weeks, down 6% on the four weeks before. 0.3% of all new papers.

Instruction-as-State: Environment-Guided and State-Conditioned Semantic Understanding for Embodied Navigation
Apr 20, 2026Zhen Liu, Yuhan Liu, Jinjun Wang +3Embodied Navigation Vision-Language Navigation
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, China · The Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xian Ning West Road No. 28, Xi’an, 710049, Shaanxi, China · School of Information Science and Technology, North China University of Technology, 100144, Beijing, China +1
DiffuSAM: Diffusion Guided Zero-Shot Object Grounding for Remote Sensing Imagery
Apr 20, 2026Geet Sethi, Panav Shah, Ashutosh Gandhe +1Remote Sensing Image Understanding Diffusion Model Guidance
Indian Institute of Technology Bombay
Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models
Apr 20, 2026Ziyao Tang, Pengkun Jiao, Bin Zhu +3VLM Evaluation VLM Robustness
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · Singapore Management University
AutoVQA-G: Self-Improving Agentic Framework for Automated Visual Question Answering and Grounding Annotation
Apr 19, 2026Rongsheng Hu, Runwei Guan, Yicheng Di +2Visual Question Answering LLM-Assisted Annotation
School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China · Thrust of Artificial Intelligence, HKUST(GZ), Guangzhou, China
More Than Meets the Eye: Measuring the Semiotic Gap in Vision-Language Models via Semantic Anchorage
Apr 19, 2026Wei HeVLM Evaluation Figurative Language Understanding
IDSAI, University of Exeter Exeter, UK
Prompt Sensitivity in Vision-Language Grounding: How Small Changes in Wording Affect Object Detection
Apr 18, 2026Dawar Jyoti Deka, Amit Sethi, Syed Mohammad AliOpen-Vocabulary Object Detection VLM Robustness
Department of Aerospace Engineering, Indian Institute of Technology Bombay, Mumbai, India · Department of Electrical Engineering, Indian Institute of Technology Bombay, Mumbai, India
DO-Bench: An Attributable Benchmark for Diagnosing Object Hallucination in Vision-Language Models
Apr 18, 2026JiYang Wang, Jiawei Chen, Mengqi Xiao +3VLM Evaluation Hallucination Detection in VLMs
Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University · Zhongguancun Academy
Zoom Consistency: A Free Confidence Signal in Multi-Step Visual Grounding Pipelines
Apr 15, 2026Keon Kim, Krish ChelikavadaConfidence Estimation for VLMs Vision-Language Grounding
Om Labs
UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding
Apr 15, 2026Fei Tang, Bofan Chen, Zhengxi Lu +8GUI Grounding Vision-Language Grounding
Zhejiang University · Ant Group
SmoGVLM: A Small, Graph-enhanced Vision-Language Model
Apr 15, 2026Debjyoti Mondal, Rituraj Singh, Subhadarshi PandaVLM Reasoning Vision-Language Grounding
Samsung R&D Institute India-Bangalore
Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation
Apr 14, 2026Sihang Jia, Shuliang Liu, Songbo Yang +3VLM Hallucination Mitigation Multimodal Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)
Grounded World Model: Latent Planning with Language Goals
Apr 13, 2026Quanyi Li, Lan Feng, Haonan Zhang +4Language-Conditioned Robot Planning World Models for Robotics
EPFL · NUS
CLASP: Closed-loop Asynchronous Spatial Perception for Open-vocabulary Desktop Object Grasping
Apr 13, 2026Yiran Ling, Wenxuan Li, Siying Dong +5Spatial Grounding for VLA Models Robotic Grasping
Harbin Institute of Technology · National Key Laboratory of Smart Farm Technologies and Systems · Northeastern University +1
BareBones: Benchmarking Zero-Shot Geometric Comprehension in VLMs
Apr 12, 2026Aaditya Baranwal, Vishal Yadav, Abhishek RajoraVLM Evaluation VLM Reasoning
(Independent Researcher) · University of Calgary
ProGAL-VLA: Grounded Alignment through Prospective Reasoning in Vision-Language-Action Models
Apr 10, 2026Nastaran Darabi, Amit Ranjan TrivediSpatial Grounding for VLA Models Vision-Language-Action Models
University of Illinois Chicago, IL, USA
EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Tasks
Apr 10, 2026Lulin Liu, Dayou Li, Yiqing Liang +8Long-Horizon Planning VLM Reasoning
UMN · TAMU · Brown University +5
Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts
Apr 10, 2026Farhad Nooralahzadeh, Omid Rohanian, Yi Zhang +2Vision-Language Grounding Visual Evidence Sensitivity in VLMs
Institute of Computer Science, Zurich University of Applied Sciences, Switzerland · University of Oxford, UK.
See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignment
Apr 10, 2026Mohammad Anas Azeez, Ankan Deria, Zohaib Hasan Siddiqui +5VLM Hallucination Mitigation Object Hallucination in VLMs
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia · Macquarie University, Sydney, Australia +2
Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models
Apr 9, 2026Marcel Gröpl, Jaewoo Jung, Seungryong Kim +2Vision-Language Grounding VLM Interpretability
ETH Zürich · KAIST AI · ETH AI Center
InstAP: Instance-Aware Vision-Language Pre-Train for Spatial-Temporal Understanding
Apr 9, 2026Ashutosh Kumar, Rajat Saini, Jingjing Pan +5Vision-Language Pretraining Vision-Language Alignment
Woven by Toyota
Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding
Apr 9, 2026Xuezhen Tu, Jingyu Wu, Fangyu Kang +4Vision-Language Grounding Temporal Video Grounding
Shanghai Jiao Tong University China · ZTE Corporation China
Decompose, Look, and Reason: Reinforced Latent Reasoning for VLMs
Apr 8, 2026Mengdan Zhu, Senhao Cheng, Liang ZhaoVision-Language Grounding VLM Interpretability
Emory University · University of Michigan, Ann Arbor
ValueGround: Evaluating Culture-Conditioned Visual Value Grounding in MLLMs
Apr 7, 2026Zhipin Wang, Christoph Leiter, Christian Frey +3Cultural Knowledge in Language Models VLM Evaluation
University of Technology Nuremberg (UTN), Germany
Reasoning-Guided Grounding: Elevating Video Anomaly Detection through Multimodal Large Language Models
Apr 7, 2026Sakshi Agarwal, Aishik Konwer, Ankit Parag ShahVideo Anomaly Detection Interpretable Anomaly Detection
Center of Advanced AI, Accenture.
Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models
Apr 7, 2026Keuntae Kim, Mingyu Kang, Yong Suk ChoiVLM Reasoning Diffusion Language Models
Department of Computer Science, Hanyang University · Department of Artificial Intelligence, Hanyang University
Watch Before You Answer: Learning from Visually Grounded Post-Training
Apr 6, 2026Yuxuan Zhang, EunJeong Hwang, Huaisong Zhang +8VLM Post-Training Vision-Language Models
University of British Columbia · Vector Institute · Etude AI +4
Firebolt-VL: Efficient Vision-Language Understanding with Cross-Modality Modulation
Apr 6, 2026Quoc-Huy Trinh, Mustapha Abdullahi, Bo Zhao +1Efficient VLM Inference Vision-Language Grounding
Aalto University · University of South Dakota
Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward
Apr 6, 2026Shizhan Gong, Minda Hu, Qiyuan Zhang +2VLM Reasoning Faithfulness in VLMs
The Chinese University of Hong Kong, Hong Kong, China · City University of Hong Kong, Hong Kong, China
VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors
Apr 2, 2026Haz Sameen Shahgir, Xiaofu Chen, Yu Fu +4VLM Reasoning Fine-Grained Visual Perception
University of California, Riverside 1 · MBZUAI2
Beyond Referring Expressions: Scenario Comprehension Visual Grounding
Apr 2, 2026Ruozhen He, Nisarg A. Shah, Qihua Dong +3Vision-Language Grounding Visually Grounded Reasoning
Rice University · Johns Hopkins University · Northeastern University

Apr 20, 2026J/K move · Enter open · S save

Instruction-as-State: Environment-Guided and State-Conditioned Semantic Understanding for Embodied Navigation

Zhen Liu, Yuhan Liu, Jinjun Wang, Jianyi Liu+2

National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, China · The Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xian Ning West Road No. 28, Xi’an, 710049, Shaanxi, China · School of Information Science and Technology, North China University of Technology, 100144, Beijing, China+1

Embodied Navigation Vision-Language Navigation Vision-Language Grounding

Open paper AskPDF arXiv

Vision-and-Language Navigation requires agents to follow natural-language instructions in visually changing environments. A central challenge is the dynamic entanglement between language and observations: the meaning of instruction shifts as the agent's field of view and spatial context evolve. However, many existing models encode the instruction as a static global representation, limiting their ability to adapt instruction meaning to the current visual context. We therefore model instruction understanding as an Instruction-as-State variable: a decision-relevant, token-level instruction state that evolves step by step conditioned on the agent's perceptual state, where the perceptual state denotes the observation-grounded navigation context at each step. To realize this principle, we introduce State-Entangled Environment-Guided Instruction Understanding (S-EGIU), a coarse-to-fine framework for state-conditioned segment activation and token-level semantic refinement. At the coarse level, S-EGIU activates the instruction segment whose semantics align with the current observation. At the fine level, it refines the activated segment through observation-guided token grounding and contextual modeling, sharpening its internal semantics under the current observation. Together, these stages maintain an instruction state that is continuously updated according to the agent's perceptual state during navigation. S-EGIU delivers strong performance on several key metrics, including a +2.68% SPL gain on REVERIE Test Unseen, and demonstrates consistent efficiency gains across multiple VLN benchmarks, underscoring the value of dynamic instruction--perception entanglement.