Visual Grounding

Follow this topic Ask about this topic

Visual grounding studies how systems identify the image regions or objects referred to by language, often producing boxes, points, or masks. Common approaches include aligning text and visual features, selecting among detected objects, using attention or high-resolution crops, and training with localization annotations or reinforcement-learning rewards. The field spans referring-expression benchmarks such as RefCOCO, fine-grained spatial reasoning, medical and remote-sensing images, video, and embodied tasks such as navigation and manipulation. A central challenge is that correct answers do not prove a model used visual evidence: systems still struggle with precise localization, changing scenes, and reliable visual dependence while balancing accuracy, efficiency, and robustness.

Latest papers

155 papers23 papers in the last four weeks, up 28% on the four weeks before. 0.1% of all new papers.

From Priors to Perception: Grounding Video-LLMs in Physical Reality
May 6, 2026Zicheng Zhao, Chaofan Gan, Shijie Li +1VLM Adaptation VLM Reasoning
Shanghai Jiao Tong University
DIAGRAMS: A Review Framework for Reasoning-Level Attribution in Diagram QA
Apr 29, 2026Anirudh Iyengar Kaniyar Narayana Iyengar, Tampu Ravi Kumar, Manan Suri +4Visual Question Answering Human-in-the-Loop Annotation
Arizona State University · University of Maryland · IIITDM +1
DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams
Apr 28, 2026Anirudh Iyengar Kaniyar Narayana Iyengar, Tampu Ravi Kumar, Gaurav Najpande +4VLM Evaluation Visual Reasoning
Arizona State University · Adobe Research · University of Maryland
See Further, Think Deeper: Advancing VLM's Reasoning Ability with Low-level Visual Cues and Reflection
Apr 27, 2026Zhiheng Wu, Tong Wang, Shuning Wang +2Tool Use in VLMs VLM Reasoning
1Baidu Inc. · †Independent researcher. · 2Zhejiang University +1
MIRAGE: A Micro-Interaction Relational Architecture for Grounded Exploration in Multi-Figure Artworks
Apr 26, 2026Jui-Cheng Chiu, Yu-Chao Wang, Shengyang Luo +4Visual Spatial Reasoning Object Hallucination in VLMs
Purdue University
Grounding Video Reasoning in Physical Signals
Apr 23, 2026Alibay Osmanli, Zixu Cheng, Shaogang GongTemporal Video Grounding Video QA
Queen Mary University of London
PrecisionCUA: Iterative Visual Refinement for Pixel-Precise Cursor Grounding in Code Editors
Apr 14, 2026Himangi Mittal, Gaurav Mittal, Nelson Daniel Troncoso +1Visual Grounding GUI Automation
Microsoft · Carnegie Mellon University
TraversalBench: Challenging Paths to Follow for Vision Language Models
Apr 13, 2026Clara Petrova, Zhuo Chen, Marin SoljačićVLM Evaluation Visual Grounding
Massachusetts Institute of Technology, Department of Physics · NSF AI Institute for Artificial Intelligence and Fundamental Interactions · Massachusetts Institute of Technology, Institute for Data, Systems, and Society
ParseBench: A Document Parsing Benchmark for AI Agents
Apr 9, 2026Boyang Zhang, Sebastián G. Acosta, Preston Carlson +4Document Parsing Benchmarks Document Parsing
V-Reflection: Transforming MLLMs from Passive Observers to Active Interrogators
Mar 31, 2026Jiazhou Zhou, Yucheng Chen, Hongyang Li +4VLM Hallucination Mitigation Latent Visual Reasoning
AI Thrust, The Hong Kong University of Science and Technology (Guangzhou) · International Digital Economy Academy · MedVisAI Lab, Lee Kong Chian School of Medicine, Nanyang Technological University, and Centre of AI in Medicine +2
Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Vision
Mar 27, 2026Ling Li, Bowen Liu, Zinuo Zhan +4Multimodal Grounding Visual Grounding
Tsinghua University, Beijing, China · Dalian University of Technology, Dalian, China · Northwestern Polytechnical University, Xi’an, China +1
Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification
Mar 27, 2026Shuai Lv, Chang Liu, Feng Tang +5Multimodal Large Language Models Multimodal Reasoning
University of Science and Technology of China, Hefei, China · Huawei Foundation Model Department, Shanghai, China · The Chinese University of Hong Kong, Hong Kong, China +1
LanteRn: Latent Visual Structured Reasoning
Mar 26, 2026André G. Viveiros, Nuno Gonçalves, Matthias Lindemann +1Latent Visual Reasoning VLM Reasoning
Instituto de Telecomunicações · Instituto Superior Técnico, Universidade de Lisboa · Carnegie Mellon University
OptiSAR-Net++: A Large-Scale Benchmark and Transformer-Free Framework for Cross-Domain Remote Sensing Visual Grounding
Mar 25, 2026Xiaoyu Tang, Jun Dong, Jintao Cheng +1Remote Sensing Vision-Language Grounding
School of Electronics and Information Engineering, and Xingzhi College, South China Normal University, Foshan 528225, China · School of Data Science and Engineering, and Xingzhi College, South China Normal University, Shanwei, 516600, China · Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong SAR, China +1
Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding
Mar 24, 2026Mincheol Kwon, Minseung Lee, Seonga Choi +7Efficient VLM Inference Large Vision-Language Models
Korea University · KT Corporation · Soongsil University +1
VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models
Mar 23, 2026Zixuan Wang, Yuxin Chen, Yuqi Liu +6Vision-Language-Action Models Reasoning for VLA Models
HKUST · CUHK · SmartMore
Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs
Mar 21, 2026Huan Zheng, Yucheng Zhou, Tianyi Yan +6Endoscopy Medical Diagnosis
SKL-IOTSC, CIS, University of Macau · 2Shanghai Jiao Tong University · 3COWARobot Co. Ltd. +1
MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI
Mar 20, 2026Rozain Shakeel, Abdul Rahman Mohammad Ali, Muneeb Mushtaq +2GUI Grounding Multimodal Grounding
Gaash Research Lab, National Institute of Technology Srinagar, India · e& Group, UAE · Mohammad Bin Zayed University of Artificial Intelligence (MBZUAI), UAE +1
LoFi: Location-Aware Fine-Grained Representation Learning for Chest X-ray
Mar 19, 2026Myeongkyun Kang, Yanting Yang, Xiaoxiao LiMedical Image Representation Learning Medical VLMs
The University of British Columbia, Vancouver, BC V6T 1Z4, Canada · Vector Institute, Toronto, ON M5G 0C6, Canada
MultihopSpatial: Multi-hop Compositional Spatial Reasoning Benchmark for Vision-Language Model
Mar 19, 2026Youngwan Lee, Soojin Jang, Yoorhim Cho +3Spatial Reasoning Benchmarks Spatial Reasoning
Electronics and Telecommunications Research Institute, South Korea · Korea Advanced Institute of Science and Technology, South Korea · Sungkyunkwan University, South Korea +1
HomeGuard: VLM-based Embodied Safeguard for Identifying Contextual Risk in Household Task
Mar 15, 2026Xiaoya Lu, Yijin Zhou, Zeren Chen +6AI Agent Safety VLMs for Robotics
Shanghai AI Laboratory · Shanghai Jiao Tong University · Beihang University +1
Deeper Thought, Weaker Aim: Understanding and Mitigating Perceptual Impairment during Reasoning in Multimodal Large Language Models
Mar 15, 2026Ruiying Peng, Xueyu Wu, Jing Lei +3Multimodal Large Language Models VLM Reasoning
Tsinghua Shenzhen International Graduate School · Huawei Technologies
Think and Answer ME: Benchmarking and Exploring Multi-Entity Reasoning Grounding in Remote Sensing
Mar 13, 2026Shuchang Lyu, Haiquan Wen, Guangliang Cheng +5Remote Sensing Visual Grounding Visual Grounding
Beihang University, Beijing, China. · University of Liverpool, Liverpool, UK. · Institute of Acoustics, Chinese Academy of Sciences, Beijing, China
Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encoding
Mar 11, 2026Lin Chen, Bolin Ni, Qi Yang +5Multimodal Large Language Models Long-Context Modeling
1MAIS, Institute of Automation, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · 3Tencent +1
Perception-Aware Multimodal Spatial Reasoning from Monocular Images
Mar 7, 2026Yanchun Cheng, Rundong Wang, Xulei Yang +4Visual Spatial Reasoning VLMs for Autonomous Driving
NUS, Singapore · A*STAR, Singapore · NTU, Singapore +1
Multi-label Instance-level Generalised Visual Grounding in Agriculture
Mar 5, 2026Mohammadreza Haghighat, Alzayat Saleh, Mostafa Rahimi AzghadiVisual Grounding Agricultural Image Analysis
College of Science and Engineering, James Cook University, Townsville, QLD, Australia · Centre for AI and Data Science Innovation, James Cook University, Townsville, QLD, Australia
Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning
Mar 3, 2026Anas Zafar, Leema Krishna Murali, Ashish VashistVisual Grounding Medical VQA
The University of Texas MD Anderson Cancer Center · Cohere Labs Community · Eisai Inc. +1
3D-DRES: Detailed 3D Referring Expression Segmentation
Mar 3, 2026Qi Chen, Changli Wu, Jiayi Ji +23D Referring Segmentation Referring Expression Segmentation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Shanghai Innovation Institute · National University of Singapore
HART: High-Resolution Annotation-Free Reasoning Technique through a Closed-loop Framework
Feb 27, 2026Jiacheng Yang, Anqi Chen, Yunkai Dang +5Efficient Multimodal Inference VLM Reasoning
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · Institute of Brain-inspired Intelligence, Nanjing University, Nanjing, China · Shenzhen Research Institute of Nanjing University, Shenzhen, China
TraceVision: Trajectory-Aware Vision-Language Model for Human-Like Spatial Understanding
Feb 23, 2026Fan Yang, Shurong Zheng, Hongyin Zhao +5Visual Grounding VLM Interpretability
Foundation Model Research Center, Institute of Automation · Peng Cheng Laboratory, Shenzhen, China · School of Artificial Intelligence, University of Chinese Academy of Science, Beijing, China +2

May 6, 2026J/K move · Enter open · S save

From Priors to Perception: Grounding Video-LLMs in Physical Reality

Zicheng Zhao, Chaofan Gan, Shijie Li, Weiyao Lin

Shanghai Jiao Tong University

VLM Adaptation VLM Reasoning Visually Grounded Reasoning Visual Grounding Physical Reasoning in Video

Open paper AskPDF arXiv

While Video Large Language Models (Video-LLMs) excel in general understanding, they exhibit systematic deficits in fine-grained physical reasoning. Existing interventions not only suffer from limited generalization but fundamentally conflate generative artifacts with genuine physical fallacies. Furthermore, we find that models fail systematically not only in anti-physics anomalies but also in counter-intuitive scenarios where visual facts contradict statistical expectations. Accordingly, we propose the Unified Attribution Theory: this dual failure stems not from perception deficiency, but from Semantic Prior Dominance -- the reasoning mechanism is deeply hijacked by internal narrative scripts. To address this, we construct the Programmatic Adversarial Curriculum (PACC), the first high-fidelity adversarial video dataset synthesized based on physical laws, thoroughly decoupling visual artifacts from logical errors. Concurrently, we design the Visual-Anchored Reasoning Chain (VARC) to force models to explicitly ground their judgments in low-level visual facts prior to logical adjudication. Experiments demonstrate that without invasive architectural modifications, standard LoRA fine-tuning with the PACC curriculum effectively neutralizes prior interference in state-of-the-art (SOTA) models, yielding a substantial leap in physical reasoning capabilities.