Vision-Language Grounding

Follow this topic Ask about this topic

Vision-language grounding studies how models connect language to specific visual evidence, enabling them to locate a described object, segment a region, track an entity through video, or act on an instruction. Common approaches align text with image regions using contrastive learning, detection, segmentation, and point or box prediction; newer multimodal models also use attention, structured scene representations, visual search, and step-by-step evidence grounding. Evaluation spans referring-expression benchmarks, high-resolution and video localization, document understanding, and embodied tasks such as navigation and robotic manipulation. A central challenge is that correct answers do not prove models used the image: language priors, ambiguity, small or changing targets, and long-horizon context can all undermine faithful and robust grounding.

Latest papers

420 papers49 papers in the last four weeks, down 6% on the four weeks before. 0.3% of all new papers.

Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Grounding
May 8, 2026Yuan Yao, Qiushi Yang, Humen Zhong +5Image Segmentation Referring Expression Segmentation
Tongyi Lab, Alibaba Group
Retrieve, Integrate, and Synthesize: Spatial-Semantic Grounded Latent Visual Reasoning
May 8, 2026Jin Cui, Xinyue Long, Xunyong Zhang +5Visual Spatial Reasoning Visual Reasoning
1State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · School of Software Engineering, State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University
MSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluation
May 7, 2026Shichao Kan, Xuyang Zhang, Haojie Zhang +7Cross-Modal Alignment Vision-Language Grounding
School of Computer Science and Engineering, Central South University, Changsha, 410083, China · School of Computer Science and Technology and the State Key Laboratory of Advanced Rail Autonomous Operation, Beijing Jiaotong University, Beijing, 100044, China · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, 101408, China +1
Towards Self-Explainable Document Visual Question Answering with Chain-of-Explanation Predictions
May 7, 2026Kjetil Indrehus, Adrian Duric, Changkyu Choi +1Vision-Language Grounding Document QA
Department of Informatics, University of Oslo · Integreat – Norwegian Centre for Knowledge-driven Machine Learning · TRUST – The Norwegian Centre for Trustworthy AI
RemoteZero: Geospatial Reasoning with Zero Human Annotations
May 6, 2026Liang Yao, Fan Liu, Shengxiang Xu +4Self-Training Remote Sensing
1Hohai University · 2Southeast University
Does it Really Count? Assessing Semantic Grounding in Text-Guided Class-Agnostic Counting
May 4, 2026Giacomo Pacini, Luca Ciampi, Nicola Messina +3VLM Evaluation VLM Robustness
Institute of Information Science and Technologies of the National Research Council (ISTI-CNR), Via G. Moruzzi 1, 56124, Pisa, Italy · University of Pisa - Department of Information Engineering, Via G. Caruso 16, 56122 Pisa, Italy
AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding
May 4, 2026Ruilin Yao, Shegnwu Xiong, Tianyu Zou +2Uncertainty Quantification GUI Grounding
Wuhan University of Technology
Referring Multiple Regions with Large Multimodal Models via Contextual Latent Steering
May 3, 2026Yun Xing, Hanyuan Liu, Jiahao Nie +1Referring Expression Comprehension Multimodal Large Language Models
Nanyang Technological University
Two-Pass Zero-Shot Temporal-Spatial Grounding of Rare Traffic Events in Surveillance Video
May 2, 2026Jiantang HuangTemporal Video Grounding Large Vision-Language Models
Chain of Evidence: Pixel-Level Visual Attribution for Iterative Retrieval-Augmented Generation
May 2, 2026Peiyang Liu, Ziqiang Cui, Xi Wang +2Retrieval-Augmented Generation Iterative RAG
National Engineering Research Center for Software Engineering, Peking University Beijing, China · City University of Hong Kong Hong Kong SAR, China · Tencent Technology Beijing, China
FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Setting
Apr 30, 2026Fengxian Ji, Jingpu Yang, Zirui Song +5GUI Agents Vision-Language Grounding
MBZUAI · Northeastern University, China
COHERENCE: Benchmarking Fine-Grained Image-Text Alignment in Interleaved Multimodal Contexts
Apr 30, 2026Bingli Wang, Huanze Tang, Haijun Lv +5Vision-Language Models Vision-Language Grounding
1Southeast University · 2Shanghai AI Laboratory
InterPartAbility: Phrase-Region Grounding for Interpretable Text-to-Image Person Re-Identification
Apr 29, 2026Shakeeb Murtaza, Aryan Shukla, Rajarshi Bhattacharya +2VLM Interpretability Vision-Language Grounding
LIVIA, Dept. of Systems Engineering, ETS Montreal, Canada · Genetec Inc.
Beyond Shortcuts: Mitigating Visual Illusions in Frozen VLMs via Qualitative Reasoning
Apr 29, 2026Hao Guo, Fei Wang, Junjie Chen +4VLM Robustness Visual Reasoning
Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, China · Anhui Polytechnic University, Wuhu, China · Hefei University of Technology, Hefei, China +2
Instruction-Evidence Contrastive Dual-Stream Decoding for Grounded Vision-Language Reasoning
Apr 28, 2026Yashwant Pravinrao Bangde, Debaditya RoyVLM Inference Object Hallucination in VLMs
Department of Computer Science and Engineering Indian Institute of Technology Kharagpur Kharagpur, India 721302
A systematic evaluation of vision-language models for observational astronomical reasoning tasks
Apr 27, 2026Wenke Ren, Hengxiao Guo, Wenwen Zuo +1VLM Evaluation Vision-Language Models
Shanghai Astronomical Observatory, Chinese Academy of Sciences, Shanghai, China · Harvard University, Boston, Massachusetts, USA
Global Context or Local Detail? Adaptive Visual Grounding for Hallucination Mitigation
Apr 27, 2026Yubo Jiang, Xin Yang, Abudukelimu Wuerkaixi +7Hallucination in Language Models Object Hallucination in VLMs
School of Astronautics, Beihang University, Beijing 102206, China · Longcat Interaction Team, Meituan, Beijing 100102, China · Tianmushan Laboratory, Beihang University, Hangzhou 311115, China
Robust Grounding with MLLMs Against Occlusion and Small Objects via Language-Guided Semantic Cues
Apr 27, 2026Beomchan Park, Seongho Kim, Hyunjun Kim +2VLM Robustness LLM Grounding
Integrated Vision Language Lab., KAIST, Republic of Korea
MIRAGE: A Micro-Interaction Relational Architecture for Grounded Exploration in Multi-Figure Artworks
Apr 26, 2026Jui-Cheng Chiu, Yu-Chao Wang, Shengyang Luo +4Visual Spatial Reasoning Object Hallucination in VLMs
Purdue University
Breaking Lock-In: Preserving Steerability under Low-Data VLA Post-Training
Apr 25, 2026Suning Huang, Jiaqi Shao, Ke Wang +5VLA Model Post-Training Vision-Language Grounding
Stanford University
CAGE-SGG: Counterfactual Active Graph Evidence for Open-Vocabulary Scene Graph Generation
Apr 24, 2026Suiyang Guang, Chenyu Liu, Ruohan Zhang +1Vision-Language Grounding
Institute of Intelligent Vision and Embodied Cognition
Do MLLMs Understand Pointing? Benchmarking and Enhancing Referential Reasoning in Egocentric Vision
Apr 23, 2026Chentao Li, Zirui Gao, Mingze Gao +3Multimodal QA Vision-Language Grounding
Department of Automation, Tsinghua University · Academy of Art & Design, Tsinghua University
VG-CoT: Towards Trustworthy Visual Reasoning via Grounded Chain-of-Thought
Apr 23, 2026Byeonggeuk Lim, Kyeonghyun Kim, JungMin Yun +1VLM Evaluation CoT Reasoning
Graduate School of Advanced Imaging Science, Multimedia & Film, Chung-Ang University1 · Department of Artificial Intelligence, Chung-Ang University2
Can MLLMs "Read" What is Missing?
Apr 23, 2026Jindi Guo, Chaozheng Huang, Xi FangVLM Evaluation Document Understanding
DP Technology
Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding
Apr 23, 2026Wenkai Wang, Xiyun Li, Hongcan Guo +5Reinforcement Learning GUI Grounding
Zhejiang University · Tencent AI Lab · The University of Hong Kong
Beyond Pixels: Introspective and Interactive Grounding for Visualization Agents
Apr 22, 2026Yiyang Lu, Woong Shin, Ahmad Maroof Karimi +3Scientific Visualization Data Visualization
William & Mary · Oak Ridge National Laboratory
The Expense of Seeing: Attaining Trustworthy Multimodal Reasoning Within the Monolithic Paradigm
Apr 22, 2026Karan GoyalVLM Evaluation Multimodal Reasoning
IIIT Delhi, India
From Scene to Object: Text-Guided Dual-Gaze Prediction
Apr 22, 2026Zehong Ke, Yanbo Jiang, Jinhao Li +5Gaze Estimation Autonomous Driving Perception
School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China · Cho Chun Shik Graduate School of Mobility, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea
PLaMo 2.1-VL Technical Report
Apr 21, 2026Tommi Kerola, Yuya Masuda, Takashi Masuko +5Visual Question Answering Vision-Language Models
Preferred Networks, Inc. (PFN)
AnchorSeg: Language Grounded Query Banks for Reasoning Segmentation
Apr 20, 2026Rui Qian, Chuanhang Deng, Qiang Huang +6Image Segmentation Text-Guided Image Segmentation
College of Computer Science and Artificial Intelligence, Fudan University · 2BEDI Cloud

May 8, 2026J/K move · Enter open · S save

Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Grounding

Yuan Yao, Qiushi Yang, Humen Zhong, Jiangning Wei+4

Tongyi Lab, Alibaba Group

Image Segmentation Referring Expression Segmentation Multimodal Large Language Models Vision-Language Grounding

Open paper AskPDF arXiv

Open-world referring segmentation requires grounding unconstrained language expressions to precise pixel-level regions. Existing multimodal large language models (MLLMs) exhibit strong open-world visual grounding, but their outputs remain limited to sparse bounding-box coordinates and are insufficient for dense visual prediction. Recent MLLM-based segmentation methods either directly predict sparse contour coordinates, struggling to reconstruct continuous object boundaries, or rely on external segmentation foundation models such as the Segment Anything Model (SAM), introducing substantial architectural and deployment overhead. We present Qwen3-VL-Seg, a parameter-efficient framework that treats the MLLM-predicted box as a semantically grounded structural prior and decodes it into pixel-level referring segmentation. At its core, a lightweight box-guided mask decoder combines multi-scale spatial feature injection, spatial-semantic query construction, box-guided high-resolution pixel fusion, and iterative mask-aware query refinement, introducing only 17M parameters (about 0.4% of the base model). For scalable open-world training, we construct SA1B-ORS, an SA-1B-derived dataset with two subsets: SA1B-CoRS (category-oriented samples) and SA1B-DeRS (descriptive, instance-specific samples). For evaluation, we curate ORS-Bench, a manually screened benchmark with in-distribution and out-of-distribution subsets covering diverse referring expression types. Extensive experiments on referring expression segmentation, visual grounding, and ORS-Bench show that Qwen3-VL-Seg performs strongly across closed-set and open-world settings, with clear advantages on language-intensive instructions and strong out-of-distribution generalization. Evaluations on general multimodal benchmarks further show that the model broadly preserves general-purpose multimodal competence after segmentation-oriented adaptation.