Referring Expression Segmentation

Follow this topic Ask about this topic

The task is to find and outline the object a person describes, including cases that depend on attributes, object parts, or spatial relations. Methods commonly fuse language and visual features with cross-modal attention, or use a ground-then-segment pipeline in which a vision-language model selects a region and a segmentation model such as SAM produces the mask; newer systems also use multimodal language models to reason over candidates or ask clarifying questions. Evaluation spans 2D benchmarks such as RefCOCO, RefCOCO+, RefCOCOg, and ReasonSeg, as well as 3D scenes such as ScanRefer and specialized remote-sensing settings. A central challenge is separating correct target identification from accurate mask generation, especially for ambiguous or open-world expressions, while improving generalization without excessive training or inference cost.

Latest papers

28 papers6 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Angular Supervision for Visual Object Grounding in Autoregressive Vision-Language Models
Oct 23, 2026Parsa Esmaeilkhani, Longin Jan LateckiVision-Language Grounding VLM Interpretability
Temple University, Philadelphia, USA
UniRAP: Towards Unified Part-level Physical Affordance Reasoning and Actionable Perception
Oct 21, 2026Linfei Li, Ruining Hu, Lin Zhang +5Robotic Affordance Inference VLMs for Robotics
School of Computer Science and Technology, Tongji University · School of Electrical Engineering and Computer Science, The University of Queensland · IEIT SYSTEMS (Beijing) Co., Ltd +1
Segment Where You Look: Attention as the Interface Between Language Models and Segmenters
Oct 13, 2026Eslam Abdelrahman, Hamza Omar Alkaf, Yuheng Li +5Vision-Language Models Referring Expression Segmentation
King Abdullah University of Science and Technology (KAUST) · Adobe Research
ContourVLA: A Closed-Loop Perception-Action Contour Policy for Generalized Referring Expression Segmentation
Oct 8, 2026Ruicheng Zhang, Kaiwen Shen, Jiaqi Hou +6Referring Expression Segmentation Referring Image Segmentation
Sun Yat-sen University · Tsinghua University · The Chinese University of Hong Kong, Shenzhen +1
InstanceBench: Diagnosing Referential Reasoning and Target Identity in Referring Expression Segmentation
Oct 7, 2026Yuchen Li, Shaoyang Zhou, Yiran Wang +5Referring Expression Segmentation Referring Expression Comprehension
The University of Sydney · The ATLAS Institute · Shanghai Artificial Intelligence Laboratory
Commanding at the Speed of Sound: Audio-Centric Referring Expression Segmentation for Aerial Scenes
Sep 28, 2026Ziyun Chen, Yijun Wang, Chuanyi Zhang +5Referring Expression Segmentation Audio-Visual Understanding
Hohai University, Nanjing, China · Key Laboratory of Water Big Data Technology of Ministry of Water Resources, Hohai University, Nanjing, China
DeCo: Efficient Decouple-to-Couple Learning for Multi-Task Visual Grounding
Sep 21, 2026Xiaoqiang Lu, Licheng Jiao, Long Sun +5Multimodal Disentangled Representation Learning Referring Expression Segmentation
Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education School of Artificial Intelligence, Xidian University Xi’an, Shaanxi 710071, China
AgriScope: Pixel-Grounded Multimodal Understanding for Agricultural Images
Sep 17, 2026Abderrahmene Boudiaf, Mohamad Alanssari, Irfan Hussain +1Agricultural Image Analysis Multimodal Grounding
Department of Computer Science, Khalifa University of Science and Technology, P.O. Box 127788, Abu Dhabi, United Arab Emirates · Department of Mechanical and Nuclear Engineering, Khalifa University of Science and Technology, P.O. Box 127788, Abu Dhabi, United Arab Emirates
DRAgent: Discriminative Reasoning Agent for Referring Expression Segmentation
Aug 24, 2026Yujie Qi, Luyan ZhangReferring Expression Segmentation Text-Guided Image Segmentation
School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, 310018, China. · Khoury College of Computer Sciences, Northeastern University, Vancouver, BC V6B 1Z3, Canada.
Falcon Perception-HD: High Density Perception via Reinforcement Learning
Aug 19, 2026Sofian Chaybouti, Yasser Dahou, Ngoc Dung Huynh +2Fine-Grained Visual Perception RL for VLMs
Technology Innovation Institute, Abu Dhabi, UAE · Tübingen AI Center / University of Tübingen · MIT-IBM Watson AI Lab
Groundbench: Multi-Resolution Polygon Grounding Exposes the Geometry Gap in Vision-Language Models
Aug 18, 2026Zhonghan Bian, Zhenran Wang, Jinsong Li +1Visual Grounding VLM Evaluation
Equal contribution.
GroupForward: Building Referable 3D Scenes via Instance-Grouped Feed-Forward Gaussian Splatting
Aug 18, 2026Qijian Tian, Zimeng Wu, Xuhong Wang +23D Referring Segmentation Feed-Forward 3D Reconstruction
School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China · School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China +1
Beyond Similarity Matching: Structured Reasoning for Open-Vocabulary Referring Segmentation in 3DGS
Aug 17, 2026Yizhao Wang, Xinfa Wang, Jingbo Wang +53D Referring Segmentation Referring Expression Segmentation
School of Computer Science, Henan Institute of Science and Technology
VespaSeg: A Resource-Aware Ground-then-Segment Pipeline for Referring Expression Segmentation
Aug 2, 2026Savindu Dilshan WickramasinghePromptable Image Segmentation Efficient VLM Inference
University of Moratuwa · Department of Electronic and Telecommunication Engineering, University of Moratuwa Moratuwa, Sri Lanka
GeoSelect: Spatial-Program Execution for Training-Free Referring Remote Sensing Image Segmentation
Jul 4, 2026Yuhang Jiang, Guohui Deng, Miaozhong Xu +3Remote Sensing Image Segmentation Referring Expression Segmentation
School of Internet, Anhui University, Hefei, China · National Engineering Research Center for Agro-Ecological Big Data Analysis & Application, Anhui University, Hefei, China · State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China
Segment and Select: Vision-Language Segmentation in 3D Scenarios
Jun 9, 2026Yulin Chen, Zhihang Zhong, Yuenan Hou3D Point Cloud Segmentation 3D Referring Segmentation
Shanghai AI Laboratory · University of Science and Technology of China · Shanghai Jiaotong University
Learning to Label: A Reinforced Self-Evolving Framework for Semi-supervised Referring Expression Segmentation
May 27, 2026Runlong Cao, Ying Zang, Chuanwei Zhou +4Referring Expression Segmentation Pseudo-Labeling
School of Computer Science and Engineering, Nanjing University of Science and Technology · School of Information Engineering, Huzhou Normal University · School of Artificial Intelligence, Nanjing University of Posts and Telecommunications +3
Don't Guess, Just Ask: Resolving Ambiguity in Referring Segmentation via Multi-turn Clarification
May 17, 2026Yuting Yang, Haichao Jiang, Tianming Liang +2Image Segmentation Video Object Segmentation
School of Computer Science and Engineering, Sun Yat-sen University, China · Guangdong Province Key Laboratory of Information Security Technology, China · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China
PRIMED: Adaptive Modality Suppression for Referring Audio-Visual Segmentation via Biased Competition
May 8, 2026Yuchen He, Jing ZhangAudio-Visual Understanding Referring Expression Segmentation
School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China
Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Grounding
May 8, 2026Yuan Yao, Qiushi Yang, Humen Zhong +5Image Segmentation Referring Expression Segmentation
Tongyi Lab, Alibaba Group
Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentation
Apr 9, 2026Weiming Zhang, Dingwen Xiao, Songyue Guo +5Referring Expression Segmentation Text-Guided Image Segmentation
The Hong Kong University of Science and Technology (Guangzhou) · Nanyang Technological University
ProVG: Progressive Visual Grounding via Language Decoupling for Remote Sensing Imagery
Apr 2, 2026Ke Li, Ting Wang, Di Wang +4Remote Sensing Visual Grounding Remote Sensing VLMs
Xidian University, China · UC San Diego, USA · SUST, China
Perceptio: Perception Enhanced Vision Language Models via Spatial Token Generation
Mar 19, 2026Yuchen Li, Amanmeet Garg, Shalini Chaudhuri +2Large Vision-Language Models Visual Spatial Reasoning
Amazon
PC-CrossDiff: Point-Cluster Dual-Level Cross-Modal Differential Attention for Unified 3D Referring and Segmentation
Mar 18, 2026Wenbin Tan, Jiawen Lin, Fangyong Wang +43D Referring Segmentation 3D Visual Grounding
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, School of Informatics, Xiamen University, Xiamen, China · School of Big Data, Tongren University, Tongren, China · Hanjiang National Laboratory, Wuhan, China +2
SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentation
Mar 18, 2026Wei Tang, Xuejing Liu, Yanpeng Sun +1Referring Expression Segmentation Text-Guided Image Segmentation
School of Computer Science and Engineering, Nanjing University of Science and Technology, No. 200 Xiaolingwei Road, Nanjing 210094, China. · Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China. · NExT++ Lab, School of Computing, National University of Singapore, Singapore 119077, Singapore.
Hierarchical Collaborative Fusion for 3D Instance-aware Referring Expression Segmentation
Mar 6, 2026Keshen Zhou, Runnan Chen, Mingming Gong +13D Referring Segmentation Referring Expression Segmentation
The University of Sydney · The University of Melbourne · Mohamed bin Zayed University of Artificial Intelligence
Discriminative Perception via Anchored Description for Reasoning Segmentation
Mar 4, 2026Tao Yang, Qing Zhou, Yanliang Li +1Reasoning Segmentation VLM Reasoning
Northwestern Polytechnical University
3D-DRES: Detailed 3D Referring Expression Segmentation
Mar 3, 2026Qi Chen, Changli Wu, Jiayi Ji +23D Referring Segmentation Referring Expression Segmentation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Shanghai Innovation Institute · National University of Singapore

Oct 23, 2026J/K move · Enter open · S save

Angular Supervision for Visual Object Grounding in Autoregressive Vision-Language Models

Parsa Esmaeilkhani, Longin Jan Latecki

Temple University, Philadelphia, USA

Vision-Language Grounding VLM Interpretability Representation Geometry in VLMs Referring Expression Segmentation

Open paper AskPDF arXiv

Autoregressive vision-language models (VLMs) struggle to ground generated text in the correct image regions, and existing explainability methods, such as attention maps and Logit Lens, often produce poor spatial explanations due to weak text-to-vision attention and sink tokens. We propose Angular Relevance (

R_L

), a simple visualization measure based on cosine similarity between visual token embeddings and unembedding-matrix rows. Because

R_L

is invariant to token norm, it produces substantially sharper and more faithful heatmaps than attention maps or Logit Lens on out-of-the-box models. Without any training,

R_L

yields competitive referring-expression segmentation (RES), outperforming prior diagnostic maps and CLIP-based training-free methods. To preserve localized object information during cross-modal mixing, we introduce Angular Object-Aware Loss (AOL), an auxiliary loss that directly supervises visual token embeddings to align angularly with the vocabulary directions of their corresponding object concepts. Finetuning with AOL alongside the standard next-token prediction objective significantly sharpens object localization in heatmaps, showing that the VLM's own visual tokens can be used directly for grounding and improving RES, without training any segmentation heads and without degrading general VQA performance. We further demonstrate that the learned grounding also transfers to category-level segmentation of object categories not seen during training.