VIPA: Visual Informative Part Attention for Referring Image Segmentation

Yubin Cho; Hyunwoo Yu; Kyeongbo Kong; Kyomin Sohn; Bongjoon Hyun; Suk-Ju Kang

cs.CVFeb 16, 2026

VIPA: Visual Informative Part Attention for Referring Image Segmentation

Authors: Yubin Cho, Hyunwoo Yu, Kyeongbo Kong, Kyomin Sohn, Bongjoon Hyun, Suk-Ju Kang

Organizations: School of Artificial Intelligence, Sogang University · AI Lab of CTO division, LG Electronics · School of Electronic Engineering, Sogang University · School of Electronic Engineering, Pusan University ·

Abstract

Referring Image Segmentation (RIS) aims to segment a target object described by a natural language expression. Existing methods have evolved by leveraging the vision information into the language tokens. To more effectively exploit visual contexts for fine-grained segmentation, we propose a novel Visual Informative Part Attention (VIPA) framework for referring image segmentation. VIPA leverages the informative parts of visual contexts, called a visual expression, which can effectively provide the structural and semantic visual target information to the network. This design reduces high-variance cross-modal projection and enhances semantic consistency in an attention mechanism of the referring image segmentation. We also design a visual expression generator (VEG) module, which retrieves informative visual tokens via local-global linguistic context cues and refines the retrieved tokens for reducing noise information and sharing informative visual attributes. This module allows the visual expression to consider comprehensive contexts and capture semantic visual contexts of informative regions. In this way, our framework enables the network's attention to robustly align with the fine-grained regions of interest. Extensive experiments and visual analysis demonstrate the effectiveness of our approach. Our VIPA outperforms the existing state-of-the-art methods on four public RIS benchmarks.

Figures & tables

Fig. 1 : (a) Example of a transformer decoder used to extract language outputs. (b) Illustration of different RIS frameworks. Different from previous works, our approach leverages visual expression , generated from the retrieved informative parts of visual contexts, as a key-value set in the Transformer-based segmentation decoder. (c) Visual comparison of two different key-value sets. Yellow dotted boxes are incorrect predictions. The visual expression (VE) robustly guides the network’s attention to the regions of interest by enhancing semantic coherence in the attention mechanism of referring image segmentation, whereas the advanced language expression (LE) results in incomplete predictions.

Fig. 2 : Overview of Visual Informative Part Attention (VIPA) framework. VIPA robustly guides the network’s attention to the region of interest by exploiting the visual expression generated from the retrieved informative parts of visual contexts.

Method	Publication	Vision	Language	RefCOCO			RefCOCO+			RefCOCOg		ReferIt
		Model	Model	val	test A	test B	val	test A	test B	val	test	test
LLM-based RIS methods + additional vision-language training datasets
LISA-7B [ 6 ]	CVPR 2024	SAM-H	Vicuna-7B	74.9	79.1	72.3	65.1	70.8	58.1	67.9	70.6	-
PixelLM [ 15 ]	CVPR 2024	CLIP-VIT-L	Vicuna-7B	73.0	76.5	68.2	66.3	71.7	58.3	69.3	70.5	-
GSVA-7B [ 16 ]	CVPR 2024	SAM-H	Vicuna-7B	77.2	78.9	73.5	65.9	69.6	59.8	72.7	73.3	-
GLaMM [ 17 ]	CVPR 2024	SAM-H	Vicuna-7B	79.5	83.2	76.9	72.6	78.7	64.6	74.2	74.9	-

TABLE I : Performance comparison with the state-of-the-art methods on four public referring image segmentation datasets. LLM-based models are marked in gray . † indicates models trained on multiple RefCOCO series datasets with removed validation and testing images to prevent data leakage. For ReferIt dataset, only ReferIt training set is used.

Fig. 3 : Qualitative comparison with the LLM-based RIS model [ 6 ] on RefCOCO+.

Method	Publication	RefCOCO			RefCOCO+			RefCOCOg
Method	Publication	val	test A	test B	val	test A	test B	val	test
C3VG [ 64 ]	AAAI’25	81.37	82.93	79.12	77.05	79.61	72.40	76.34	77.10
DeRIS-B [ 65 ]	ICCV’25	81.99	82.97	80.14	75.62	79.16	71.63	76.30	77.15
VIPA (Ours)	-	82.41	83.64	80.57	77.36	80.19	72.48	76.72	77.45

TABLE II : Comparison of more recent transformer-based methods using BEiT3 [ 63 ] as encoder.

Method	GFLOPs	oIoU (%)	Inference Time (ms)
GLaMM-7B ‡ [ 17 ]	-	74.2	786 (BF16)
LISA-7B ‡ [ 6 ]	13,026	67.9	368 (BF16)
VIPA † (Ours)	431	70.01	82 (FP32)
DMMI [ 5 ]	392	63.46	74 (FP32)
CGFormer [ 29 ]	950	64.68	105 (FP32)
VIPA (Ours)	431	65.93	82 (FP32)

TABLE III : Computational comparison with recent methods on RefCOCOg val . Inference times are measured in Brain Floating Point 16 (BF16) for LLM-based models and Floating Point 32 (FP32) for Transformer-based models. LLM-based models are marked in gray .

\ddagger

indicates models trained with additional vision-language datasets.

\dagger

indicates models trained on multiple RefCOCO series datasets.

Method	RefCOCOg val		RefCOCOg test

	seen	unseen	seen	unseen
CRIS [ 4 ]	58.64	42.63	59.68	38.88
LAVT [ 36 ]	60.16	42.33	60.37	41.38
CGFormer [ 29 ]	65.60	46.11	65.67	42.31
VIPA (Ours)	66.52	46.74	66.93	43.06

TABLE IV : Comparison for generalization setting using mIoU.

Key-Value Set	RefCOCO+ val				RefCOCOg val
	[email protected]	[email protected]	mIoU	oIoU	[email protected]	[email protected]	mIoU	oIoU
Vanilla LE	73.77	64.89	63.93	62.44	72.84	59.98	62.80	61.74
Advanced LE	75.26	66.49	66.14	64.16	74.35	61.49	65.06	63.52
VE	78.64	69.62	70.42	66.91	77.03	65.26	69.98	65.93

TABLE V : Ablation for the effectiveness of the visual informative part attention framework. LE: Linguistic Expression tokens. VE: Visual Expression tokens (Ours). are ablation methods that leverage language-based tokens as key-value components in the segmentation decoder. Advanced LE indicates a method using visual-aware language tokens extracted by cross-attention with the vision features. is our method that leverages the visual expression, generated from the retrieved informative parts of visual contexts.

	Design		RefCOCO+ val		RefCOCOg val
(a)	Step 1	Step 2	mIoU	oIoU	mIoU	oIoU

	✗	✓	66.67	64.72	66.31	64.04
	✓	✗	66.29	64.54	65.82	63.79
	✓	✓	70.42	66.91	69.98	65.93
(b)	Global	Local	mIoU	oIoU	mIoU	oIoU

TABLE VI : Ablation studies for the design of the visual expression generator module. Our default design is marked in .

Encoder Feature Extraction	Method	RefCOCO+
	Method
		val	test A	test B
w/o Fusion	w/o VE	62.74	68.41	55.82
	w/ VE	65.15	71.17	58.34
Late Fusion	w/o VE	63.88	69.65	57.07
	w/ VE	66.74	72.21	59.73

TABLE VII : Ablation for different encoder fusion types.

Panel of Figure 11 — Fig. 4 : Ablation study on the number of the retrieved informative visual tokens.

Method	RefCOCO+ val	RefCOCO+ test A	RefCOCO+ test B
w/o Supervision	65.93	71.50	59.26
Direct thresholding loss	66.52	71.93	59.60
Pixel contrastive loss	66.91	72.44	60.15

TABLE VIII : Additional ablation on supervision by the contrastive loss. Our default design is marked in .

Method	RefCOCO val		RefCOCO+ val		RefCOCOg val

	mIoU	oIoU	mIoU	oIoU	mIoU	oIoU
w/o articles	77.80	75.11	69.35	66.32	67.74	64.99
All words	78.68	75.84	70.42	66.91	69.98	65.93

TABLE IX : Ablation study on the use of the article tokens at the process of retrieving informative visual contexts. Our default design is marked in .

Fig. 5 : t-SNE visualizations and qualitative comparisons between the ablation method ( i.e ., advanced language expression) and our method ( i.e ., visual expression).

Fig. 6 : (a-b) Visualized comparison with the ablation method ( i.e ., advanced Language Expression (LE)) and our method ( i.e ., Visual Expression (VE)). (c) Visual analysis of the attention map between vision query features and visual expression key-value set, and the attention map between vision query features and advanced language expression key-value set.

Fig. 7 : (a) Qualitative comparison with Transformer-based RIS methods. (b) Our results on long and difficult language inputs.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 8 : Additional visualization comparison of our method and the ablated method on various target object categories, where the ablation model misses the target regions.

Fig. 9 : Additional visualization comparison of our method and the ablated method on various target object categories, where the ablation model segments even non-target regions.

Fig. 10 : Additional visualization comparison of our method and the ablated method on the target object of the person, where the ablation model fails to capture the target regions.

Referring Image Segmentation

Referring image segmentation maps a natural-language description of an object or region to its pixels in an image.

See all 45 papers

Spatio-Semantic Expert Routing Architecture with Mixture-of-Experts for Referring Image Segmentation
Mar 13, 2026Alaa Dalaq, Muzammil BehzadReferring Image Segmentation Boundary-Aware Segmentation
King Fahd University of Petroleum and Mineral, Dhahran, 31261, Saudi Arabia · SDAIA-KFUPM Joint Research Center for Artificial Intelligence, Dhahran, 31261, Saudi Arabia
Progressive Prompt-Guided Cross-Modal Reasoning for Referring Image Segmentation
Mar 30, 2026Jiachen Li, Hongyun Wang, Jinyu Xu +5Referring Image Segmentation Promptable Image Segmentation
School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan, China · School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China · University of Electronic Science and Technology of China, Chengdu, China +1
0.5%>100%: Bidirectional Reciprocal Learning for Referring Image Segmentation
Sep 21, 2026Xiaoqiang Lu, Licheng Jiao, Lingling Li +5VLM Adaptation Vision Foundation Model Adaptation
Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, International Research Center for Intelligent Perception and Computation, School of Artificial Intelligence, Xidian University, Xi’an, Shaanxi Province, 710071, China

Mar 13, 2026J/K move · Enter open · S save

Spatio-Semantic Expert Routing Architecture with Mixture-of-Experts for Referring Image Segmentation

Alaa Dalaq, Muzammil Behzad

King Fahd University of Petroleum and Mineral, Dhahran, 31261, Saudi Arabia · SDAIA-KFUPM Joint Research Center for Artificial Intelligence, Dhahran, 31261, Saudi Arabia

Referring Image Segmentation Boundary-Aware Segmentation Mixture of Experts

Open paper AskPDF arXiv

Referring image segmentation aims to produce a pixel-level mask for the image region described by a natural-language expression. Although pretrained vision-language models have improved semantic grounding, many existing methods still rely on uniform refinement strategies that do not fully match the diverse reasoning requirements of referring expressions. Because of this mismatch, predictions often contain fragmented regions, inaccurate boundaries, or even the wrong object, especially when pretrained backbones are frozen for computational efficiency. To address these limitations, we propose SERA, a Spatio-Semantic Expert Routing Architecture for referring image segmentation. SERA introduces lightweight, expression-aware expert refinement at two complementary stages within a vision-language framework. First, we design SERA-Adapter, which inserts an expression-conditioned adapter into selected backbone blocks to improve spatial coherence and boundary precision through expert-guided refinement and cross-modal attention. We then introduce SERA-Fusion, which strengthens intermediate visual representations by reshaping token features into spatial grids and applying geometry-preserving expert transformations before multimodal interaction. In addition, a lightweight routing mechanism adaptively weights expert contributions while remaining compatible with pretrained representations. To make this routing stable under frozen encoders, SERA uses a parameter-efficient tuning strategy that updates only normalization and bias terms, affecting less than 1% of the backbone parameters. Experiments on standard referring image segmentation benchmarks show that SERA consistently outperforms strong baselines, with especially clear gains on expressions that require accurate spatial localization and precise boundary delineation.

Abstract

Referring Image Segmentation

Explore similar work

Spatio-Semantic Expert Routing Architecture with Mixture-of-Experts for Referring Image Segmentation

Progressive Prompt-Guided Cross-Modal Reasoning for Referring Image Segmentation

0.5%>100%: Bidirectional Reciprocal Learning for Referring Image Segmentation