AMLRIS: Alignment-aware Masked Learning for Referring Image Segmentation

Tongfei Chen; Shuo Yang; Yuguang Yang; Linlin Yang; Runtang Guo; Changbai Li; He Long; Chunyu Xie

cs.CVFeb 26, 2026

AMLRIS: Alignment-aware Masked Learning for Referring Image Segmentation

Authors: Tongfei Chen, Shuo Yang, Yuguang Yang, Linlin Yang, Runtang Guo, Changbai Li, He Long, Chunyu Xie, +2 more

Organizations: School of Artificial Intelligence, Beihang University, China · School of Electronic Information Engineering, Beihang University, China · AI Research, Qihoo 360, China · State Key Laboratory of Media Convergence and Communication, Communication University of China, China · School of Automation Science and Electrical Engineering, Beihang University, China · Hangzhou International Innovation Institute, Beihang University

Abstract

Referring Image Segmentation (RIS) aims to segment the object in an image uniquely referred to by a natural language expression. However, RIS training often contains hard-to-align and instance-specific visual signals; optimizing on such pixels injects misleading gradients and drives the model in the wrong direction. By explicitly estimating pixel-level vision-language alignment, the learner can suppress low-alignment regions, concentrate on reliable cues, and acquire more generalizable alignment features. In this paper, we propose Alignment-Aware Masked Learning (AML), a simple yet effective training strategy that quantifies region-referent alignment (PMME) and filters out unreliable pixels during optimization (AFM). Specifically, each sample first computes a similarity map between visual and textual features, and then masks out pixels falling below an adaptive similarity threshold, thereby excluding poorly aligned regions from the training process. AML does not require architectural changes and incurs no inference overhead, directing attention to the areas aligned with the textual description. Experiments on the RefCOCO (vanilla/+/g) datasets show that AML achieves state-of-the-art results across all 8 splits, and beyond improving RIS performance, AML also enhances the model's robustness to diverse descriptions and scenarios. Code is available at https://github.com/pipashu1/AMLRIS.

Figures & tables

Figure 1: Overall results of AMLRIS, including qualitative examples, benchmark comparisons (oIoU), and cross-dataset robustness evaluation, where the model is trained only on RefCOCO+ and evaluated on RefCOCOg and RefCOCO under seven perturbation scenarios.

Figure 2: Overview of Alignment-aware Masked Learning (AML) framework.

Figure 3: Architecture for PMME and AFM modules.

Panel of Figure 4 — Figure 4: Qualitative and quantitative comparison with the Random Mask strategy.

Method	RefCOCO			RefCOCO+			RefCOCOg		Avg
	val	testA	testB	val	testA	testB	val	test
Standard: Training on the training split of each dataset.
mIoU
CRIS ( Wang et al., 2022a )	70.47	73.18	66.10	62.27	68.08	53.68	59.87	60.36	64.3
ETRIS ( Xu et al., 2023 )	71.06	74.11	66.66	62.23	68.51	52.79	60.28	60.42	64.5
BarLeRIa ( Wang et al., 2024 )	72.40	75.90	68.30	65.00	70.80	56.90	-	-	-

Table 1: Comparison with SOTA methods. * denotes the reproduced results across all experiments. Bold and underline numbers indicate the best and the second best performance. The last column is the average score across all available splits. Gray texts denote large language model based methods.

Table 6

Table 7

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: More comparison of prediction maps: Baseline , Random Mask (RM), and AML.

Panel of Figure 6 — Figure 6: Contribution of similarity scores.

Dim ( $D_{a}$ )	oIoU	mIoU	Pr@50	Pr@90	JL Error Bound ( $\epsilon$ )
1024	66.82	70.83	80.15	36.27	0.2955
2048	67.37	71.33	80.82	36.35	0.2088
4096	67.08	71.21	80.85	36.22	0.1478

Appendix

Table 8: Performance of different projection dimensions

D_{a}

and corresponding JL error bounds.

Method	Batch Size	Memory(MB)	Time/Epoch(min)
CARIS	16	35,168	30.16
CARIS+AML	16	36,902( +4.9% )	35.35( +17.2% )
DETRIS	16	12,206	27.03
DETRIS+AML	16	12,860 ( +5.3% )	32.34 ( +19.6% )

Appendix

Table 9: Comparison of training overhead.

Method	Memory (MB)	Time/Epoch (min)	10 epochs	30 epochs	50 epochs
CARIS	35,168	30.16	60.2 / 62.3	64.7 / 68.3	65.5 / 69.3
+AML / same time	36,902 ( +4.9% )	35.35 ( +17.2% )	61.8 / 64.2	65.3 / 69.5	66.7 / 70.8
+AML / same steps	36,902 ( +4.9% )	35.35 ( +17.2% )	63.8 / 65.4	66.1 / 70.1	67.4 / 71.3

Appendix

Table 10: Analysis of training overhead and fairness on CARIS. Memory and time are measured with batch size 16. Performance is reported as mIoU / oIoU on RefCOCO+.

Method	cIoU@5k	gIoU@5k	cIoU@150k	gIoU@150k
ReLA	28.49	17.33	62.50	63.25
ReLA + AML	30.86	21.08	62.61	63.52

Appendix

Table 11: Performance comparison on GRES dataset.

Metric / Epoch	1	2	3	4	5	6	7	8	9	10
$\text{sim}<0.20$ $\downarrow$	0.0041	0.0010	0.0008	0.0007	0.0008	0.0008	0.0007	0.0008	0.0007	0.0008
$\text{sim}\geq 0.50$ $\uparrow$	0.8021	0.8464	0.8620	0.8656	0.8650	0.8700	0.8720	0.8769	0.8836	0.8863
$\text{sim}\geq 0.90$ $\uparrow$	0.4192	0.4543	0.4667	0.4779	0.5022	0.5079	0.5076	0.5178	0.5196	0.5252
$\text{mean}(\text{sim})$ $\uparrow$	0.7542	0.7821	0.7917	0.7963	0.8031	0.8061	0.8142	0.8117	0.8182	0.8211
$\text{std}(\text{sim})$ $\downarrow$	0.2393	0.2230	0.2171	0.2161	0.2097	0.2075	0.2051	0.2044	0.2053	0.2041

Appendix

Table 12: Evolution of cross-modal alignment metrics across training epochs 1–10. Arrows indicate desired optimization direction:

\downarrow

for minimization,

\uparrow

for maximization.

$\tau$ / Epoch	0	1	2	3	4	5	6	7	8	9	10
0.20	0.0021	0.0007	0.0004	0.0005	0.0005	0.0005	0.0005	0.0005	0.0004	0.0005	0.0004
0.30	0.0097	0.0057	0.0058	0.0053	0.0052	0.0051	0.0049	0.0044	0.0046	0.0043	0.0043
0.40	0.0259	0.0178	0.0157	0.0139	0.0135	0.0129	0.0130	0.0125	0.0121	0.0122	0.0120
0.60	0.0726	0.0628	0.0571	0.0559	0.0558	0.0543	0.0537	0.0526	0.0523	0.0519	0.0520

Appendix

Table 13: Mask-on-ground-truth coverage rates across different threshold configurations during epochs 0–10. Lower values indicate better masking precision.

Early Training Phase (Epochs 1–17)
Epoch	1	3	5	7	9	11	13	15	17
CARIS	0.3114	0.2060	0.1636	0.1413	0.1237	0.1109	0.1002	0.0919	0.0846
AML	0.2903	0.1803	0.1434	0.1209	0.1060	0.0928	0.0816	0.0742	0.0669
Improvement	6.8%	12.5%	12.3%	14.4%	14.3%	16.3%	18.6%	19.3%	20.9%
Mid Training Phase (Epochs 18–34)
Epoch	18	20	22	24	26	28	30	32	34

Appendix

Table 14: Training loss comparison between CARIS baseline and AML across 50 epochs.

Figure 7: Visualization of the seven visual transformations used to simulate diverse visual conditions. From left to right: Original, Haze, Highlight, Lowlight, Contrast, Occlusion Box, Patch Masking, and Color Jittering.

Transformation	RefCOCO		RefCOCOg
Transformation	Baseline	Ours (AML)	Baseline	Ours (AML)
Haze	59.33 / 57.85 / 70.18	62.54 / 60.63 / 73.32	51.23 / 48.69 / 58.66	53.38 / 50.60 / 60.99
Highlight	57.81 / 59.65 / 69.76	62.28 / 59.85 / 72.76	51.74 / 48.98 / 59.74	53.66 / 50.83 / 61.50
Lowlight	58.32 / 56.81 / 69.07	61.63 / 59.81 / 72.46	50.46 / 48.02 / 58.03	52.07 / 49.43 / 60.13
Contrast	60.05 / 58.15 / 70.29	63.05 / 60.58 / 73.62	52.03 / 49.09 / 59.82	54.68 / 51.83 / 62.85
Occlusion Box	57.20 / 55.78 / 67.23	61.94 / 60.26 / 72.62	49.40 / 47.20 / 56.47	53.11 / 50.89 / 60.99

Appendix

Table 15: Cross-dataset robustness under various visual transformations. We train on RefCOCO+ and evaluate on the validation splits of RefCOCO and RefCOCOg. Results are reported as mIoU/oIoU/pr@50. Our method (AML) consistently outperforms the baseline.

Figure 8: Visual segmentation results on the Flickr30k dataset, where models trained on RefCOCO+.

Setting	mIoU	oIoU	Pr@50	Pr@70	Pr@90	Avg. Metric	Gain over Baseline
Baseline	62.69	57.97	70.24	62.41	31.27	56.92	—
$\tau=0.3,\rho=1/2$	63.79	59.19	71.69	63.69	33.42	58.36	+1.44
$\tau=0.3,\rho=1/4$	64.82	59.74	72.65	65.23	34.77	59.44	+2.52
$\tau=0.3,\rho=1/6$	64.61	59.41	72.18	64.41	34.73	59.07	+2.15
$\tau=0.4,\rho=1/2$	64.15	58.95	72.08	63.98	34.77	58.59	+1.67
$\tau=0.4,\rho=1/4$	64.61	59.51	72.04	64.63	34.26	59.01	+2.09

Appendix

Table 16: Performance on RefCOCO+ testB with different dropout ratios

\rho

. We report individual metrics, their average (Avg. Metric), and the average gain over the baseline.

Threshold $\tau$	RefCOCO		RefCOCO+		RefCOCOg
	val	testA	val	testA	val
0.2	75.00	78.14	66.99	58.05	65.78
0.3	75.81	78.09	66.82	59.19	65.13
0.4	75.45	78.33	67.37	59.51	65.67

Appendix

Table 17: Performance with different threshold

\tau

values across multiple datasets. Best performance on each dataset is bolded .

Method	val	testA	testB	Avg.
Baseline	65.54	71.86	57.97	65.12
+ Feature Mask	66.34 (+0.80)	71.21 (-0.65)	58.32 (+0.35)	65.29 (+0.17)
+ AML	67.37 (+1.83)	73.19 (+1.33)	59.51 (+1.54)	66.69 (+1.57)

Appendix

Table 18: Comparison of different masking strategies on RefCOCO+ dataset using oIoU metric.

Explore similar work

0.5%>100%: Bidirectional Reciprocal Learning for Referring Image Segmentation
Sep 21, 2026Xiaoqiang Lu, Licheng Jiao, Lingling Li +5VLM Adaptation Vision Foundation Model Adaptation
Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, International Research Center for Intelligent Perception and Computation, School of Artificial Intelligence, Xidian University, Xi’an, Shaanxi Province, 710071, China
Channel Attention-Guided Cross-Modal Knowledge Distillation for Referring Image Segmentation
Apr 18, 2026Chen YangCross-Modal Learning Cross-Modal Knowledge Distillation
School of Information Science and Technology, TaiShan University, Taian, China
Towards Motion-aware Referring Image Segmentation
Mar 18, 2026Chaeyun Kim, Seunghoon Yi, Yejin Kim +2Referring Image Segmentation Multimodal Contrastive Learning
Seoul National University · AIM Intelligence

Sep 21, 2026J/K move · Enter open · S save

0.5%>100%: Bidirectional Reciprocal Learning for Referring Image Segmentation

Xiaoqiang Lu, Licheng Jiao, Lingling Li, Yuting Yang+4

Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, International Research Center for Intelligent Perception and Computation, School of Artificial Intelligence, Xidian University, Xi’an, Shaanxi Province, 710071, China

VLM Adaptation Vision Foundation Model Adaptation Referring Image Segmentation Vision-Language Alignment Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Recent advances in vision foundation models (VFMs) have shown remarkable capabilities across diverse unimodal visual tasks. However, adapting VFMs to referring image segmentation (RIS) typically necessitates precise vision-language alignment via full fine-tuning, incurring substantial computational overhead and risking catastrophic forgetting. While existing parameter-efficient fine-tuning (PEFT) methods enable safe knowledge transfer with minimal training costs, they predominantly operate independently within individual modalities or focus exclusively on unidirectional guidance from language to vision, overlooking progressive cross-modal interaction and visual feedback for textual refinement. To address these limitations, we propose

\textbf{B}

idirectional

\textbf{R}

eciprocal

\textbf{L}

earning (

\textbf{BRL}

), a novel adapter-based PEFT framework that facilitates hierarchical, bidirectional information flow within both token-mixing and channel-mixing layers of frozen foundation models. Specifically, BRL introduces two complementary lightweight modules. The Reciprocal Attention Adapter (RAA) performs cross-modal query-key exchanges at the token level, enabling visual and linguistic tokens to mutually attend to each other for fine-grained spatial grounding. The Reciprocal Gate Adapter (RGA) generates cross-modal gating signals at the channel level, allowing global semantic context from one modality to adaptively recalibrate channel activations of the other. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg benchmarks demonstrate the superiority of BRL over prior RIS methods, achieving state-of-the-art performance while requiring less than 0.5% backbone parameter updates. Code and models will be released at https://github.com/xiaoqiang-lu/BRL.

Abstract

Explore similar work

0.5%>100%: Bidirectional Reciprocal Learning for Referring Image Segmentation

Channel Attention-Guided Cross-Modal Knowledge Distillation for Referring Image Segmentation

Towards Motion-aware Referring Image Segmentation