Towards Motion-aware Referring Image Segmentation

Chaeyun Kim; Seunghoon Yi; Yejin Kim; Yohan Jo; Joonseok Lee

cs.CVMar 18, 2026

Towards Motion-aware Referring Image Segmentation

Authors: Chaeyun Kim, Seunghoon Yi, Yejin Kim, Yohan Jo, Joonseok Lee

Organizations: Seoul National University · AIM Intelligence

Abstract

Referring Image Segmentation (RIS) requires identifying objects from images based on textual descriptions. We observe that existing methods significantly underperform on motion-related queries compared to appearance-based ones. To address this, we first introduce an efficient data augmentation scheme that extracts motion-centric phrases from original captions, exposing models to more motion expressions without additional annotations. Second, since the same object can be described differently depending on the context, we propose Multimodal Radial Contrastive Learning (MRaCL), performed on fused image-text embeddings rather than unimodal representations. For comprehensive evaluation, we introduce a new test split focusing on motion-centric queries, and introduce a new benchmark called M-Bench, where objects are distinguished primarily by actions. Extensive experiments show our method substantially improves performance on motion-centric queries across multiple RIS models, maintaining competitive results on appearance-based descriptions. Codes are available at https://github.com/snuviplab/MRaCL

Figures & tables

Figure 1: Appearance-based (left) vs. motion-centric (right) queries in RIS. Existing methods handle the former well but struggle with the latter.

	mIoU		oIoU
Models	Static	Motion	Static	Motion
CRIS ( Wang et al.,, 2022 )	68.66	52.53	65.67	45.98
LAVT ( Yang et al.,, 2022 )	71.52	53.28	70.84	46.58
DMMI ( Hu et al.,, 2023 )	73.41	51.62	71.54	45.83
ASDA ( Yue et al.,, 2024 )	76.26	54.08	75.19	48.03

Table 1: Performance gap between appearance-centric and motion-centric queries on G-Ref UMD test set.

Figure 2: Overview of the MRaCL framework , (a) We apply CE Loss to output masks of the original pairs, same as the baselines. The Fuser mix text embeddings and cross-modal embeddings from the decoder, returns multimodal representations $\textbf{z}^{(i)}$ . (b) With those multimodal latents, we calculate similarity scores and filter out false negatives. For example, with $\textbf{z}^{(1)}$ as the anchor, $\textbf{z}^{(2)}$ is considered as false negative sample and masked out as shown in the bottom right diagram. Here, components with marked as ‘ $*$ ’ indicates that it originated form the baseline model.

Figure 3: Anisotropy phenomenon observed in a baseline model (CRIS) . We plot the distribution of pairwise angular distances of 100K pairs, (a) from the original CRIS and (b) with our MRaCL loss. We clearly see that the model utilizes a significantly wider representation space using our MRaCL loss.

Figure 4: Overview of the MBench dataset annotation pipeline.

Method	+MRaCL	RefCOCO(UNC)			RefCOCO+(UNC+)			G-Ref(UMD)		Ref-ZOM	M-Ref				MBench
			oIoU			oIoU		oIoU		oIoU	mIoU		oIoU		mIoU	oIoU
		val	testA	testB	val	testA	testB	val(U)	test(U)	test(F)	static	motion	static	motion
CRIS	✗	66.68	70.62	59.93	56.94	64.20	46.97	55.91	57.05	58.62	68.66	52.53	65.67	45.98	46.21	47.62
( Wang et al.,, 2022 )	✓	67.61	71.89	61.81	57.26	65.04	47.25	58.05	58.93	60.97	70.14	54.25	67.13	49.05	47.25	49.43
	Diff	+0.93	+1.27	+1.88	+0.32	+0.84	+0.28	+2.14	+1.88	+2.35	+1.48	+1.94	+1.46	+3.07	+1.04	+1.81
LAVT	✗	72.72	75.74	68.56	61.86	67.97	54.30	61.24	62.09	64.48	71.52	53.28	70.84	46.58	52.28	57.48

Table 2: Overall comparison of the RIS models . Green cells indicate statistically significant improvement, and gray cells mean neutral. ( Red cells mean significant performance drop, but we have no such case.)

Figure 5: Visualization of activation maps with and without MRaCL on CRIS.

			Prec@
Method	mIoU	oIoU	0.5	0.7	0.9
CRIS	59.17	55.91	67.95	55.45	15.27
CRIS + L2	58.99	55.17	67.85	55.09	14.87
CRIS + SimCSE	59.77	56.09	68.15	55.45	14.42
CRIS + InfoNCE (MCC)	59.46	55.75	67.87	55.35	14.95
CRIS + MRaCL (ours)	60.87	58.05	69.73	57.45	15.85

Table 3: Comparison with various contrastive objectives , measured on G-Ref validation set (UMD).

Figure 6: Qualitative results from MBench test set.

Ratio	mIoU	oIoU	[email protected]	[email protected]	[email protected]
0	60.87	58.05	69.73	57.45	15.85
0.03	59.22	56.51	67.95	55.03	14.89
0.1	46.84	42.42	51.35	44.76	10.03
0.2	43.94	41.25	47.39	42.08	10.34

Table 4: Ablation on semi-hard negative mining. A ratio of 0 corresponds to our default augmentation-based alignment without semi-hard negatives.

Augment	MRaCL	Filtering	mIoU (%)	oIoU (%)
✗	✗	✗	59.28	55.91
✗	✓	✗	59.45	56.02
✓	✗	✗	59.31	57.34
✗	✓	✓	59.89	56.11
✓	✓	✗	60.18	57.33
✓	✓	✓	60.87	58.05

Table 5: Ablation on proposed components on G-Ref. Motion-centric augmented phrases ( Sec. 3.1 ), angular distance-based MRaCL loss ( Sec. 3.3 ), and filtering for potential false negatives ( Sec. 3.2 ).

Hyperparameter		val(UMD)		Prec(val)
		mIoU	oIoU	0.5	0.7	0.9
Margin ( $m$ )	4	59.13	56.04	68.05	54.82	15.01
	8	59.66	56.80	68.71	56.85	15.08
	10	60.08	57.51	69.36	57.16	15.12
	12	60.87	58.05	69.73	57.45	15.85
	15	60.06	56.83	69.04	57.09	15.22

Table 6: Ablation on hyperparameter selection. Experiments were conducted on G-Ref.

Filtering Option	mIoU		oIoU
	val	test	val	test
No Filtering	58.96	57.98	56.71	57.66
Exclude $>3$	59.26	59.96	57.21	57.95
Exclude $>2$	60.54	60.38	57.83	58.39
Exclude $>1$ (Ours)	60.87	60.47	58.05	58.95

Table 7: Ablation on ambiguity filtering. The strictest threshold achieves the best results on G-Ref.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure I: Examples of motion and static descriptions in M-Ref.

Figure II: Analysis of expression length and instance size

Figure III: Most frequent verbs, nouns and adverbs

Figure IV: Visualization of MBench image and its highlighted mask

Panel of Figure 18 — Figure V: Additional visualizations of predictions from the MBench test set. For each example, the left-most column presents the input expression, followed by the input image, and the ground truth mask overlaid on the image for reference. Subsequent columns across each row exhibit the comparative visualizations of model predictions before and after applying our MRaCL based framework.

Method	RefCOCO (UNC)			RefCOCO+ (UNC+)			G-Ref (UMD)
Method	Val	TestA	TestB	Val	TestA	TestB	Val	Test
Latent-VG (reproduced)	76.23	78.39	73.33	69.16	72.83	62.18	69.07	69.83
Latent-VG + MRaCL	76.94	78.87	73.95	69.49	73.51	62.56	70.02	70.66

Appendix

Table I: Additional results with LatentVG ( Yu et al.,, 2025 ) as baseline (oIoU). Applying MRaCL to a more recent and stronger baseline yields consistent improvements across all splits of all three datasets.

Explore similar work

AMLRIS: Alignment-aware Masked Learning for Referring Image Segmentation
Feb 26, 2026Tongfei Chen, Shuo Yang, Yuguang Yang +7Referring Image Segmentation Vision-Language Alignment
School of Artificial Intelligence, Beihang University, China · School of Electronic Information Engineering, Beihang University, China · AI Research, Qihoo 360, China +3
Channel Attention-Guided Cross-Modal Knowledge Distillation for Referring Image Segmentation
Apr 18, 2026Chen YangCross-Modal Learning Cross-Modal Knowledge Distillation
School of Information Science and Technology, TaiShan University, Taian, China
TALENT: Target-aware Efficient Tuning for Referring Image Segmentation
Apr 1, 2026Shuo Jin, Siyue Yu, Bingfeng Zhang +3Referring Image Segmentation Parameter-Efficient Fine-Tuning
XJTLU · University of Liverpool · China University of Petroleum (East China) +2

Feb 26, 2026J/K move · Enter open · S save

AMLRIS: Alignment-aware Masked Learning for Referring Image Segmentation

Tongfei Chen, Shuo Yang, Yuguang Yang, Linlin Yang+6

School of Artificial Intelligence, Beihang University, China · School of Electronic Information Engineering, Beihang University, China · AI Research, Qihoo 360, China+3

Referring Image Segmentation Vision-Language Alignment

Open paper AskPDF arXiv

Referring Image Segmentation (RIS) aims to segment the object in an image uniquely referred to by a natural language expression. However, RIS training often contains hard-to-align and instance-specific visual signals; optimizing on such pixels injects misleading gradients and drives the model in the wrong direction. By explicitly estimating pixel-level vision-language alignment, the learner can suppress low-alignment regions, concentrate on reliable cues, and acquire more generalizable alignment features. In this paper, we propose Alignment-Aware Masked Learning (AML), a simple yet effective training strategy that quantifies region-referent alignment (PMME) and filters out unreliable pixels during optimization (AFM). Specifically, each sample first computes a similarity map between visual and textual features, and then masks out pixels falling below an adaptive similarity threshold, thereby excluding poorly aligned regions from the training process. AML does not require architectural changes and incurs no inference overhead, directing attention to the areas aligned with the textual description. Experiments on the RefCOCO (vanilla/+/g) datasets show that AML achieves state-of-the-art results across all 8 splits, and beyond improving RIS performance, AML also enhances the model's robustness to diverse descriptions and scenarios. Code is available at https://github.com/pipashu1/AMLRIS.

Abstract

Explore similar work

AMLRIS: Alignment-aware Masked Learning for Referring Image Segmentation

Channel Attention-Guided Cross-Modal Knowledge Distillation for Referring Image Segmentation

TALENT: Target-aware Efficient Tuning for Referring Image Segmentation