Towards Motion-aware Referring Image Segmentation

Chaeyun Kim; Seunghoon Yi; Yejin Kim; Yohan Jo; Joonseok Lee

cs.CVMar 18, 2026

Towards Motion-aware Referring Image Segmentation

Authors: Chaeyun Kim, Seunghoon Yi, Yejin Kim, Yohan Jo, Joonseok Lee

Organizations: Seoul National University · AIM Intelligence

Abstract

Referring Image Segmentation (RIS) requires identifying objects from images based on textual descriptions. We observe that existing methods significantly underperform on motion-related queries compared to appearance-based ones. To address this, we first introduce an efficient data augmentation scheme that extracts motion-centric phrases from original captions, exposing models to more motion expressions without additional annotations. Second, since the same object can be described differently depending on the context, we propose Multimodal Radial Contrastive Learning (MRaCL), performed on fused image-text embeddings rather than unimodal representations. For comprehensive evaluation, we introduce a new test split focusing on motion-centric queries, and introduce a new benchmark called M-Bench, where objects are distinguished primarily by actions. Extensive experiments show our method substantially improves performance on motion-centric queries across multiple RIS models, maintaining competitive results on appearance-based descriptions. Codes are available at https://github.com/snuviplab/MRaCL

Figures & tables

Figure 1: Appearance-based (left) vs. motion-centric (right) queries in RIS. Existing methods handle the former well but struggle with the latter.

	mIoU		oIoU
Models	Static	Motion	Static	Motion
CRIS ( Wang et al.,, 2022 )	68.66	52.53	65.67	45.98
LAVT ( Yang et al.,, 2022 )	71.52	53.28	70.84	46.58
DMMI ( Hu et al.,, 2023 )	73.41	51.62	71.54	45.83
ASDA ( Yue et al.,, 2024 )	76.26	54.08	75.19	48.03

Table 1: Performance gap between appearance-centric and motion-centric queries on G-Ref UMD test set.

Figure 2: Overview of the MRaCL framework , (a) We apply CE Loss to output masks of the original pairs, same as the baselines. The Fuser mix text embeddings and cross-modal embeddings from the decoder, returns multimodal representations $\textbf{z}^{(i)}$ . (b) With those multimodal latents, we calculate similarity scores and filter out false negatives. For example, with $\textbf{z}^{(1)}$ as the anchor, $\textbf{z}^{(2)}$ is considered as false negative sample and masked out as shown in the bottom right diagram. Here, components with marked as ‘ $*$ ’ indicates that it originated form the baseline model.

Figure 3: Anisotropy phenomenon observed in a baseline model (CRIS) . We plot the distribution of pairwise angular distances of 100K pairs, (a) from the original CRIS and (b) with our MRaCL loss. We clearly see that the model utilizes a significantly wider representation space using our MRaCL loss.

Figure 4: Overview of the MBench dataset annotation pipeline.

Method	+MRaCL	RefCOCO(UNC)			RefCOCO+(UNC+)			G-Ref(UMD)		Ref-ZOM	M-Ref				MBench
			oIoU			oIoU		oIoU		oIoU	mIoU		oIoU		mIoU	oIoU
		val	testA	testB	val	testA	testB	val(U)	test(U)	test(F)	static	motion	static	motion
CRIS	✗	66.68	70.62	59.93	56.94	64.20	46.97	55.91	57.05	58.62	68.66	52.53	65.67	45.98	46.21	47.62
( Wang et al.,, 2022 )	✓	67.61	71.89	61.81	57.26	65.04	47.25	58.05	58.93	60.97	70.14	54.25	67.13	49.05	47.25	49.43
	Diff	+0.93	+1.27	+1.88	+0.32	+0.84	+0.28	+2.14	+1.88	+2.35	+1.48	+1.94	+1.46	+3.07	+1.04	+1.81
LAVT	✗	72.72	75.74	68.56	61.86	67.97	54.30	61.24	62.09	64.48	71.52	53.28	70.84	46.58	52.28	57.48

Table 2: Overall comparison of the RIS models . Green cells indicate statistically significant improvement, and gray cells mean neutral. ( Red cells mean significant performance drop, but we have no such case.)

Figure 5: Visualization of activation maps with and without MRaCL on CRIS.

			Prec@
Method	mIoU	oIoU	0.5	0.7	0.9
CRIS	59.17	55.91	67.95	55.45	15.27
CRIS + L2	58.99	55.17	67.85	55.09	14.87
CRIS + SimCSE	59.77	56.09	68.15	55.45	14.42
CRIS + InfoNCE (MCC)	59.46	55.75	67.87	55.35	14.95
CRIS + MRaCL (ours)	60.87	58.05	69.73	57.45	15.85

Table 3: Comparison with various contrastive objectives , measured on G-Ref validation set (UMD).

Figure 6: Qualitative results from MBench test set.

Ratio	mIoU	oIoU	[email protected]	[email protected]	[email protected]
0	60.87	58.05	69.73	57.45	15.85
0.03	59.22	56.51	67.95	55.03	14.89
0.1	46.84	42.42	51.35	44.76	10.03
0.2	43.94	41.25	47.39	42.08	10.34

Table 4: Ablation on semi-hard negative mining. A ratio of 0 corresponds to our default augmentation-based alignment without semi-hard negatives.

Augment	MRaCL	Filtering	mIoU (%)	oIoU (%)
✗	✗	✗	59.28	55.91
✗	✓	✗	59.45	56.02
✓	✗	✗	59.31	57.34
✗	✓	✓	59.89	56.11
✓	✓	✗	60.18	57.33
✓	✓	✓	60.87	58.05

Table 5: Ablation on proposed components on G-Ref. Motion-centric augmented phrases ( Sec. 3.1 ), angular distance-based MRaCL loss ( Sec. 3.3 ), and filtering for potential false negatives ( Sec. 3.2 ).

Hyperparameter		val(UMD)		Prec(val)
		mIoU	oIoU	0.5	0.7	0.9
Margin ( $m$ )	4	59.13	56.04	68.05	54.82	15.01
	8	59.66	56.80	68.71	56.85	15.08
	10	60.08	57.51	69.36	57.16	15.12
	12	60.87	58.05	69.73	57.45	15.85
	15	60.06	56.83	69.04	57.09	15.22

Table 6: Ablation on hyperparameter selection. Experiments were conducted on G-Ref.

Filtering Option	mIoU		oIoU
	val	test	val	test
No Filtering	58.96	57.98	56.71	57.66
Exclude $>3$	59.26	59.96	57.21	57.95
Exclude $>2$	60.54	60.38	57.83	58.39
Exclude $>1$ (Ours)	60.87	60.47	58.05	58.95

Table 7: Ablation on ambiguity filtering. The strictest threshold achieves the best results on G-Ref.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure I: Examples of motion and static descriptions in M-Ref.

Figure II: Analysis of expression length and instance size

Figure III: Most frequent verbs, nouns and adverbs

Figure IV: Visualization of MBench image and its highlighted mask

Panel of Figure 18 — Figure V: Additional visualizations of predictions from the MBench test set. For each example, the left-most column presents the input expression, followed by the input image, and the ground truth mask overlaid on the image for reference. Subsequent columns across each row exhibit the comparative visualizations of model predictions before and after applying our MRaCL based framework.

Method	RefCOCO (UNC)			RefCOCO+ (UNC+)			G-Ref (UMD)
Method	Val	TestA	TestB	Val	TestA	TestB	Val	Test
Latent-VG (reproduced)	76.23	78.39	73.33	69.16	72.83	62.18	69.07	69.83
Latent-VG + MRaCL	76.94	78.87	73.95	69.49	73.51	62.56	70.02	70.66

Appendix

Table I: Additional results with LatentVG ( Yu et al.,, 2025 ) as baseline (oIoU). Applying MRaCL to a more recent and stronger baseline yields consistent improvements across all splits of all three datasets.

Explore similar work

Feb 26, 2026

AMLRIS: Alignment-aware Masked Learning for Referring Image Segmentation

Referring Image Segmentation (RIS) aims to segment the object in an image uniquely referred to by a natural language expression. However, RIS training often contains hard-to-align and instance-specific visual signals; optimizing on such pixels injects misleading gradients and drives the model in the wrong direction. By explicitly estimating pixel-level vision-language alignment, the learner can suppress low-alignment regions, concentrate on reliable cues, and acquire more generalizable alignment features. In this paper, we propose Alignment-Aware Masked Learning (AML), a simple yet effective training strategy that quantifies region-referent alignment (PMME) and filters out unreliable pixels during optimization (AFM). Specifically, each sample first computes a similarity map between visual and textual features, and then masks out pixels falling below an adaptive similarity threshold, thereby excluding poorly aligned regions from the training process. AML does not require architectural changes and incurs no inference overhead, directing attention to the areas aligned with the textual description. Experiments on the RefCOCO (vanilla/+/g) datasets show that AML achieves state-of-the-art results across all 8 splits, and beyond improving RIS performance, AML also enhances the model's robustness to diverse descriptions and scenarios. Code is available at https://github.com/pipashu1/AMLRIS.

Tongfei Chen, Shuo Yang, Yuguang Yang +7

School of Artificial Intelligence, Beihang University, China · School of Electronic Information Engineering, Beihang University, China · AI Research, Qihoo 360, China +3

Referring Image Segmentation Vision-Language Alignment

Ask PDF

Apr 18, 2026

Channel Attention-Guided Cross-Modal Knowledge Distillation for Referring Image Segmentation

Referring image segmentation (RIS) requires accurate segmentation of target regions in images according to language descriptions, which is a cross-modal task integrating vision and language. Existing RIS methods typically employ large-scale vision and language encoding models to improve performance, but their enormous parameter size severely restricts deployment in scenarios with limited computing resources. To solve this problem, this paper proposes a channel attention-guided cross-modal knowledge distillation method, which transfers the high-order fine-grained correlations between vision and language learned by the teacher network, as well as the correlations between semantic components represented by each channel, to the student network. Compared with the traditional pixel-wise relational distillation, this method not only enables the student to learn the knowledge of the teacher, but also retains part of its independent learning ability, alleviating the transfer of learning bias. Experimental results on two public datasets show that the proposed distillation method does not introduce additional parameters during inference and can achieve significant performance improvement for the student model.

Chen Yang

School of Information Science and Technology, TaiShan University, Taian, China

Cross-Modal Learning Cross-Modal Knowledge Distillation Image Segmentation Vision-Language Models Channel Attention Referring Image Segmentation

Ask PDF

Apr 1, 2026

TALENT: Target-aware Efficient Tuning for Referring Image Segmentation

Referring image segmentation aims to segment specific targets based on a natural text expression. Recently, parameter-efficient tuning (PET) has emerged as a promising paradigm. However, existing PET-based methods often suffer from the fact that visual features can't emphasize the text-referred target instance but activate co-category yet unrelated objects. We analyze and quantify this problem, terming it the

non-target activation' (NTA) issue. To address this, we propose a novel framework, TALENT, which utilizes target-aware efficient tuning for PET-based RIS. Specifically, we first propose a Rectified Cost Aggregator (RCA) to efficiently aggregate text-referred features. Then, to calibrate

NTA' into accurate target activation, we adopt a Target-aware Learning Mechanism (TLM), including contextual pairwise consistency learning and target-centric contrastive learning. The former uses the sentence-level text feature to achieve a holistic understanding of the referent and constructs a text-referred affinity map to optimize the semantic association of visual features. The latter further enhances target localization to discover the distinct instance while suppressing associations with other unrelated ones. The two objectives work in concert and address `NTA' effectively. Extensive evaluations show that TALENT outperforms existing methods across various metrics (e.g., 2.5% mIoU gains on G-Ref val set). Our codes will be released at: https://github.com/Kimsure/TALENT.

Shuo Jin, Siyue Yu, Bingfeng Zhang +3

XJTLU · University of Liverpool · China University of Petroleum (East China) +2

Referring Image Segmentation Parameter-Efficient Fine-Tuning

Ask PDF