Intra- and Cross-Scale Feature Interaction Network for Referring Remote Sensing Image Segmentation

Zhigang Yang; Huiguang Yao; Linmao Tian; Xuezhi Zhao; Qiang Li; Qi Wang

cs.CVJun 4, 2025

Intra- and Cross-Scale Feature Interaction Network for Referring Remote Sensing Image Segmentation

Authors: Zhigang Yang, Huiguang Yao, Linmao Tian, Xuezhi Zhao, Qiang Li, Qi Wang

Organizations: School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University, Xi’an 710072, P.R. China · School of Computer Science, Northwestern Polytechnical University, Xi’an 710072, P.R. China · department of Software, Northwest Institute of Nuclear Technology, Xi’an 710072, P.R. China

Abstract

Referring remote sensing image segmentation (RRSIS) is a challenging multimodal task that requires accurate localization and pixel-level segmentation of targets specified by natural-language expressions. Existing RRSIS datasets remain limited in image resolution, scene diversity, and category coverage, which restricts the generalization ability and practical applicability of current methods. To advance this field, we introduce NWPU-Refer, the largest and most diverse bilingual RRSIS dataset to date. It contains 15,003 high-resolution images (1024--2048 px) collected from more than 30 countries across five continents, together with 49,745 annotated targets covering single-object, multi-object, and no-target scenarios. We further propose the Multi-scale Referring Segmentation Network (MRSNet), a framework tailored to the characteristics of remote sensing imagery. Specifically, MRSNet incorporates an Intra-scale Feature Interaction Module (IFIM) to refine fine-grained features within each encoder stage and a Hierarchical Feature Interaction Module (HFIM) to integrate cross-scale information while preserving spatial structure. Extensive experiments on NWPU-Refer and RRSIS-D demonstrate that MRSNet achieves strong and competitive performance across multiple metrics, especially under strict IoU thresholds and IoU-based evaluation, validating the effectiveness of the proposed dataset and model. The dataset and code are publicly available at https://github.com/CVer-Yang/NWPU-Refer.

Figures & tables

Fig. 1: Comparison of refer segmentation datasets for four target categories: bridges, sports fields, ships, and airplanes. As shown in the left column, existing datasets primarily consist of simple scenes with prominent and easily segmented objects. In contrast, our proposed dataset (right column) introduces greater detail and complexity, better capturing the intricate real-world environments encountered in practical applications.

Fig. 2: Illustration of data distribution characteristics. (A)–(B) Word clouds from annotations highlight the frequency of target-related terms. Common targets (e.g., cars and ships) appear more frequently, while rare targets (e.g., badminton courts) appear less often. (C) The annotation-category distribution reflects the class imbalance between frequent and rare targets. (D) The annotation-type distribution shows the proportion of single-target, multi-target, and non-target descriptions.

Fig. 3: Visualization examples of diverse segmentation tasks in the NWPU-Refer dataset. Each sample includes Chinese and English expressions, together with the corresponding target regions.

Dataset	Image Resolution / Size	Images	Annotations	Object Categories	Attribute Types	Single-object	Multi-object	Non-object	Annotation Generation
RefSegRS	0.13m / 512	4420	4420	20	3	$\checkmark$	$\checkmark$	$\times$	Manual
RRSIS-D	0.5m–30m / 800	17402	17402	20	7	$\checkmark$	$\times$	$\times$	Semi-auto
NWPU-Refer	0.12m–0.5m / 1024-2048	15003	49745	32	6	$\checkmark$	$\checkmark$	$\checkmark$	Manual

TABLE I: Comparison with existing remote sensing referring segmentation datasets in terms of spatial resolution, image size, annotation scale, object-category coverage, attribute diversity, and annotation generation.

Fig. 4: Overview of the proposed MRSNet. The inputs are a remote sensing image and language tokens. Encoder layers denote the four stages of the Swin-Base backbone, and each stage is equipped with an Intra-scale Feature Interaction Module (IFIM). The resulting stage features are passed to the Hierarchical Feature Integration Module (HFIM) for cross-scale aggregation before decoder prediction.

Fig. 5: The structure of the proposed pyramidal spatial-spectral refinement submodule.

Fig. 6: The structure of the proposed context-aware spatial relation modeling submodule.

Fig. 7: The structure of the proposed Cross-Modal Alignment submodule.

Method	Publication	[email protected]		[email protected]		[email protected]		[email protected]		[email protected]		oIoU		mIoU
Method	Publication	val	test	val	test	val	test	val	test	val	test	val	test	val	test
LAVT [ 42 ]	CVPR 22	37.46	37.04	31.78	30.99	25.22	23.29	17.16	15.50	7.31	6.92	62.23	55.61	34.66	33.90
RMSIN [ 16 ]	CVPR 24	44.57	47.95	38.69	40.87	31.51	32.03	20.37	19.92	9.50	7.70	62.66	55.64	41.75	43.52
MagNet [ 8 ]	CVPR 24	36.36	35.90	30.76	29.92	23.58	23.29	16.88	15.07	8.34	6.82	62.83	55.78	33.73	33.27
FIANet [ 26 ]	TGRS 25	7.18	6.04	3.83	3.87	1.64	1.36	0.41	0.26	0.00	0.03	11.56	13.28	10.74	8.29
Aeroformer [ 47 ]	Arxiv 25	44.36	42.50	36.91	34.89	27.89	25.57	17.70	16.08	8.00	6.47	61.86	56.77	40.61	39.27

TABLE II: Performance comparison across methods on the NWPU-refer dataset. The best results are in bold.

Fig. 8: Qualitative comparison on representative NWPU-Refer queries.

Method	Publication	[email protected]		[email protected]		[email protected]		[email protected]		[email protected]		oIoU		mIoU
Method	Publication	val	test	val	test	val	test	val	test	val	test	val	test	val	test
RRN [ 37 ]	CVPR 18	51.09	51.07	42.47	42.11	33.04	32.77	20.80	21.57	6.14	6.37	66.53	66.43	46.06	45.64
CMSA [ 40 ]	CVPR 19	55.68	55.32	48.04	46.45	38.27	37.43	26.55	25.39	9.02	8.15	69.68	69.39	48.85	48.54
LSCM [ 48 ]	ECCV 20	57.12	56.02	48.04	46.25	37.87	37.70	26.37	25.28	7.93	8.27	69.28	69.05	50.36	49.92
CMPC [ 23 ]	CVPR 20	57.93	55.83	48.85	47.40	38.50	36.94	25.28	25.45	9.31	9.19	70.15	69.22	50.41	49.24
BRINet [ 24 ]	CVPR 20	58.79	56.90	49.54	48.77	39.65	39.12	28.21	27.03	9.19	8.73	70.73	69.88	51.14	49.65

TABLE III: Performance comparison across methods on the RRSIS-D dataset.

Fig. 9: Visualization results of MRSNet for Chinese referring segmentation. The left, middle, and right parts show single-object, multi-object, and non-object queries, respectively.

Setting	IFIM			HFIM	[email protected]	[email protected]	[email protected]	oIoU	mIoU
Setting	PSR	CSR	CMA	HFIM	[email protected]	[email protected]	[email protected]	oIoU	mIoU
Model1	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	21.57	12.51	4.35	48.04	36.82
Model2	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	23.18	16.21	6.16	52.10	40.08
Model3	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	14.98	9.45	3.54	31.64	25.21
Model4	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	27.06	17.02	6.63	55.91	41.22
MRSNet	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	30.90	20.47	8.54	57.04	43.60

TABLE IV: Ablation results on the NWPU-Refer test set.

Referring Image Segmentation

Referring image segmentation maps a natural-language description of an object or region to its pixels in an image.

See all 45 papers

Cross-Scale Sparse Attention for Referring Segmentation in Remote Sensing
Oct 24, 2026Yuxi Li, Lu Si, Xia Liao +4Referring Image Segmentation Remote Sensing Image Segmentation
Westlake Institute for Advanced Study, Hangzhou, Zhejiang, China · School of Engineering, Westlake University, Hangzhou, Zhejiang, China · Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, China +2
DiCoR: Decoupled Referent Disambiguation and Contour Recalibration for Efficient Referring Remote Sensing Image Segmentation
Aug 13, 2026Ziyang Gao, Zhizhuo Jiang, Jingjing Chang +5Remote Sensing Image Segmentation Referring Image Segmentation
School of Integrated Circuits, School of Information Science and Electronic Engineering, Shanghai Jiao Tong University, Shanghai 200240, China · College of Computer Science, Nankai University, Tianjin 300350, China · Department of Electronic Engineering, Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China +1
Image-Conditioned Instance Prompt Network for Referring Remote Sensing Image Segmentation
May 23, 2026Biaoyu Ren, Qingsheng Wang, Cun Xu +2Remote Sensing Image Segmentation Referring Image Segmentation
School of Computer Science, Northwestern Polytechnical University, Xi'an, China · School of Computer Science, Northwestern Polytechnical University, Xi’an, China · College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China +1

Oct 24, 2026J/K move · Enter open · S save

Cross-Scale Sparse Attention for Referring Segmentation in Remote Sensing

Yuxi Li, Lu Si, Xia Liao, Yidong Shen+3

Westlake Institute for Advanced Study, Hangzhou, Zhejiang, China · School of Engineering, Westlake University, Hangzhou, Zhejiang, China · Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, China+2

Referring Image Segmentation Remote Sensing Image Segmentation Multi-Scale Attention Sparse Attention Compositional Reasoning in VLMs

Open paper AskPDF arXiv

High-resolution remote sensing (RS) imagery provides fine spatial detail over large areas, making it essential for fine-grained scene understanding. Referring Remote Sensing Image Segmentation (RRSIS) aims to segment regions described by natural language, yet many existing methods rely on sequential alignment--fusion pipelines with limited joint modeling of cross-scale visual hierarchies and linguistic dependencies. Moreover, dense attention mechanisms struggle with the sparsity and wide spatial dispersion of RS targets, while current datasets offer limited structured evaluation of multi-target references, multi-hop relational reasoning, and multi-granularity descriptions, limiting both model development and evaluation. To address these challenges, we propose a Cross-Scale Sparse Attention Model (CSSAM), which aligns multiscale visual features with complex referring expressions through two key modules. The Cross-scale Modality-aligned Fusion Module (CMFM) performs unified cross-scale cross-modal fusion, while the Dynamic Multimodal Sparse Hybrid Attention (DMSHA) selectively attends to informative visual--textual tokens, reducing dense visual interactions. Furthermore, we introduce M

^{3}

RRSIS-D, a RRSIS dataset explicitly designed around three core reasoning dimensions: multi-target (M

^{1}

), reference-dependent multi-hop (M

^{2}

), and multi-granularity (M

^{3}

) expressions. Extensive experiments on M

^{3}

RRSIS-D and two public RRSIS benchmarks demonstrate that CSSAM achieves competitive performance across accuracy, fine-grained reasoning, and high-IoU localization metrics, validating its effectiveness for complex semantic reasoning in realistic RS scenarios. Code, dataset, and model weights will be released upon acceptance.

Abstract

Referring Image Segmentation

Explore similar work

Cross-Scale Sparse Attention for Referring Segmentation in Remote Sensing

DiCoR: Decoupled Referent Disambiguation and Contour Recalibration for Efficient Referring Remote Sensing Image Segmentation

Image-Conditioned Instance Prompt Network for Referring Remote Sensing Image Segmentation