Hierarchical Adaptive Feature Refinement Network for VHR Remote Sensing Image Segmentation

Shuaishuai Cao; Meng Tang; Shuwei Peng; Xuan Liu; Min Huang; Jie Chen; Jiacheng Niu; Yong Chen

cs.CVAug 16, 2026

Hierarchical Adaptive Feature Refinement Network for VHR Remote Sensing Image Segmentation

Authors: Shuaishuai Cao, Meng Tang, Shuwei Peng, Xuan Liu, Min Huang, Jie Chen, Jiacheng Niu, Yong Chen,

Abstract

Semantic segmentation of very-high-resolution (VHR) remote sensing imagery increasingly benefits from strong pretrained hierarchical encoders, yet exploiting their multi-stage representations remains difficult. Nearby regions demand different balances between fine detail and semantic context, aggressive task-specific transformations perturb useful pretrained features, and conventional semantic supervision provides limited structural guidance. We present HAFR-Net, a progressive refinement framework that adaptively organizes and conservatively refines hierarchical representations instead of replacing them with a monolithic decoder transformation. Heterogeneity-Guided Stage-Adaptive Fusion (HG-SAF) predicts dense stage weights conditioned on local feature variation. A Frequency-Residual Adapter (FRA) then injects frequency information through a bounded, zero-initialized residual branch that keeps the fused representation as its reference. A Confusion-Aware Tri-Prior Decoder (CATP) finally regularizes the prediction with boundary, objectness, and training-derived class-relation cues. Under a matched Swin-B training and single-scale inference protocol, HAFR-Net attains 84.12%, 87.86%, 55.17%, and 67.70% mIoU on ISPRS Vaihingen, ISPRS Potsdam, LoveDA, and OpenEarthMap, improving the matched UPerNet baseline by 0.55, 0.95, 1.55, and 1.84 percentage points, respectively. Controlled analyses further show consistent spatial reweighting beyond content-only routing, improved boundary and thin-structure accuracy over matched spatial and spectral alternatives, and reduced confusion on pre-declared class pairs.

Figures & tables

Fig. 1: Positioning of HAFR-Net. (a)–(c) CNN, Transformer and state-space decoders combine the encoder hierarchy through a fixed aggregation rule. (d) HAFR-Net keeps the pretrained hierarchy and refines it progressively, through stage-specialized preparation, heterogeneity-guided stage-adaptive fusion, bounded residual frequency adaptation, and confusion-aware tri-prior decoding.

Fig. 2: Overall architecture of HAFR-Net. A pretrained Swin-B encoder produces four hierarchical stages, which pass fixed preparation blocks (SSDB at the two high-resolution stages, BCS-Mamba at the two low-resolution stages). HG-SAF then performs pixel-wise adaptive fusion, FRA applies a bounded residual frequency correction, SOE acts on the refined feature, and CATP produces the semantic prediction. Auxiliary heads are used only in training.

Fig. 3: Fixed stage preparation blocks, shown with the shared sub-blocks on the right. (a) SSDB decouples a normalized stage feature into a learnable complex spectral branch and a depthwise spatial branch, and merges them by a gated residual update. (b) BCS-Mamba scans rows and columns as independent selective state-space sequences, restores cross-axis context through two orthogonal depthwise bridges, and merges the two directions by content-adaptive weights. Both blocks are held identical across the module studies of Section VI ; their measured effect is reported in Section VI-E .

Fig. 4: The two refinement blocks after the encoder pyramid. (a) HG-SAF predicts per-pixel stage weights from $B$ and $\mathcal{H}$ with a zero-initialized gate. (b) FRA adds a channel-bottlenecked residual rFFT branch with a tanh-bounded gate and $\gamma{=}0$ .

Fig. 5: Confusion-aware tri-prior decoder. (a) The small-object enhancement block (SOE), a fixed block on the fused path rather than a claimed contribution. (b) Semantic, boundary, objectness and prototype-feature maps on a Vaihingen tile. The objectness head predicts a coarse foreground prior, which regularizes $\bar{F}$ without selecting objects by size.

Branch	Init.	First forward	Role
HG-SAF	last gate $=0$	uniform $1/4$ fusion	adaptive fusion
FRA	$\gamma=0$	exact identity	residual refine.
CATP fb.	$\beta_{e}{=}0.05$	near-ref. residual	struct. feedback

TABLE I: Conservative Initialization of the Three Refinement Branches

Method (Swin-B)	Vaih.	Pots.	LoveDA	OEM	Avg.
UPerNet [ 13 ]	83.57	86.91	53.62	65.86	72.49
FPN [ 12 ]	83.21	86.54	53.18	65.41	72.09
DeepLabv3+ [ 7 ]	83.38	86.72	53.35	65.58	72.26
SegFormer MLP [ 9 ]	83.44	86.80	53.48	65.71	72.36
UNetFormer dec. [ 14 ]	83.79	87.14	54.02	66.18	72.78
SE-gated fusion [ 31 ]	83.68	87.05	53.88	66.02	72.66

TABLE II: Controlled Comparison of Decoders Under a Matched Protocol

CNN-based
	ISPRS Vaihingen						ISPRS Potsdam
Method	Imp.S.	Build.	L.Veg.	Tree	Car	mIoU	Imp.S.	Build.	L.Veg.	Tree	Car	mIoU
UNet [ 5 ]	90.1	88.0	69.1	78.4	64.5	78.02	82.8	89.9	73.8	75.6	83.2	81.06
PSPNet [ 6 ]	91.0	88.7	69.5	79.1	66.8	79.02	84.2	91.0	74.8	76.5	85.2	82.34
DenseASPP [ 60 ]	90.3	87.6	68.4	77.9	62.5	77.34	83.3	90.3	73.5	75.3	83.8	81.24
MANet [ 28 ]	91.2	89.2	70.1	79.4	68.9	79.76	84.8	91.8	75.4	77.1	86.2	83.06

TABLE III: Contextual Comparison With Published Results on ISPRS Vaihingen and Potsdam

CNN-based
Method	Backgr.	Building	Road	Water	Barren	Forest	Agricult.	mIoU
UNet [ 5 ]	50.24	59.77	52.79	57.02	31.08	41.73	51.81	49.20
PSPNet [ 6 ]	52.50	59.46	51.15	61.10	27.46	37.48	50.32	48.49
DenseASPP [ 60 ]	50.77	55.67	53.08	55.49	23.99	35.24	43.54	45.40
MANet [ 28 ]	49.82	58.08	52.98	54.30	25.21	37.65	51.87	47.13
HRNet [ 25 ]	49.90	57.33	55.94	52.34	31.64	41.94	48.50	48.23

TABLE IV: Contextual Comparison With Published Results on the LoveDA Validation Split

CNN-based
Method	Bareland	Range.	Develop.	Road	Tree	Water	Agricult.	Building	mIoU
UNet [ 5 ]	33.6	49.2	49.8	56.1	64.7	71.4	70.9	72.3	58.50
PSPNet [ 6 ]	35.2	50.1	50.6	57.4	65.8	72.6	71.8	73.0	59.56
DenseASPP [ 60 ]	32.9	48.4	49.1	55.6	63.9	70.7	70.1	71.6	57.79
MANet [ 28 ]	36.0	50.8	51.2	58.0	66.5	73.4	72.5	73.8	60.27
HRNet [ 25 ]	35.6	50.5	50.9	57.7	66.1	73.0	72.1	73.4	59.91

TABLE V: Contextual Comparison With Published Results on OpenEarthMap

	Setting	Pixel-wise	Extra cue	mIoU	$\Delta$
Panel A: fusion-gate controls
H1	Mean fusion	–	–	84.28	$\downarrow$ 0.20
H2	Global-pool gate	–	GAP	84.32	$\downarrow$ 0.16
H3	Pixel gate, $B$ only	✓	–	84.34	$\downarrow$ 0.14
H4	Pixel gate $+$ input gradient	✓	$\lvert\nabla I\rvert$	84.40	$\downarrow$ 0.08
H5	Pixel gate $+$ entropy	✓	$\mathbb{H}(p)$	84.37	$\downarrow$ 0.11

TABLE VI: Analysis of Heterogeneity-Guided Stage-Adaptive Fusion on ISPRS Vaihingen

Fig. 6: Measured HG-SAF routing on ISPRS Vaihingen. (a) Input tile and (b) ground truth. (c) The heterogeneity statistic of ( 2 ) responds to object outlines and small structures. (d) Predicted shallow-stage weight $w_{1}{+}w_{2}$ of ( 3 ), with the colour range set to the $1$ – $99$ percentile of the tile so that the actual spatial variation is visible. (e) Pixel density of $w_{1}{+}w_{2}$ against $\mathcal{H}$ over the $113$ test patches, with the binned median; the rank correlation over all pixels is $\rho{=}0.10$ . (f) Mean stage weights per region, i.e. the quantities in Panel C of Table VI . The routing shift is consistent in sign across strata while deep stages keep the larger share everywhere.

Fig. 7: Illustrative HG-SAF fusion on eight tiles ( Input , shallow visual cues, deep semantic context, fused map). Quantitative routing statistics are in Table VI .

Variant	Par.	mIoU	BIoU	BF	Thin	Small
None (spatial only)	–	84.31	71.42	74.86	68.26	76.61
Spatial residual (matched)	1.1	84.22	71.78	75.14	68.58	76.84
Standard FFT filter	1.2	84.19	72.06	75.41	68.94	76.72
FFC-like branch [ 17 ]	3.4	84.08	71.64	75.02	68.41	76.38
GFNet mixer [ 16 ]	4.8	83.96	71.28	74.71	68.05	76.12
GFNet $+$ zero-init res.	4.8	84.28	72.54	75.88	69.42	77.05

TABLE VII: Matched Frequency-Module Alternatives on ISPRS Vaihingen

Panel A: matched auxiliary-loss controls (Vaihingen)
	Variant	mIoU	$\Delta$
S1	CE $+$ FocalDice only	84.26	$\downarrow$ 0.22
S2	$+$ boundary loss [ 22 ]	84.28	$\downarrow$ 0.20
S3	$+$ objectness loss	84.29	$\downarrow$ 0.19
S4	$+$ pixel contrast [ 23 ]	84.31	$\downarrow$ 0.17
S5	all three conventional losses	84.32	$\downarrow$ 0.16

TABLE VIII: Confusion-Aware Tri-Prior Decoder Analysis

Panel A: factorial combinations
HG-SAF	FRA	CATP	Vaih.	Pots.	LoveDA	OEM	Avg.
–	–	–	83.81	87.30	53.91	66.32	72.84
✓	–	–	84.04	87.65	54.46	66.85	73.25
–	✓	–	83.99	87.52	54.18	66.61	73.08
–	–	✓	84.19	87.78	54.58	66.94	73.37
✓	✓	–	84.26	87.93	54.81	67.27	73.57

TABLE IX: Component Complementarity and Initialization Study

	Preparation	mIoU	$\Delta$
T1	none (reference decoder)	83.57	–
T2	SSDB only	83.66	$+0.09$
T3	BCS-Mamba only	83.71	$+0.14$
T4	SOE only	83.64	$+0.07$
T5	SSDB $+$ BCS-Mamba $+$ SOE	83.79	$+0.22$

TABLE X: Individual Effect of the Fixed Preparation Blocks

Fig. 8: Representative ISPRS examples. Boxes mark vehicles and adjacent class boundaries discussed with the HG-SAF and CATP analyses.

Fig. 9: Representative LoveDA examples. Boxes mark thin rural roads and low-texture Barren regions.

Fig. 10: Representative OpenEarthMap examples. Boxes mark Rangeland $\leftrightarrow$ Agriculture transitions and thin roads.

Method	Par.	GFLOPs	Mem.	Lat.	FPS	mIoU
Swin-B $+$ FPN	90.1	102.6	1.72	18.9	52.9	72.09
Swin-B $+$ UPerNet	89.3	118.4	1.84	21.4	46.7	72.49
Swin-B $+$ UNetFormer dec.	92.6	124.1	1.91	22.6	44.2	72.78
Swin-B $+$ FFT mixer	93.8	129.7	1.98	23.5	42.6	72.56
HAFR-Net	97.8	131.2	2.12	24.8	40.3	73.71

TABLE XI: Efficiency of Five Re-Implemented Swin-B Decoders

Fig. 11: Published accuracy and cost, shown as context. Panel (a) gives the four-dataset mean mIoU of the methods present in all three contextual tables; for HAFR-Net this is the

73.89\%

mean, which uses TTA on the two ISPRS sets. Panel (b) plots the reported parameter count against that mean, with bubble area proportional to the reported GFLOPs. The entries come from different backbones and inference settings, so no frontier or optimality is drawn: the matched comparison is Table II and its measured cost Table XI .

Remote Sensing Image Segmentation

Remote sensing image segmentation labels land cover and objects in satellite, aerial, radar, and multispectral imagery.

See all 220 papers

May 25, 2026

SFR-Net: Learning Scale-Frustum Representations for Ultra-Wide Area Remote Sensing Image Segmentation

Pixel count and geographical coverage are two key characteristics of remote sensing images. Existing remote sensing image segmentation methods typically focus on images with either a small pixel count or a large pixel count but limited geographical coverage. In this paper, we introduce a novel segmentation task targeting ultra-wide area (UWA) remote sensing images, characterized by both a large pixel count and extremely wide geographical coverage. The core challenges of UWA segmentation lie in simultaneously handling ground objects with significantly varying scales and maintaining long-range contextual semantic continuity. To address these challenges, we propose the Scale-Frustum Representation Network (SFR-Net). Inspired by the viewing frustums of remote sensing images captured from different altitudes, we construct scale-frustum representations, enabling unified modeling of ground objects and contextual features at different scales. Furthermore, we design a cascaded cross-scale fusion mechanism to effectively integrate these representations, enhancing local semantic understanding while ensuring long-range contextual continuity. Experimental results on GID and FBPS demonstrate that SFR-Net achieves state-of-the-art performance, improving mIoU by 1.72% and 4.29%, respectively, over the strongest competing methods. In addition, the proposed scale-frustum representations can be integrated into generic segmentation networks to improve both segmentation accuracy and convergence speed. The implementation code will be publicly available at https://github.com/ChuyuZhong/SFR-Net.

Chuyu Zhong, Keyan Chen, Qinzhe Yang +3

Department of Aerospace Intelligent Science and Technology, School of Astronautics, Beihang University, Beijing 100191, China · Key Laboratory of Spacecraft Design Optimization and Dynamic Simulation Technologies, Ministry of Education, Beihang University, Beijing 100191, China · College of Computing and Data Science, Nanyang Technological University, Singapore +1

Remote Sensing Image Segmentation Multi-Scale Feature Fusion

Ask PDF

Apr 8, 2026

HQF-Net: A Hybrid Quantum-Classical Multi-Scale Fusion Network for Remote Sensing Image Segmentation

Remote sensing semantic segmentation requires models that can jointly capture fine spatial details and high-level semantic context across complex scenes. While classical encoder-decoder architectures such as U-Net remain strong baselines, they often struggle to fully exploit global semantics and structured feature interactions. In this work, we propose HQF-Net, a hybrid quantum-classical multi-scale fusion network for remote sensing image segmentation. HQF-Net integrates multi-scale semantic guidance from a frozen DINOv3 ViT-L/16 backbone with a customized U-Net architecture through a Deformable Multiscale Cross-Attention Fusion (DMCAF) module. To enhance feature refinement, the framework further introduces quantum-enhanced skip connections (QSkip) and a Quantum bottleneck with Mixture-of-Experts (QMoE), which combines complementary local, global, and directional quantum circuits within an adaptive routing mechanism. Experiments on three remote sensing benchmarks show consistent improvements with the proposed design. HQF-Net achieves 0.8568 mIoU and 96.87% overall accuracy on LandCover.ai, 71.82% mIoU on OpenEarthMap, and 55.28% mIoU with 99.37% overall accuracy on SeasoNet. An architectural ablation study further confirms the contribution of each major component. These results show that structured hybrid quantum-classical feature processing is a promising direction for improving remote sensing semantic segmentation under near-term quantum constraints.

Md Aminur Hossain, Ayush V. Patel, Siddhant Gole +2

Space Applications Centre, ISRO, India · Indian Institute of Technology Bombay

Remote Sensing Image Segmentation Multi-Scale Feature Fusion Hybrid Quantum-Classical ML Quantum Neural Networks

Ask PDF

Oct 6, 2025

Semantics-Aware Hierarchical Consensus Learning for Remote Sensing Image Classification

Deep learning has become increasingly important in remote sensing image classification due to its ability to extract semantic information from complex data. Classification tasks often include predefined label hierarchies that represent the semantic relationships among classes. However, these hierarchies are frequently overlooked, and most approaches focus only on fine-grained classification schemes. In this paper, we present a novel Semantics-Aware Hierarchical Consensus (SAHC) approach that integrates hierarchical-level-specific classification heads within a deep network architecture and combines their output through cross-level probability projectors. Direct and projected predictions are fused into a geometric consensus distribution, which is used for self-consistent training and optional hierarchy-aware inference. This mechanism acts as a geometric ensemble that leverages the inherent structure of the hierarchical classification task. The projectors are initialized from the user-defined taxonomy (i.e., the hierarchical label structure), and can be adaptively refined during optimization. The proposed SAHC method is evaluated on two benchmark datasets with different degrees of hierarchical complexity on different tasks, considering varying spectral and spatial resolutions. Experimental results show both the effectiveness of the proposed approach in guiding network learning and the robustness of the hierarchical consensus for remote sensing image classification tasks. The source code is available at https://github.com/rslab-unitrento/sahc.

Giulio Weikmann, Gianmarco Perantoni, Lorenzo Bruzzone

Department of Information Engineering and Computer Science, University of Trento, 38123 Trento, Italy

Hierarchical Classification Remote Sensing Remote Sensing Image Understanding Ensemble Learning

Ask PDF

Abstract

Remote Sensing Image Segmentation

Explore similar work

SFR-Net: Learning Scale-Frustum Representations for Ultra-Wide Area Remote Sensing Image Segmentation

HQF-Net: A Hybrid Quantum-Classical Multi-Scale Fusion Network for Remote Sensing Image Segmentation

Semantics-Aware Hierarchical Consensus Learning for Remote Sensing Image Classification