Organizations: the BRain and Artificial INtelligence Lab (BRAIN LAB), School of Automation, Northwestern Polytechnical University, Xi’an, 710072, China · the Shenzhen Research Institute of Northwestern Polytechnical University, Shenzhen, 518057, China · the National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University, Beijing, 100871, China · the National Key Laboratory of Microwave Imaging Technology, Chinese Academy of Sciences, Beijing, 100090, China · the Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing, 100094, China
PolSAR data presents unique challenges due to its rich and complex characteristics. Existing data representations, such as complex-valued data, polarimetric features, and amplitude images, are widely used. However, these formats often face issues related to usability, interpretability, and data integrity. Most feature extraction networks for PolSAR are small, limiting their ability to capture features effectively. To address these issues, We propose the Polarimetric Scattering Mechanism-Informed SAM (PolSAM), an enhanced Segment Anything Model (SAM) that integrates domain-specific scattering characteristics and a novel prompt generation strategy. PolSAM introduces Microwave Vision Data (MVD), a lightweight and interpretable data representation derived from polarimetric decomposition and semantic correlations. We propose two key components: the Feature-Level Fusion Prompt (FFP), which fuses visual tokens from pseudo-colored SAR images and MVD to address modality incompatibility in the frozen SAM encoder, and the Semantic-Level Fusion Prompt (SFP), which refines sparse and dense segmentation prompts using semantic information. Experimental results on the PhySAR-Seg datasets demonstrate that PolSAM significantly outperforms existing SAM-based and multimodal fusion models, improving segmentation accuracy, reducing data storage, and accelerating inference time. The source code and datasets will be made publicly available at https://github.com/XAI4SAR/PolSAM.
Figures & tables
Fig. 1: (a) Different input representations of PolSAR data and their corresponding attributes, including lightweight (Lw), physical interpretability (Phys. i.), and information integrity (Inf. int.), are explored. (b) Scattering mechanisms depict polarization characteristics with physical interpretability and exhibit a strong correlation with semantic information. (c) Original SAM architecture; (d) Our PolSAM model.
Fig. 2: Data processing pipeline: unsupervised GD-Wishart classification, MVD generation through re-clustering of scattering mechanisms, and semantic annotation, illustrated with the PhySAR-Seg-1 dataset.
Fig. 3: Visualization of the PhySAR-Seg-1 dataset, including the pseudo-colored image, MVD, semantic label, MVD legend, and semantic legend. The white dashed lines on the pseudo-colored image divide the dataset into train, val, and test sets.
Fig. 4: Visualization of the PhySAR-Seg-2 dataset, including the pseudo-colored image, MVD, semantic label, MVD legend, and semantic legend. The white dashed lines on the pseudo-colored image divide the dataset into train and val sets.
Fig. 5: Overview of the proposed PolSAM. It consists of a novel data product, MVD, a Feature-Level Fusion Prompt (FFP) module and a progressive Semantic-Level Fusion Prompt (SFP) module. MVD is lightweight, encapsulates physical information, and correlates with semantic information. The FFP module combines data in a trainable manner for initial integration, while the SFP module generates sparse and dense prompts through a progressive two-level information interaction, guiding the segmentation task. By leveraging domain-specific knowledge of PolSAR data, PolSAM achieves enhanced segmentation performance with high data efficiency.
Fig. 6: Detailed design of module FFP.
Fig. 7: Detailed design of the proposed SFP: PS∈RN×C and PD∈RH×W×C represent sparse prompts and dense prompts, respectively.
Models
IoU Per Category (%)
Overall Metrics (%)
BK
Urban surfaces
Water
Building
Barren land
Vegetation
mAcc
mF1 score
mIoU
BER
HQ-SAM [ 47 ]
22.05
4.92
10.76
1.05
1.01
27.34
31.49
25.70
11.19
71.76
SAM-LST [ 46 ]
32.26
60.20
79.60
57.80
54.73
66.14
72.72
72.70
58.45
29.45
Personalize-SAM [ 43 ]
44.65
65.12
83.99
60.93
73.29
69.8
78.26
79.39
66.30
23.97
Mobile-SAM [ 44 ]
22.91
52.09
77.28
42.85
44.24
55.62
64.18
64.84
49.17
38.43
RSAM-Seg [ 53 ]
49.55
69.32
85.63
66.15
74.60
73.23
81.18
81.67
69.75
18.43
TABLE I: Comparison of PolSAM performance on the PhySAR-Seg-1 dataset: methods above the dashed line are SAM-based, and those below are multimodal. Metrics include per-category IoU and overall performance. Best results are in bold, second-best underlined, and complete PolSAM is marked in red. For BER, a lower value indicates better performance.
Models
IoU Per Category (%)
Overall Metrics (%)
BK
Montain
Vegetation
Developed
I-urban
H-urban
L-urban
Water
mAcc
mF1 score
mIoU
BER
HQ-SAM [ 47 ]
7.91
15.89
0.77
0.11
0.00
6.88
7.98
44.71
29.69
21.88
10.53
78.83
SAM-LST [ 46 ]
29.22
78.26
18.27
15.55
27.44
32.67
47.75
93.38
74.19
58.29
42.82
44.38
Personalize-SAM [ 43 ]
28.61
65.63
14.68
5.74
1.74
32.06
47.06
94.62
70.24
50.39
36.27
51.83
Mobile-SAM [ 44 ]
28.86
68.54
13.18
5.91
1.98
33.98
50.74
93.87
70.76
49.07
37.13
48.69
RSAM-Seg [ 53 ]
28.11
73.45
12.47
11.24
17.99
33.32
35.64
94.58
68.59
56.38
38.35
48.65
TABLE II: Comparison of PolSAM performance on the PhySAR-Seg-2 dataset. For BER, a lower value indicates better performance.
Fig. 8: Visualization of segmentation results on the PhySAR-Seg-1 (rows 1-3) and PhySAR-Seg-2 (rows 4-6) datasets: each row displays a randomly selected image from the test set, while each column shows results from different models.
Fig. 9: Visualization of feature activation maps and t-SNE distributions before (w/o) and after (w/) applying adapters. The adapters significantly enhance feature discriminability and semantic clustering, mitigating the modality gap.
TABLE III: Ablation studies on module design of PolSAM on PhySAR-Seg-1 and PhySAR-Seg-2. For BER, a lower value indicates better performance.
Fig. 10: (a) Visualization of labels, dense prompt embedding VD , and class-aware dense prompt embedding VS,D . (b) Visualization of class-specific fusion prompt embedding VS,Di , where i corresponds to channels labeled 1–6 in the figure, representing both the channel and category indices. Pseudo-colored images I1 , MVD I2 , and corresponding labels are shown above.
PhySAR-Seg-1 (%)
PhySAR-Seg-2 (%)
Strategy
Params
mAcc
mF1
mIoU
mAcc
mF1
mIoU
T (100%)
94.1M
79.51
80.07
67.51
75.55
57.21
43.40
T (75%)
71.5M
82.60
83.35
71.91
72.49
56.34
42.47
T (50%)
50.2M
82.78
83.60
72.42
74.73
59.18
45.10
T (25%)
28.9M
82.58
83.32
71.80
75.20
57.56
43.59
Ours (F+A)
14.0M
83.70
84.71
73.55
76.47
61.40
46.55
TABLE IV: Ablation Study on Encoder Fine-tuning Strategies
No. of SP
mAcc
mF1 score
mIoU
3
83.40
84.07
72.68
6 (classes)
83.70
84.71
73.55
2 * 6
82.34
83.12
71.42
4 * 6
83.43
84.29
73.01
10 * 6
83.07
83.78
72.25
20 * 6
83.37
84.28
73.06
TABLE V: Ablation study on the number of sparse prompts (SP)
Metrics
Inputs: H/A/Alpha( α ) [ 75 ] T [ 76 ]
(%)
None
H/A/ α
T(6)
T(9)
H/A/ α _T
MVD
BK
29.53
33.93
33.74
32.20
32.68
34.83
Montain
64.25
74.96
66.87
63.28
65.59
75.88
Vegetation
13.23
19.90
17.81
10.34
15.02
22.63
Developed
9.21
23.12
15.48
18.21
19.73
18.13
I-urban
0.01
15.12
15.57
19.66
11.93
30.76
TABLE VI: Ablation experiment on physical information input forms in the PhySAR-Seg-2 dataset
Polarimetric synthetic aperture radar (PolSAR) image classification is a representative task for physics-aware GeoAI, where land-cover semantics are closely coupled with electromagnetic scattering mechanisms. Many existing complex-valued networks can preserve amplitude-phase information, but they are often limited in long-range spatial dependency modeling and usually incorporate polarimetric priors only as input-level or shallow auxiliary features. As a result, physical knowledge is insufficiently used to guide deep feature evolution. To address this issue, this paper proposes CV-SSMNet, a physics-aware complex-valued state-space network with scattering-aware feature modulation for PolSAR image classification. The proposed method builds a complex-valued state-space model (CV-SSM) in the original complex domain to capture long-range spatial dependencies while preserving polarimetric amplitude-phase coupling. Meanwhile, seven physically meaningful scattering priors, are encoded as FiLM-style modulation signals to adaptively recalibrate complex-valued representations during feature evolution. CV-SSMNet further integrates multi-scale complex convolutions, branch-wise CV-SSM encoding, prior-guided recalibration, and lightweight global context aggregation, enabling physically guided representation learning from local scattering structures to global spatial context. Experiments on three L-band benchmark datasets and an additional P-band BIOMASS evaluation demonstrate that CV-SSMNet achieves competitive accuracy, improved regional consistency, and better boundary preservation, supporting the effectiveness of embedding polarimetric scattering mechanisms into complex-valued long-range GeoAI representation learning.
Fangyan Zhang, Fan Zhang, Shiqi Zhou +3
The College of Information Science and Technology, Beijing University of Chemical Technology, Beijing, China · The School of Information Science and Engineering, Yunnan University, Kunming, China · The Department of Signal Theory and Communications, Polytechnic University of Catalonia, Barcelona, Spain
This paper presents DDF2Pol, a lightweight dual-domain convolutional neural network for PolSAR image classification. The proposed architecture integrates two parallel feature extraction streams, one real-valued and one complex-valued, designed to capture complementary spatial and polarimetric information from PolSAR data. To further refine the extracted features, a depth-wise convolution layer is employed for spatial enhancement, followed by a coordinate attention mechanism to focus on the most informative regions. Experimental evaluations conducted on two benchmark datasets, Flevoland and San Francisco, demonstrate that DDF2Pol achieves superior classification performance while maintaining low model complexity. Specifically, it attains an Overall Accuracy (OA) of 98.16% on the Flevoland dataset and 96.12% on the San Francisco dataset, outperforming several state-of-the-art real- and complex-valued models. With only 91,371 parameters, DDF2Pol offers a practical and efficient solution for accurate PolSAR image analysis, even when training data is limited. The source code is publicly available at https://github.com/mqalkhatib/DDF2Pol
Mohammed Q. Alkhatib
College of Engineering and IT, University of Dubai, Dubai, 14143, UAE
Recently, convolutional neural networks (CNNs) have become popular for image classification due to their effectiveness in computer vision tasks. Now, researchers are exploring the potential of vision transformers (ViTs) in remote sensing and Earth observation. However, traditional Real-Valued networks often overlook important phase information in Complex-Valued (CV) data like polarimetric synthetic aperture radar (PolSAR) data. To address this, new CV deep architectures have emerged. HybridCVNet, a novel hybrid network, blends CV-CNN and CV vision transformer (CV-ViT) techniques. It efficiently combines CV 3D and 2D CNNs as feature extractors, enhancing PolSAR image classification by extracting complementary information and effectively leveraging interdependencies within the data. Experimental results from widely-used PolSAR datasets show HybridCVNet outperforms other methods, achieving an overall accuracy of 97.39% on the Flevoland dataset and showing promise even with just a 1% sampling ratio, with a Kappa value of 0.972 on the San Francisco dataset. Source code is accessible through https://github.com/mqalkhatib/HybridCVNet