Organizations: the BRain and Artificial INtelligence Lab (BRAIN LAB), School of Automation, Northwestern Polytechnical University, Xi’an, 710072, China · the Shenzhen Research Institute of Northwestern Polytechnical University, Shenzhen, 518057, China · the National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University, Beijing, 100871, China · the National Key Laboratory of Microwave Imaging Technology, Chinese Academy of Sciences, Beijing, 100090, China · the Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing, 100094, China
PolSAR data presents unique challenges due to its rich and complex characteristics. Existing data representations, such as complex-valued data, polarimetric features, and amplitude images, are widely used. However, these formats often face issues related to usability, interpretability, and data integrity. Most feature extraction networks for PolSAR are small, limiting their ability to capture features effectively. To address these issues, We propose the Polarimetric Scattering Mechanism-Informed SAM (PolSAM), an enhanced Segment Anything Model (SAM) that integrates domain-specific scattering characteristics and a novel prompt generation strategy. PolSAM introduces Microwave Vision Data (MVD), a lightweight and interpretable data representation derived from polarimetric decomposition and semantic correlations. We propose two key components: the Feature-Level Fusion Prompt (FFP), which fuses visual tokens from pseudo-colored SAR images and MVD to address modality incompatibility in the frozen SAM encoder, and the Semantic-Level Fusion Prompt (SFP), which refines sparse and dense segmentation prompts using semantic information. Experimental results on the PhySAR-Seg datasets demonstrate that PolSAM significantly outperforms existing SAM-based and multimodal fusion models, improving segmentation accuracy, reducing data storage, and accelerating inference time. The source code and datasets will be made publicly available at https://github.com/XAI4SAR/PolSAM.
Figures & tables
Fig. 1: (a) Different input representations of PolSAR data and their corresponding attributes, including lightweight (Lw), physical interpretability (Phys. i.), and information integrity (Inf. int.), are explored. (b) Scattering mechanisms depict polarization characteristics with physical interpretability and exhibit a strong correlation with semantic information. (c) Original SAM architecture; (d) Our PolSAM model.
Fig. 2: Data processing pipeline: unsupervised GD-Wishart classification, MVD generation through re-clustering of scattering mechanisms, and semantic annotation, illustrated with the PhySAR-Seg-1 dataset.
Fig. 3: Visualization of the PhySAR-Seg-1 dataset, including the pseudo-colored image, MVD, semantic label, MVD legend, and semantic legend. The white dashed lines on the pseudo-colored image divide the dataset into train, val, and test sets.
Fig. 4: Visualization of the PhySAR-Seg-2 dataset, including the pseudo-colored image, MVD, semantic label, MVD legend, and semantic legend. The white dashed lines on the pseudo-colored image divide the dataset into train and val sets.
Fig. 5: Overview of the proposed PolSAM. It consists of a novel data product, MVD, a Feature-Level Fusion Prompt (FFP) module and a progressive Semantic-Level Fusion Prompt (SFP) module. MVD is lightweight, encapsulates physical information, and correlates with semantic information. The FFP module combines data in a trainable manner for initial integration, while the SFP module generates sparse and dense prompts through a progressive two-level information interaction, guiding the segmentation task. By leveraging domain-specific knowledge of PolSAR data, PolSAM achieves enhanced segmentation performance with high data efficiency.
Fig. 6: Detailed design of module FFP.
Fig. 7: Detailed design of the proposed SFP: PS∈RN×C and PD∈RH×W×C represent sparse prompts and dense prompts, respectively.
Models
IoU Per Category (%)
Overall Metrics (%)
BK
Urban surfaces
Water
Building
Barren land
Vegetation
mAcc
mF1 score
mIoU
BER
HQ-SAM [ 47 ]
22.05
4.92
10.76
1.05
1.01
27.34
31.49
25.70
11.19
71.76
SAM-LST [ 46 ]
32.26
60.20
79.60
57.80
54.73
66.14
72.72
72.70
58.45
29.45
Personalize-SAM [ 43 ]
44.65
65.12
83.99
60.93
73.29
69.8
78.26
79.39
66.30
23.97
Mobile-SAM [ 44 ]
22.91
52.09
77.28
42.85
44.24
55.62
64.18
64.84
49.17
38.43
RSAM-Seg [ 53 ]
49.55
69.32
85.63
66.15
74.60
73.23
81.18
81.67
69.75
18.43
TABLE I: Comparison of PolSAM performance on the PhySAR-Seg-1 dataset: methods above the dashed line are SAM-based, and those below are multimodal. Metrics include per-category IoU and overall performance. Best results are in bold, second-best underlined, and complete PolSAM is marked in red. For BER, a lower value indicates better performance.
Models
IoU Per Category (%)
Overall Metrics (%)
BK
Montain
Vegetation
Developed
I-urban
H-urban
L-urban
Water
mAcc
mF1 score
mIoU
BER
HQ-SAM [ 47 ]
7.91
15.89
0.77
0.11
0.00
6.88
7.98
44.71
29.69
21.88
10.53
78.83
SAM-LST [ 46 ]
29.22
78.26
18.27
15.55
27.44
32.67
47.75
93.38
74.19
58.29
42.82
44.38
Personalize-SAM [ 43 ]
28.61
65.63
14.68
5.74
1.74
32.06
47.06
94.62
70.24
50.39
36.27
51.83
Mobile-SAM [ 44 ]
28.86
68.54
13.18
5.91
1.98
33.98
50.74
93.87
70.76
49.07
37.13
48.69
RSAM-Seg [ 53 ]
28.11
73.45
12.47
11.24
17.99
33.32
35.64
94.58
68.59
56.38
38.35
48.65
TABLE II: Comparison of PolSAM performance on the PhySAR-Seg-2 dataset. For BER, a lower value indicates better performance.
Fig. 8: Visualization of segmentation results on the PhySAR-Seg-1 (rows 1-3) and PhySAR-Seg-2 (rows 4-6) datasets: each row displays a randomly selected image from the test set, while each column shows results from different models.
Fig. 9: Visualization of feature activation maps and t-SNE distributions before (w/o) and after (w/) applying adapters. The adapters significantly enhance feature discriminability and semantic clustering, mitigating the modality gap.
TABLE III: Ablation studies on module design of PolSAM on PhySAR-Seg-1 and PhySAR-Seg-2. For BER, a lower value indicates better performance.
Fig. 10: (a) Visualization of labels, dense prompt embedding VD , and class-aware dense prompt embedding VS,D . (b) Visualization of class-specific fusion prompt embedding VS,Di , where i corresponds to channels labeled 1–6 in the figure, representing both the channel and category indices. Pseudo-colored images I1 , MVD I2 , and corresponding labels are shown above.
PhySAR-Seg-1 (%)
PhySAR-Seg-2 (%)
Strategy
Params
mAcc
mF1
mIoU
mAcc
mF1
mIoU
T (100%)
94.1M
79.51
80.07
67.51
75.55
57.21
43.40
T (75%)
71.5M
82.60
83.35
71.91
72.49
56.34
42.47
T (50%)
50.2M
82.78
83.60
72.42
74.73
59.18
45.10
T (25%)
28.9M
82.58
83.32
71.80
75.20
57.56
43.59
Ours (F+A)
14.0M
83.70
84.71
73.55
76.47
61.40
46.55
TABLE IV: Ablation Study on Encoder Fine-tuning Strategies
No. of SP
mAcc
mF1 score
mIoU
3
83.40
84.07
72.68
6 (classes)
83.70
84.71
73.55
2 * 6
82.34
83.12
71.42
4 * 6
83.43
84.29
73.01
10 * 6
83.07
83.78
72.25
20 * 6
83.37
84.28
73.06
TABLE V: Ablation study on the number of sparse prompts (SP)
Metrics
Inputs: H/A/Alpha( α ) [ 75 ] T [ 76 ]
(%)
None
H/A/ α
T(6)
T(9)
H/A/ α _T
MVD
BK
29.53
33.93
33.74
32.20
32.68
34.83
Montain
64.25
74.96
66.87
63.28
65.59
75.88
Vegetation
13.23
19.90
17.81
10.34
15.02
22.63
Developed
9.21
23.12
15.48
18.21
19.73
18.13
I-urban
0.01
15.12
15.57
19.66
11.93
30.76
TABLE VI: Ablation experiment on physical information input forms in the PhySAR-Seg-2 dataset
The College of Information Science and Technology, Beijing University of Chemical Technology, Beijing, China · The School of Information Science and Engineering, Yunnan University, Kunming, China · The Department of Signal Theory and Communications, Polytechnic University of Catalonia, Barcelona, Spain