Component Pooling Trained Across Orientations for Few-Shot Remote Sensing Scene Classification
Authors: Bingchen Yan, Wenheng Guo, Yuye Zhou
Organizations: School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin, 541004, China · School of Information Engineering, Institute of Disaster Prevention, Langfang, 065201, China · School of Computing & Technology, Asia Pacific University of Technology & Innovation, Kuala Lumpur, 57000, Malaysia
Few-shot remote sensing scene classification (FSRSSC) aims to recognize novel scene categories from only a few labeled images. Most FSRSSC methods summarize the feature map of an image by global average pooling, which treats the few objects that define an aerial scene and the large background around them alike, and they rarely consider that aerial images have no canonical orientation. To address these issues, we propose RoCoP, a parameter-free component pooling network trained across orientations. RoCoP consists of a feature extractor, a non-negative decomposition module (NDM) and a component scoring module (CSM). NDM factorizes the local descriptors of each image into a few additive components with a deterministic initialization and a single multiplicative update, so that the positions of one object are grouped together. CSM weights each component by its activation magnitude and converts the weights into a soft mask for pooling. Because both modules are invariant to permutations of feature positions, the only rotation-sensitive part of the pipeline is the backbone, which we train to classify randomly rotated views of every image. Experiments on NWPU-RESISC45, AID and UC Merced show that component pooling and rotated-view training are complementary: together they improve a strong transfer-learning baseline by about 2.5 points in 5-way 1-shot accuracy on NWPU-RESISC45 and AID, without adding parameters at inference. Under an identical protocol, RoCoP outperforms re-implemented metric-based and task-adaptive methods, and NDM and CSM also improve existing methods when plugged into them. We further identify the regime in which the method does not apply, namely very small base sets.
Figures & tables
Figure 1. Overview of RoCoP, drawn with real NWPU-RESISC45 images and the outputs of the trained model. (a) Base training on base classes: an image and a rotated copy pass through a shared ResNet-12; NDM factorizes the 8×8×640 feature map into r=5 components ( A1 – A5 , overlaid on the image), CSM scores each component by its mean descriptor norm and scales the scores within the image, and the resulting mask M pools the descriptors for a linear classifier trained on both views. (b) Few-shot test on novel classes (5-way 1-shot shown): the frozen backbone, NDM and CSM encode the support images and the query (masks overlaid), and a logistic regression trained on the support vectors predicts the class of the query (bars: class probabilities; red: prediction). No parameters are added at test time.
Method
Backbone
NWPU-RESISC45
UC Merced
AID
Source
ProtoNet [ 8 ]
ResNet-12
64.52 / 81.95
53.15 / 71.54
67.08 / 82.44
re-impl. in [ 30 ]
DeepEMD [ 16 ]
ResNet-12
67.50 / 80.54
52.11 / 69.82
63.78 / 80.26
re-impl. in [ 30 ]
FRN [ 17 ]
ResNet-12
66.31 / 79.67
51.14 / 65.46
66.97 / 78.82
re-impl. in [ 30 ]
DeepBDC [ 21 ]
ResNet-12
68.84 / 84.12
54.20 / 71.94
66.56 / 84.15
re-impl. in [ 30 ]
PSM [ 20 ]
ResNet-12
66.17 / 83.03
54.35 / 71.86
68.89 / 83.75
re-impl. in [ 30 ]
PA-SRM
ResNet-12
72.65 / 83.64
60.79 / 78.64
68.75 / 81.47
as quoted in [ 42 ]
Table 1. 5-way accuracy (%) reported in the literature and ours (1-shot / 5-shot). The Source column gives the paper from which the numbers are taken: “re-impl. in” marks ResNet-12 re-implementations reported by [ 30 ] . Settings differ across papers; the table is a reference, not a ranking.
Method
NWPU-RESISC45
AID
ProtoNet [ 8 ]
66.12 / 82.05
69.54 / 84.21
ProtoNet + NDM + CSM (plug-in)
66.94 / 82.63
70.41 / 85.02
Δ (paired)
+0.82 / +0.58
+0.87 / +0.81
FEAT [ 18 ]
70.38 / 85.47
73.86 / 87.52
FRN [ 17 ]
68.74 / 84.36
72.15 / 86.28
DeepEMD [ 16 ]
67.43 / 82.71
71.08 / 85.34
Table 2. Same-protocol comparison (5-way test accuracy, %, 1-shot / 5-shot, one training seed, the same 2000 episodes). Δ : difference to the row above, the same episodes. ‡ Paired 95% interval excludes 0 (Sec. 4.4 ).
LR (default)
Proto.
Config
Pooling
Rot.
Mask loss
1-shot
5-shot
1-shot
5-shot
A
GAP
–
–
69.19 ± 0.46
85.44 ± 0.26
67.83 ± 0.47
84.45 ± 0.26
B
position-wise
–
–
70.25 ± 0.46
85.77 ± 0.26
69.11 ± 0.47
84.79 ± 0.27
C
NDM+CSM
–
–
69.85 ± 0.45
86.00 ± 0.25
68.58 ± 0.46
84.66 ± 0.26
D
GAP
✓
–
70.33 ± 0.45
86.30 ± 0.24
69.04 ± 0.47
85.21 ± 0.25
I
position-wise
✓
–
71.86 ± 0.45
86.80 ± 0.24
70.77 ± 0.47
85.89 ± 0.24
Table 3. NWPU-RESISC45 test classes, 5-way accuracy (%) ± 95% episode interval, one training seed, with logistic regression (LR) and with the mask-weighted prototype classifier (Proto.).
LR (default)
Proto.
Comparison
Question
1-shot
5-shot
1-shot
5-shot
E − A
RoCoP vs. GAP
+2.51 [+2.23, +2.79]
+1.73 [+1.56, +1.89]
+2.86 [+2.57, +3.14]
+1.67 [+1.49, +1.85]
D − A
rotated views, GAP
+1.14 [+0.91, +1.36]
+0.86 [+0.71, +1.01]
+1.21 [+0.98, +1.44]
+0.76 [+0.60, +0.91]
C − A
NDM+CSM, no rotated views
+0.66 [+0.38, +0.94]
+0.57 [+0.39, +0.74]
+0.75 [+0.47, +1.03]
+0.21 [+0.02, +0.39]
E − D
NDM+CSM, with rotated views
+1.37 [+1.09, +1.66]
+0.86 [+0.70, +1.03]
+1.65 [+1.36, +1.94]
+0.92 [+0.74, +1.09]
(E − D) − (C − A)
interaction
+0.72 [+0.39, +1.04]
+0.30 [+0.09, +0.50]
+0.90 [+0.57, +1.23]
+0.71 [+0.49, +0.92]
Table 4. Paired differences on the same 2000 NWPU-RESISC45 episodes (points, 95% interval).
Figure 2. Per-class 1-shot accuracy gain of RoCoP over GAP on the NWPU-RESISC45 test classes (2000 episodes, logistic regression). A real test image of each class is shown under its bar (one image per class, fixed random seed).
Table 5. Per-image versus support-level calibration of the weights on NWPU-RESISC45 (5-way test accuracy, %, 1-shot / 5-shot, same checkpoints and 2000 episodes). Δ : paired difference, support minus per image, with 95% interval.
NWPU-RESISC45 (17.5k base images)
AID ( ≈ 5k)
UC Merced (1k)
A: GAP
69.19 / 85.44
73.20 / 87.17
55.28 / 74.03
C: NDM+CSM
69.85 / 86.00
73.73 / 87.12
53.31 / 70.84
D: GAP + rotated views
70.33 / 86.30
74.39 / 88.07
56.32 / 73.79
I: position-wise + rotated views
71.86 / 86.80
74.87 / 88.10
–
E: RoCoP
71.70 / 87.16
75.73 / 88.58
53.70 / 72.31
E − A (RoCoP vs. GAP)
+2.51 / +1.73
+2.53 / +1.41
− 1.58 / − 1.72
Table 6. 5-way test accuracy (%, 1-shot / 5-shot) and paired differences (points) across datasets, one training seed, logistic regression unless stated. n.s.: 95% episode interval includes 0. ∗ : difference of the means; paired interval not computed.
Config
Rot.
Acc. drop
Vector cos.
Mask corr.
Comp. agree.
A: GAP
–
0.09 / 0.06
0.975 ± .001
–
–
C: NDM
–
0.17 / 0.11
0.968 ± .001
0.684 ± .007
0.588 ± .005
D: GAP
✓
0.07 / 0.01
0.989 ± .000
–
–
E: RoCoP
✓
− 0.04 / 0.03
0.983 ± .000
0.771 ± .006
0.663 ± .005
Table 7. Rotation behaviour on NWPU-RESISC45 test classes (1000 images × 3 rotations, 95% intervals). Drop: clean minus mean rotated accuracy, 1-shot / 5-shot.
Figure 3. Distribution over 1000 NWPU-RESISC45 test images of the correlation between the pooling mask of an image and the back-rotated mask of its 90 ∘ /180 ∘ /270 ∘ rotations (mean over the three rotations). Dashed lines: means.
r
1-shot
5-shot
Δ vs. r=5 , 1-shot
Δ vs. r=5 , 5-shot
3 †
71.28 ± 0.46
86.71 ± 0.24
− 0.42 [ − 0.65, − 0.19]
− 0.45 [ − 0.59, − 0.31]
4
71.50 ± 0.45
86.95 ± 0.24
− 0.20 [ − 0.43, 0.03]
− 0.21 [ − 0.35, − 0.07]
5
71.70 ± 0.45
87.16 ± 0.23
–
–
8
71.43 ± 0.45
86.75 ± 0.24
− 0.27 [ − 0.50, − 0.04]
− 0.41 [ − 0.55, − 0.28]
Table 8. Sensitivity to the number of components r (RoCoP, NWPU-RESISC45 test classes, 5-way, one training seed). Differences are paired over the same 2000 episodes. † Repeated run; the first run diverged (see text).
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 4. Pooling masks (per-image range) overlaid on NWPU-RESISC45 test images (one image per class, fixed random seed) for queries rotated by 0 ∘ /90 ∘ /180 ∘ /270 ∘ , each mask rotated back to the original frame. Left: NDM without rotated views (C). Right: RoCoP (E). Individual images vary; the population-level effect is quantified in Table 7 and Fig. 3 . The forest row shows a texture scene, where components are not separable.
School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China · CETC Key Laboratory of Aerospace Information Applications, Shijiazhuang, 050081, China