Attention-Guided Sequence Feature Fusion and Mixed-Space Metric Learning for Cross-View Image Sequence Geo-Localization
Authors: Nan Wu, Chunfang Yang, Jiangshan Li, Xiangyang Luo
Organizations: Information Engineering University, Zhengzhou, 450001, China, and the Henan Key Laboratory of Cyberspace Situation Awareness, Zhengzhou, 450001, China. · the College of Electronic Science and Technology, National University of Defense Technology, Changsha 410073, China.
Cross-view image sequence geo-localization aims to determine the geographic location of a sequence of ground images by matching it against geo-tagged satellite imagery. Although sequential observations provide complementary scene information, substantial overlap between adjacent images introduces feature redundancy, while conventional fusion strategies may weaken spatial relationships essential for cross-view matching. Therefore, we propose a cross-view image sequence geo-localization method via Attention-guided Sequence fusion and Metric Learning (ASML). Specifically, a Sequential Content Fusion (SCF) module models cross-image feature correlations to suppress redundant content and adaptively preserve geographically informative non-overlapping regions. Next, a Spatial Feature Aggregation (SFA) module is used to capture and aggregate the spatial and semantic information of local features, so as to maintain the spatial layout of the image sequence. Finally, a Geo-InfoNCE loss function is proposed, which comprehensively considers the feature distance and geographic distance for mixed-space metric learning, thus improving the ability of the model to distinguish hard negative samples. A new cross-view image sequence dataset CrossRoad-66K is established to provide more complex scenarios. Extensive experiments and visualizations on the SeqGeo and CrossRoad-66K datasets demonstrate that compared to state-of-the-art (SOTA) methods, the proposed method ASML achieves an improvement of 3.37% and 2.10% at R@1, respectively.
Figures & tables
Fig. 1: Quantitative and qualitative analysis of overlapping areas in cross-view image sequences. (a) The average ratio of overlapping areas among ground image sequences with different sequence lengths on the SeqGeo and CrossRoad-66K datasets. (b) Visualization of overlapping regions between adjacent ground images. This figure provides a dataset-level analysis of sequence redundancy rather than a performance comparison between different methods.
Fig. 2: A pair of samples in SeqGeo dataset. (a) shows the satellite image. (b)-(d) show the ground images captured along a straight road, whose locations are marked by green points in (a). (e)-(g) show the ground images captured around a corner, whose locations are marked by red points in (a).
Max Masks
Structural Similarity
KL Divergence
L2 Distance ↓
Cosine Similarity ↑
G2S( 10−5 ) ↓
S2G( 10−5 ) ↓
5
0.1033
0.0465
1.7548
1.7551
3
0.1019
0.0480
1.6820
1.6826
1
0.1021
0.0478
1.6935
1.6945
0
0.1024
0.0475
1.7083
1.7088
TABLE I: Comparative Analysis of Ground Image Sequence Feature with Satellite Image Feature Using Structural Similarity and KL Divergence. ↓ Indicates Lower Values with Greater Similarity and ↑ Indicates Higher Values with Greater Similarity.
Fig. 3: Overview architecture of ASML. Specifically, the Sequential Content Fusion Module (SCF) integrates the geographical semantics among the ground image sequence, and reduces the number of tokens in the subsequent processing. Spatial Feature Aggregation Module (SFA) mainly contains a Transformer Encoder, which is used to aggregate feature descriptors. The Geo-InfoNCE Loss is used to learn discriminative features in feature space and geographic space.
Fig. 4: The operational workflow of the Sequential Content Fusion Module (SCF). First, a similarity scores map Si is calculated between the adjacent local features. Next, an attention scores map Ai+1 is generated from similarity scores map, which is then translated to an attention mask Mi+1 . Finally, the attention mask is utilized to select the features in non-overlapping areas and eliminate redundant features.
Fig. 5: The main processing of the spatial feature aggregation module (SFA). (a) shows the workflow of SFA module. (b) shows the detailed structure of the Transformer Encoder in (a).
Fig. 6: Visualization of some hard negative samples in the SeqGeo dataset, which are easily confused. (a) shows one image within the ground image sequence. (b) shows the matched satellite image. (c) shows the unmatched satellite image. The left ground image and the right satellite image are a pair of difficult samples. Each image is labeled with the GPS coordinates of the capture location.
Fig. 7: The mixed distance metric optimization strategy for the samples within a batch. The samples matrix represents the relationship between the samples within a batch, where the elements on the diagonal are positive samples, while the others are negative samples. M refers to the feature distance matrix D minus the GPS distance matrix G .
Datasets
CVACT [ 42 ]
CVUSA [ 43 ]
VIGOR [ 44 ]
SeqGeo [ 20 ]
CrossRoad-66K
Satellite Images
44,416
44,416
90,618
38,863
21,320
Ground Images
44,416
44,416
105,214
118,549
66,384
Ground Images per Satellite Image
1
1
5
7
6
Seamless Sampling
×
×
✓
✓
✓
Sequential Ground Images
×
×
×
✓
✓
Full Panorama
✓
✓
✓
×
×
TABLE II: Comparison between the proposed CrossRoad-66K dataset and existing cross-view image datasets.
β
SeqGeo
CrossRoad-66K
R@1
R@5
R@1%
R@1
R@5
R@1%
8.3%
15.89%
34.61%
74.15%
4.57%
13.02%
38.30%
16.7%
21.53%
41.46%
79.04%
5.25%
15.75%
42.82%
25%
21.34%
41.69%
78.44%
4.99%
14.81%
41.76%
33.3%
19.97%
38.45%
76.37%
4.61%
14.24%
41.38%
Adap.
23.53%
44.30%
80.77%
6.20%
16.83%
43.94%
TABLE III: Hyperparameter tuning for the quantile threshold β .
Methods
SeqGeo
CrossRoad-66K
R@1
R@5
R@10
R@1%
R@1
R@5
R@10
R@1%
SeqGeo [ 20 ]
2.07%
8.12%
13.16%
40.10%
1.86%
7.69%
13.56%
35.65%
GeoDTR+ [ 45 ]
2.52%
7.29%
11.69%
36.85%
1.29%
3.89%
6.32%
17.93%
GeoDTR [ 46 ]
3.20%
9.03%
14.08%
39.98%
3.15%
9.14%
14.18%
27.33%
GAReT [ 19 ]
3.34%
11.19%
17.18%
44.39%
4.10%
14.24%
21.46%
40.48%
L2LTR [ 47 ]
3.78%
12.00%
18.21%
47.59%
2.53%
9.71%
15.74%
23.15%
TABLE IV: Comparison with existing methods. boldfaced texts denote the best results and underlined texts denote the suboptimal results.
Fig. 8: Visualization comparison of attention maps between the baseline method and ASML. (a) shows the satellite images, (b) presents the attention maps generated by the baseline method [ 33 ] , and (c) shows the attention maps produced by the proposed ASML. Compared with the baseline, ASML generates more concentrated responses on geographically discriminative regions while suppressing redundant regions, demonstrating its effectiveness in selecting informative features for cross-view matching.
Fig. 9: Qualitative comparison of ASML and other methods. (a) shows the first images of the query ground image sequences. (b) shows the ground-truth results. (c) shows the Top-3 satellite images returned from ASML. (d) shows the Top-3 satellite images returned from Sample4Geo. (e) shows the Top-3 satellite images returned from GAReT. The first and second rows show the geo-localization results of dense-building areas, the third and fourth rows show the results of sparse-building areas, and the last row shows the results of overpass-intersection areas. The satellite images marked with green boxes denote the ground-truth results, which provide a common reference for all compared methods. The images marked with red boxes represent incorrect retrieval results. The first and fourth rows are samples from the CrossRoad-66K dataset, while the rest are samples from the SeqGeo.
Ablations
SeqGeo
CrossRoad-66K
Fusion
Aggregation
Position Encoding
Loss Function
R@1
R@5
R@10
R@1%
R@1
R@5
R@10
R@1%
Concat
SFA
Compact
Geo-InfoNCE
3.95%
12.46%
19.29%
50.86%
2.24%
7.27%
11.94%
29.35%
Random Mask
SFA
Sparse
Geo-InfoNCE
20.61%
40.33%
49.21%
76.52%
5.01%
14.28%
21.12%
39.42%
SCF
GAP
Compact
Geo-InfoNCE
3.55%
12.18%
18.54%
50.08%
3.11%
10.90%
16.97%
38.89%
SCF
ViT-S
Compact
Geo-InfoNCE
12.24%
29.08%
38.79%
72.30%
3.18%
11.13%
16.81%
37.52%
SCF
SFA
Compact
Soft-Margin Triplet
13.58%
29.80%
38.17%
70.37%
1.69%
5.56%
9.79%
25.40%
TABLE V: Ablation study of each component on SeqGeo and CrossRoad-66K datasets.
Dataset
Selection Strategy
R@1
R@5
R@10
R@1%
SeqGeo
Random Mask
20.61%
40.33%
49.21%
76.52%
High-Similarity SCF
17.07%
35.85%
44.87%
72.77%
SCF
23.53%
44.30%
53.23%
80.77%
CrossRoad-66K
Random Mask
5.01%
14.28%
21.12%
39.42%
High-Similarity SCF
4.55%
14.99%
22.75%
41.41%
SCF
6.20%
16.83%
24.40%
43.94%
TABLE VI: Quantitative comparison of different feature-selection strategies on SeqGeo and CrossRoad-66K.
Fig. 10: Visualization of the attention masks on SeqGeo dataset. (a) shows the ground images. (b) shows the corresponding attention masks.
K
SeqGeo
CrossRoad-66K
R@1
R@5
R@1%
R@1
R@5
R@1%
2
12.45%
30.42%
69.45%
3.41%
10.73%
33.12%
3
15.86%
34.52%
74.07%
5.26%
14.61%
38.47%
4
19.11%
37.55%
76.29%
6.20%
16.83%
43.94%
5
23.53%
44.30%
80.77%
4.21%
12.48%
38.94%
6
20.49%
39.99%
77.62%
3.67%
11.23%
35.95%
TABLE VII: Ablation study on different numbers K of ground images in a sequence.
Methods
Architecture
Parameters
GFLOPs
R@1
L2LTR [ 47 ]
CNN+Transformer
195.9M
44.06
2.53%
TransGeo [ 33 ]
Transformer
44.9M
11.34
2.37%
Sample4Geo [ 37 ]
Transformer
87.6M
20.05
2.35%
SeqGeo [ 20 ]
CNN+Transformer
47.02M
10.30
1.86%
GeoDTR [ 46 ]
CNN+Transformer
48.5M
38.89
3.15%
FlexGeo [ 31 ]
Transformer
87.6M
140.14
3.82%
TABLE VIII: Comparison with existing methods on parameters, computational cost, and R@1 (on CrossRoad-66K dataset).
λg
SeqGeo
CVZY
R@1
R@5
R@1%
R@1
R@5
R@1%
0.25
18.11%
36.39%
73.85%
4.50%
13.61%
40.86
0.5
20.00%
39.60%
77.90%
5.47%
15.81%
44.65%
0.75
23.53%
44.30%
80.77%
6.20%
16.83%
43.94%
1
23.32%
44.04%
80.48%
6.11%
16.67%
43.92%
1.25
20.17%
39.47%
78.39%
5.75%
17.14%
44.27%
TABLE IX: Sensitivity analysis of varying scaling coefficient λg .
Fig. 11: Visualization of the last image in the ground image sequence and the satellite image. The red box in the left ground image represents irrelevant content between the ground image and the satellite image, while the points in the right satellite image represent the capture positions of each image in the ground image sequence.
Dataset
Scenario
Threshold τθ /Numbers
R@1
R@5
R@1%
SeqGeo
Corner
15∘ (264)
27.27%
46.21%
84.09%
20∘ (106)
17.93%
35.85%
84.91%
25∘ (64)
14.06%
31.25%
89.06%
30∘ (41)
17.07%
34.15%
90.24%
Straight
15∘ (7,508)
23.22%
43.33%
80.16%
20∘ (7,666)
23.42%
43.53%
80.22%
TABLE X: Comparison of corner-road and straight-road scenarios under varying trajectory-change thresholds τθ .
Fig. 12: Quantitative analysis of SCF mask sensitivity to controlled perturbations on the SeqGeo and CrossRoad-66K datasets. (a) Non-overlap Precision (NoP) measures the proportion of selected features located in geometrically non-overlapping regions. (b) Overlap False Selection Rate (O-FSR) measures the proportion of overlapping features incorrectly selected. (c) ROI Δ FSR measures the increase in false selection within the perturbed overlapping region relative to the clean input. (d) Aligned Mask IoU evaluates the spatial consistency between the clean and perturbed masks. Higher NoP and Mask IoU, and lower O-FSR and ROI Δ FSR indicate better selection performance and robustness.
Fig. 13: Visualization of module-specific failure cases. (a) shows the first images of the query ground image sequences. (b) shows the Top-3 results from ASML. (c) shows the Top-3 results from the version without SCF. (d) shows the ground-truth. (e) shows the Top-3 results from the version without SFA. (f) shows the Top-3 results from the version without Geo-InfoNCE. The green boxes indicate the ground-truth satellite images, while the red boxes denote incorrect retrieval results.
Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education of China, Xidian University · Department of Automation, Tsinghua University, Beijing 100084, China · School of Telecommunications, Xidian University, Xi’an 710071, China