VidTAG: Temporally Aligned Video to GPS Geolocalization with Denoising Sequence Prediction at a Global Scale
Organizations: Institute of Artificial Intelligence, University of Central Florida, USA · Amazon Prime Video Science, USA · Lule˚a Tekniska Universitet, Sweden
Abstract
The task of video geolocalization aims to determine the precise GPS coordinates of a video's origin and map its trajectory; with applications in forensics, social media, and exploration. Existing classification-based approaches operate at a coarse city-level granularity and fail to capture fine-grained details, while image retrieval methods are impractical on a global scale due to the need for extensive image galleries which are infeasible to compile. Comparatively, constructing a gallery of GPS coordinates is straightforward and inexpensive. We propose VidTAG, a dual-encoder framework that performs frame-to-GPS retrieval using both self-supervised and language-aligned features. To address temporal inconsistencies in video predictions, we introduce the TempGeo module, which aligns frame embeddings, and the GeoRefiner module, an encoder-decoder architecture that refines GPS features using the aligned frame embeddings. Evaluations on Mapillary (MSLS) and GAMa datasets demonstrate our model's ability to generate temporally consistent trajectories and outperform baselines, achieving a 20% improvement at the 1 km threshold over GeoCLIP. We also beat current State-of-the-Art by 25% on global coarse grained video geolocalization (CityGuessr68k). Our approach enables fine-grained video geolocalization and lays a strong foundation for future research. More details on the project webpage: https://parthpk.github.io/vidtag_webpage/
Figures & tables
| Model | Frame-wise Metrics | Video Metrics | ||||||||||
| Distance ( [%] @ km) | Median Distance Error(km) | Distance ( [%] @ km) | Median Distance Error(km) | DFD | MRD | |||||||
| Sub-street | Street | Locality | City | Sub-street | Street | Locality | City | |||||
| 0.5 km (500 m) | 1 km | 5 km | 25 km | 0.5 km (500 m) | 1 km | 5 km | 25 km | |||||
| GeoCLIP 54 -ZeroShot | 0.9 | 2.7 | 22.9 | 84.6 | 11.54 | 1.3 | 3.8 | 27.6 | 89.9 | 9.85 | 24.94 | 2.83 |
| PlaNet 59 | 4.4 | 10.5 | 38.4 | 81.0 | 7.38 | 3.9 | 8.6 | 35.6 | 78.9 | 7.66 | 67.42 | 9.86 |
| ISNs 32 | 4.9 | 11.2 | 38.6 | 84.3 | 7.22 | 4.3 | 10.1 | 36.0 | 80.4 | 7.51 | 62.55 | 8.95 |
| Model | Frame-wise Metrics | Video Metrics | ||||||||||
| Distance ( [%] @ km) | Median Distance Error(km) | Distance ( [%] @ km) | Median Distance Error(km) | DFD | MRD | |||||||
| Sub-street | Street | Locality | City | Sub-street | Street | Locality | City | |||||
| 0.5 km | 1 km | 5 km | 25 km | 0.5 km | 1 km | 5 km | 25 km | |||||
| GeoCLIP 54 -ZeroShot | 4.1 | 10.1 | 33.5 | 74.9 | 11.15 | 4.0 | 9.6 | 32.2 | 75.3 | 11.08 | 14.29 | 1.51 |
| PlaNet 59 | 7.7 | 15.4 | 44.1 | 84.5 | 6.26 | 6.3 | 14.3 | 43.5 | 86.4 | 6.21 | 43.19 | 6.76 |
| ISNs 32 | 8.2 | 16.3 | 44.8 | 86.1 | 6.11 | 7.0 | 14.9 | 45.1 | 87.2 | 6.06 | 40.07 | 6.14 |
| Model | City | State | Country | Continent |
|---|---|---|---|---|
| PlaNet 59 | 55.8 | 56.3 | 60.8 | 74.1 |
| ISNs 32 | 59.5 | 59.9 | 64.1 | 75.9 |
| GeoDecoder 5 | 64.2 | 64.5 | 69.5 | 79.9 |
| GeoReasoner 26 | 38.5 | 42.8 | 64.4 | 81.9 |
| Qwen2.5-VL 2 | 55.1 | 59.9 | 78.2 | 91.3 |
| OSV 1 | 23.1 | 33.5 | 55.6 | 78.1 |
| Ablation Component | Frame-wise Metrics | Video Metrics | |||||||||
| Distance ( [%] @ km) | Median Distance Error(km) | Distance ( [%] @ km) | Median Distance Error(km) | DFD | MRD | ||||||
| Backbone | TempGeo | GeoRefiner | Street | Locality | Street | Locality | |||||
| CLIP | DINOv2 | 1 km | 5 km | 1 km | 5 km | ||||||
| ✓ | ✗ | ✗ | ✗ | 22.5 | 63.0 | 2.97 | 18.6 | 58.2 | 3.69 | 22.52 | 2.82 |
| ✗ | ✓ | ✗ | ✗ | 26.4 | 70.1 | 2.13 | 24.7 | 69.4 | 2.32 | 9.15 | 1.03 |
| ✗ | ✓ | ✓ | ✗ | 30.2 | 72.5 | 1.91 | 30.5 | 72.3 | 1.99 | 3.01 | 0.41 |
| Resolution | Distance (a_r [%] @ km) | Median Distance Error(km) | DFD | MRD | ||
|---|---|---|---|---|---|---|
| Sub-street | Street | Locality | ||||
| (km) | 0.5 km (500 m) | 1 km | 5 km | |||
| 1 | 16.7 | 38.6 | 75.1 | 1.43 | 4.21 | 1.18 |
| 0.5 | 20.1 | 40.3 | 75.5 | 1.35 | 3.98 | 1.11 |
| 0.1 | 21.5 | 41.0 | 76.7 | 1.35 | 3.87 | 1.07 |
| Encoders | Distance ( [%] @ km) | Median Distance Error(km) | |||
|---|---|---|---|---|---|
| Sub-street | Street | Locality | City | ||
| 0.5 km (500 m) | 1 km | 5 km | 25 km | ||
| CLIP + SigLIP | 11.4 | 27.8 | 71.0 | 97.1 | 2.15 |
| CLIP + DINOv2 | 22.9 | 40.1 | 75.4 | 97.4 | 1.38 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| , | Distance ( [%] @ km) | Median Distance Error(km) | DFD | MRD | |||
|---|---|---|---|---|---|---|---|
| Sub-street | Street | Locality | City | ||||
| 0.5 km (500 m) | 1 km | 5 km | 25 km | ||||
| 1,1 | 21.5 | 41.3 | 75.5 | 97.4 | 1.36 | 5.21 | 1.41 |
| 10,1 | 21.5 | 41.0 | 76.7 | 97.9 | 1.35 | 3.87 | 1.07 |
| 100,1 | 21.4 | 40.9 | 75.4 | 97.2 | 1.38 | 7.63 | 1.57 |
| No. of MLP Layers | Distance ( [%] @ km) | Median Distance Error(km) | Trainable Parameters | |||
|---|---|---|---|---|---|---|
| Sub-street 0.5 km (500 m) | Street 1 km | Locality 5 km | City 25 km | |||
| 2 | 12.5 | 32.6 | 75.4 | 98.5 | 1.71 | 55.7M |
| 3 | 13.5 | 33.1 | 77.4 | 98.7 | 1.66 | 56.3M |
| 4 | 13.8 | 31.9 | 75.6 | 98.1 | 1.72 | 57.4M |
| 5 | 12.8 | 30.6 | 76.2 | 98.8 | 1.82 | 58M |
| values | Distance ( [%] @ km) | Median Distance Error(km) | |||||
|---|---|---|---|---|---|---|---|
| Sub-street 0.5 km (500 m) | Street 1 km | Locality 5 km | City 25 km | ||||
| 13.5 | 33.1 | 77.4 | 98.7 | 1.66 | |||
| 13.5 | 32.5 | 77.4 | 98.5 | 1.71 | |||
| 13.0 | 32.7 | 75.4 | 98.7 | 1.68 | |||
| 13.3 | 32.3 | 77.3 | 98.8 | 1.71 | |||
| 13.1 | 31.5 | 76.6 | 98.8 | 1.78 | |||
| Training Sequence Length (No. of frames) | Distance ( [%] @ km) | Median Distance Error(km) | |||
|---|---|---|---|---|---|
| Sub-street 0.5 km (500 m) | Street 1 km | Locality 5 km | City 25 km | ||
| 4 | 11.0 | 26.6 | 73.2 | 99.1 | 2.13 |
| 8 | 12.8 | 31.3 | 74.9 | 98.7 | 1.80 |
| 16 | 13.5 | 33.1 | 77.4 | 98.7 | 1.66 |
| Model | Uniform Grid Gallery Evaluation | Validation-set Gallery Evaluation | [%] @ 1 km | Median Distance | ||||||||
| Distance ( [%] @ km) | Median Distance Error(km) | Distance ( [%] @ km) | Median Distance Error(km) | |||||||||
| Sub-street | Street | Locality | City | Sub-street | Street | Locality | City | |||||
| 0.5 km (500 m) | 1 km | 5 km | 25 km | 0.5 km (500 m) | 1 km | 5 km | 25 km | |||||
| GeoCLIP-FineTuned | 8.3 | 22.5 | 63.0 | 93.9 | 2.97 | 19.9 | 31.1 | 66.3 | 94.1 | 2.30 | 8.6 | 0.67 |
| VidTAG (Ours) | 21.5 | 41.0 | 76.7 | 97.9 | 1.35 | 30.8 | 46.1 | 77.8 | 96.3 | 1.17 | 5.1 | 0.18 |
| Model | Mapillary(MSLS) | GAMa | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Distance ( [%] @ km) | Median Distance Error(km) | DFD | MRD | Distance ( [%] @ km) | Median Distance Error(km) | DFD | MRD | |||||||
| Sub-street | Street | Locality | City | Sub-street | Street | Locality | City | |||||||
| 0.5 km (500 m) | 1 km | 5 km | 25 km | 0.5 km (500 m) | 1 km | 5 km | 25 km | |||||||
| Dataset-specific Models | ||||||||||||||
| GeoCLIP-FineTuned | 8.3 | 22.5 | 63.0 | 93.9 | 2.97 | 22.52 | 2.82 | 16.3 | 28.3 | 57.8 | 89.1 | 3.28 | 6.50 | 0.50 |
| VidTAG | 21.5 | 41.0 | 76.7 | 97.9 | 1.35 | 3.87 | 1.07 | 35.4 | 53.1 | 77.8 | 94.4 | 0.88 | 0.39 | 0.17 |
| Model | City | State | Country | Continent |
|---|---|---|---|---|
| CityGuessr | 69.6 | 70.2 | 74.8 | 83.8 |
| VidTAG | 94.9 | 95.5 | 96.8 | 98.5 |
| VidTAG (Unified) | 79.0 | 80.2 | 89.1 | 95.0 |
| Model | Test GMACs | Train Time | Acc@1km |
|---|---|---|---|
| GeoCLIP-ZS | 81.91 | 272 hrs | 2.7 |
| GeoCLIP (FT) | 169 hrs | 22.5 | |
| GeoCLIP (total) | 441 hrs | 22.5 | |
| Ours | 83.47 | 68 hrs | 41.0 |
| with pre-computed backbone features | |||
| GeoCLIP-ZS | - | 2.5 hours | 2.7 |
Geospatial Representation Learning
Geospatial representation learning builds reusable embeddings of places, imagery, maps, mobility, and Earth data for spatial prediction and reasoning.
See all 133 papers