cs.CVMar 21, 2026
SavePiLoT: Neural Pixel-to-3D Registration for UAV-based Ego and Target Geo-localization
Organizations: National University of Defense Technology · Zhejiang University · Westlake University · Hangzhou Dianzi University
Abstract
We present PiLoT, a unified framework that tackles UAV-based ego and target geo-localization. Conventional approaches rely on decoupled pipelines that fuse GNSS and Visual-Inertial Odometry (VIO) for ego-pose estimation, and active sensors like laser rangefinders for target localization. However, these methods are susceptible to failure in GNSS-denied environments and incur substantial hardware costs and complexity. PiLoT breaks this paradigm by directly registering live video stream against a geo-referenced 3D map. To achieve robust, accurate, and real-time performance, we introduce three key contributions: 1) a Dual-Thread Engine that decouples map rendering from core localization thread, ensuring both low latency while maintaining drift-free accuracy; 2) a large-scale synthetic dataset with precise geometric annotations (camera pose, depth maps). This dataset enables the training of a lightweight network that generalizes in a zero-shot manner from simulation to real data; and 3) a Joint Neural-Guided Stochastic-Gradient Optimizer (JNGO) that achieves robust convergence even under aggressive motion. Evaluations on a comprehensive set of public and newly collected benchmarks show that PiLoT outperforms state-of-the-art methods while running over 25 FPS on NVIDIA Jetson Orin platform. Our code and dataset is available at: https://github.com/Choyaa/PiLoT.
Figures & tables
Figure 1: Overview of PiLoT. Our system takes a live video frame and a geo-referenced 3D map as input, and outputs 1) the UAV 6-DoF pose, visualized by the tight alignment in the AR overlays (bottom row), and 2) the 3D geo-coordinates of any target pixel, as shown in the dynamic target tracking example (left, filmstrip view). PiLoT achieves drift-free , real-time , and long-term ego and target geo-localization, demonstrated on a 10 km UAV trajectory with error color-coded (green: low, red: high). The system attains a median error of 1.37 m, a per-frame latency of 30 40 ms, and 100% success rate across day-to-night and cross-season variations without GNSS and IMU signals.
| Dataset | Img | Region | Alt (m) | Pitch ( ∘ ) | Type | Cond. | 6-DoF | Depth | Seq. |
| University-1652 [ 40 ] | 37.8k | 1,652 | - | varied | Synthetic | ✗ | ✗ | ✗ | ✗ |
| SUES-200 [ 43 ] | 24.1k | 200 | 150–300 | varied | Real | ✗ | ✗ | ✗ | ✓ |
| DenseUAV [ 8 ] | 27k | 14 | 80–100 | fixed | Real | ✓ | ✗ | ✗ | ✗ |
| UAV-VisLoc [ 37 ] | 6.7k | 11 | 400–2000 | fixed | Real | ✓ | ✗ | ✗ | ✓ |
| UAVD4L [ 36 ] | 6.8k | 1 | 50–300 | varied | Real | ✗ | ✓ | ✓ | ✗ |
| GAME4Loc [ 15 ] | 33.7k | 1 | 80–650 | varied | Synthetic | ✗ | ✓ | ✗ | ✗ |
Table 1: Comparison of UAV-oriented datasets. Legend: ✓ = yes, ✗ = no. Cond. = illumination/weather changes; 6-DoF = availability of ground-truth 6-DoF pose; Depth = availability of metric depth; Seq. = contains continuous video sequences.
Figure 2: PiLoT’s Dual-Thread Framework . We decouple rendering from localization into two parallel threads. A Render Thread generates synthetic views, while a concurrent Localization Thread registers the live frame against them to compute the pose, ensuring high-frequency accuracy.
Figure 3: Overview of the PiLoT framework and localization pipeline. (a) The overall pipeline inputs a query frame and outputs the UAV’s 6-DoF ego-pose along with the target’s 3-DoF geo-location. (b) A highly efficient one-to-many paradigm matches multiple query hypotheses against a single rendered reference view via feature alignment. (c) Our coarse-to-fine optimizer iteratively narrows the search space to converge on the optimal 6-DoF pose. (d) The final estimated trajectory demonstrates robust and drift-free sequential localization.
Figure 4: Overview of our synthetic data generation and its resulting zero-shot sim-to-real performance. From left to right: (a) realistic UAV trajectories rendered over geo-referenced 3D tiles in Cesium for Unreal; (b) multi-condition diversity across weather/time and viewpoint (in-plane yaw, out-of-plane pitch/yaw, planar translation , altitude ); (c) geometric consistency: we export absolute per-pixel depth and validate by reprojection; (d) our three-level feature pyramid on query (real) vs. reference (synthetic) images.
Figure 5: Rotation-aware sampling and coarse-to-fine optimization. The figure visualizes the pose convergence process in the pitch/yaw space: it synergizes wide-area Rotation-Aware Sampling (a) with parallel, coarse-to-fine refinement (b-d) to ensure robust convergence under aggressive motion.
Figure 6: Per-frame stability on synthetic trajectories. Two trajectories with per-frame translation and rotation errors, showing that PiLoT (green) sustains a lower error distribution with fewer catastrophic outliers.
| SynthCity-6 (Synthetic) | UAVScenes (Real) | UAVD4L-2yr (Real) | |||||||||||
| Method | FPS | Med m | Med ∘ | R@1/3/5 (m, ∘ ) | Comp. | Med m | Med ∘ | R@1/3/5 (m, ∘ ) | Comp. | Med m | Med ∘ | R@1/3/5 (m, ∘ ) | Comp. |
| Hybrid Methods | |||||||||||||
| Render2ORB | 20.0 | 18.0 / 30.2 / 55.3 | 5.7 / 63.5 / 78.4 | 29.1 / 70.9 / 89.1 | |||||||||
| Render2RAFT † | 10.0 | 8.4 / 32.5 / 48.9 | 4.2 / 45.8 / 62.3 | 22.4 / 62.6 / 77.4 | |||||||||
| Absolute Localizers | |||||||||||||
| PixLoc † | 65.4 / 92.3 / 98.2 | 12.8 / 94.6 / 97.5 | 98.4 | 31.5 / 68.9 / 87.4 | |||||||||
Table 2: Comprehensive localization performance on synthetic and real-world datasets. We compare PiLoT against baselines across three diverse datasets, all using a shared map and 512 px inputs. Median errors are reported in meters (m) and degrees ( ∘ ).
Figure 7: Qualitative results for joint UAV ego localization and dynamic target geo-localization on a UAVD4L-2yr sample sequence. (Top) Estimated UAV trajectories, where our method (PiLoT, green) most closely follows the ground truth (GT, black). (Middle) The position error of the dynamic target, with color indicating Euclidean distance to the GT.
| Dynamic Target Indication (R@1/3/5) | ||
| Method | Single-Target(Real) | Multi-Target(Syn) |
| Hybrid Methods (Abs.+Rel.) | ||
| Render2ORB | 72.13 / 84.59 / 89.74 | 79.51 / 91.04 / 93.28 |
| Render2RAFT | 44.15 / 78.96 / 88.19 | 51.33 / 78.12 / 90.50 |
| Absolute Localizers | ||
| PixLoc | 83.37 / 87.29 / 91.85 | 86.15 / 91.88 / 93.91 |
Table 3: Performance on the dynamic target geo-localization task. PiLoT’s superior ego-localization accuracy translates directly into state-of-the-art target geo-localization performance. We equip Render2loc with the LoFTR matcher, while PixLoc runs with its default 150 iterations.
| Recall (%) @ 1m/1 ∘ | |||
| Ablation Configuration | w/ 3m, 3 ∘ | w/ 5m, 5 ∘ | w/ 10m, 10 ∘ |
| System Components | |||
| Off-the-shelf Backbone | 4.2 | 0.0 | 0.0 |
| + Domain-Specific Training | 51.4 | 43.2 | 15.2 |
| + Rotation-Aware Hypothesis | 83.8 | 78.9 | 70.6 |
| + Motion Regularizer (Ours) | 84.3 | 84.3 | 84.2 |
Table 4: Ablation study on system components and training data. We evaluate the contribution of each core component and the effectiveness of our proposed training dataset.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Operation | Resolution | Channels | Description |
| Encoder (MobileOne-S0) | ||||
| Input | - | 3 | Input RGB image | |
| Stage 1 ( ) | Stem + Stage 1 | First level of features (for skip) | ||
| Stage 2 ( ) | Stage 2 Blocks | Second level of features (for skip) | ||
| Stage 3 ( ) | Stage 3 Blocks | Deepest features fed to decoder | ||
| Decoder | ||||
Table 5: Detailed data flow of our MobileOne-UNet feature extractor. The table illustrates the transformation of tensor shapes through the encoder, decoder, and output heads for an input image of resolution .
Figure 8: Comparison of optimization strategies in a non-convex landscape. While (a) Random Sampling is inefficient and (b) Gradient Descent is prone to local minima, our method (c) efficiently searches for the best solution. The convergence plot below illustrates how our method (green) achieves faster convergence and a lower final loss compared to alternatives.
Figure 9: Qualitative results of our multi-hypothesis refinement process. The top row shows a challenging rural scene with significant viewpoint change, while the bottom row demonstrates robustness in a day-night urban setting. The ‘convergence basin’ visualization on the query image (right) demonstrates the subsequent refinement.
| Metric | Yaw ( ) | Pitch ( ) | Roll ( ) | Tx | Ty | Tz |
| Conv. Limit | 3.5 ∘ | 2.7 ∘ | 3.2 ∘ | 5.6m | 8.1m | 7.4m |
| Max Motion (30 fps) | 6.7 ∘ | 3.0 ∘ | ¡0.1 ∘ | 0.7m | 0.7m | 0.3m |
Table 6: Comparison of 6-DoF convergence limits vs. maximum inter-frame motion (based on DJI specs at 30 fps).
Figure 10: Convergence basin evolution across levels.
| Image Size | ONNX Runtime | C++ Reference | CUDA Port | CUDA Fused Kernel |
| 512 512 | 14.9 ms | 8.44 ms | 5.72 ms | 0.49 ms |
| 256 256 | 10.4 ms | 8.35 ms | 5.24 ms | 0.47 ms |
| 128 128 | 8.8 ms | 8.27 ms | 5.21 ms | 0.44 ms |
Table 7: Latency Comparison for a Single LM Iteration. We compare different implementations for computing the cost, gradient, and Hessian. Our fused CUDA kernel dramatically reduces latency, achieving over 30x speedup compared to the initial ONNX-based implementation.
Figure 11: Timing diagram of the dual-thread synchronization. The pipeline maintains a fixed 2-frame lag, which is compensated by motion extrapolation to ensure real-time accuracy.
Figure 12: The AirSim-Cesium-Unreal Engine Simulator Interface. Our system integrates Unreal Engine (UE) for real-time rendering, Cesium to load Google 3D Tiles models, and AirSim to simulate UAV flight missions. This pipeline enables the synchronous acquisition of photo-realistic query images and accurate 6-DoF ground truth poses within large-scale, geo-referenced environments.
Figure 13: Dataset statistics for flight altitude (left) and environmental conditions with time of day (right).
Figure 14: Geometric Consistency Validation of Synthetic Data. Top: The Cumulative Distribution Function (CDF) of bidirectional relative depth errors across different altitudes (100m–800m). Bottom: Qualitative alignment results. The middle and bottom row illustrates the accurate mapping of co-visible regions across disparate perspectives.
Figure 15: Visualization of the designed ”barrel-roll” trajectory. The top-down view (top) shows yaw variation along the S-shaped path. The perspective views (bottom) illustrate changes in altitude (up-and-down arcs) and pitch (tilting of camera frustums).
Figure 16: Data collection setup and trajectory visualization. (Left) The hardware used for data acquisition: a DJI M4T drone for capturing query images and Qianxun RTK instruments for ground truth positioning. (Right) A top-down view of the UAV flight path (query trajectory, shown in orange) and the corresponding ground truth (target trajectory, shown in purple).
Figure 17: Sample checkerboard images used for intrinsic calibration.
Figure 18: Trajectory estimation results on Long Trajectory Flights . We compare our method’s estimated trajectory (orange) against the ground truth from an RTK-IMU system (black). The plots show high consistency for planar position (XY), altitude, pitch, and yaw. The checkerboard insets provide an AR visualization, which overlays the live camera views with the rendered views based on our estimated pose.
Figure 19: Trajectory estimation results on various synthetic scenes from the SynthCity-6 dataset. The figure showcases our method’s performance across diverse synthetic conditions, including night ( Switzerland-seq4, -seq7 ), cloudy ( Switzerland-seq12 ), sunny ( USA-seq2 ), and foggy ( USA-seq5 ).
Figure 20: Trajectory estimation results on challenging real-world and long-term scenarios. This figure demonstrates the model’s robustness by evaluating on: (1) a synthetic sunset scene from SynthCity-6 (USA-seq8) , and (2) 4 real-world sequences from the UAVD4L-2yr dataset, which include challenging day/night conditions ( seq2, seq3, seq6, seq8 ).
Figure 21: Generalization performance on the standard UAVScenes benchmark dataset. This figure validates our model’s strong generalization capability on four diverse, unseen scenes from the public UAVScenes benchmark. The scenes cover a wide range of environments: a town ( AMtown ), a natural valley ( AMvalley ), an airport ( HKairport ), and an island ( HKisland ).
Figure 22: Additional examples of the refinement process and its convergence basin. For various anchor points sampled in the reference image, the initial estimates in the query image (covering a wide area, shown as a field of arrows) are effectively guided by our learned features to converge to a single, precise location.
| Sequence | Weather / Illumination | Altitude (m) | Per Trajectory | Total Camera Frames | Location |
| Switzerland-seq4 | Foggy/Night/Rainy/Sunny/Sunset | 200/500 | 900 | 9000 | Thun, Switzerland |
| Switzerland-seq7 | Foggy/Night/Rainy/Sunny/Sunset | 200/500 | 900 | 9000 | Kirchlindach, Switzerland |
| Switzerland-seq12 | Foggy/Night/Rainy/Sunny/Sunset | 200/500 | 900 | 9000 | Lausanne, Switzerland |
| USA-seq2 | Foggy/Night/Cloudy/Sunny/Sunset | 200/500 | 900 | 9000 | Chicago, USA |
| USA-seq5 | Foggy/Night/Cloudy/Sunny/Sunset | 200/500 | 900 | 9000 | New York, USA |
| USA-seq8 | Foggy/Night/Cloudy/Sunny/Sunset | 200/500 | 900 | 9000 | California, USA |
Table 8: Summary of SynthCity-6 dataset. Each trajectory provides camera frames with synchronized 6-DoF poses under varying weather, illumination, and altitude conditions.
| Sequence | Illumination | Altitude (m) | Camera Frames | Scene Type & Environment | Target Object |
| UAVD4L-2yr-seq1 | Daytime | 70 | 900 | suburban | dynamic person |
| UAVD4L-2yr-seq2 | Daytime | 70 | 900 | urban | dynamic vehicles |
| UAVD4L-2yr-seq3 | Daytime | 70 | 900 | urban | static landmark |
| UAVD4L-2yr-seq4 | Daytime | 110 | 900 | suburban | dynamic vehicles |
| UAVD4L-2yr-seq5 | Night | 220 | 900 | urban | dynamic vehicles |
| UAVD4L-2yr-seq6 | Night | 310 | 900 | urban | dynamic vehicles |
Table 9: Summary of UAVD4L-2yr dataset. Each sequence contains camera frames captured under varying illumination and altitude , covering different scene types with annotated primary target objects .
| Sequence | Weather / Illumination | Altitude (m) | Camera Frames | Target Object Number |
| UAVD4L-SynTarget-seq1 | Sunny | 130 | 304 | 100 |
| UAVD4L-SynTarget-seq2 | Sunny | 130 | 344 | 100 |
| UAVD4L-SynTarget-seq3 | Sunny | 180 | 1136 | 100 |
| UAVD4L-SynTarget-seq4 | Foggy | 180 | 1136 | 100 |
| UAVD4L-SynTarget-seq5 | Night | 180 | 1136 | 100 |
| UAVD4L-SynTarget-seq6 | Foggy | 150 | 1192 | 100 |
Table 10: Summary of UAVD4L-SynTarget dataset. Each sequence contains camera frames captured under varying weather/illumination conditions and flight altitudes , with multiple target objects annotated along with their quantities.