GeoPilot: Diagnosis-Guided Interventions and Geometric Evaluation for Aerial Reconstruction
Authors: Pengyu Yin, Jie Pan, Junxiang Tan, Mengqi Fu, Kun Cheng, Yifan Liu
Organizations: Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China · School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China
When a mesh reconstructed from aerial remote-sensing imagery disagrees with reference geometry, deciding what to change requires a diagnosis of the reconstruction process. We present GeoPilot, a workflow that turns diagnostics and prior observations into executable program or camera-configuration interventions. A large language model proposes the change, numerical tools reconstruct the mesh, and an external evaluator measures its geometry under a stated coordinate and reference protocol. The main case tests a shared-intrinsics hypothesis proposed before the comparison. On 224 Metashape-raster development images, shared intrinsics improved all six native-coordinate measures and reduced mean surface-to-reference distance by 7.04%. Reference-assisted alignment fitted and evaluated on separate within-scene checkerboard samples reversed the mean-distance ranking, limiting that gain to native-frame delivery. A separate matched comparison on publisher-undistorted images favored fixed published cameras on all six measures, reducing mean distance by 12.11%; the cameras retain upstream image and laser-scanning adjustment information. With one reconstruction per configuration, these comparisons provide conditional evidence for camera handling. Three end-to-end runs each delivered a mesh and six-metric evaluation from all 327 images of a second development scene.
Figures & tables
Fig. 1: GeoPilot overview. Selected experience and tool descriptions inform diagnosis and executable reconstruction; numerical tools produce meshes for external evaluation. The lower strip reports the reduction in native-frame surface L1 for shared versus per-image intrinsics on Dataset-1 M (0.140 m, 7.04%). Reference-assisted alignment reverses that ordering (Section V-C), limiting a local-shape interpretation. Figure 2 details the two execution paths.
Fig. 2: Two GeoPilot execution paths. The outer research agent can revise a checked program under fixed tools or introduce a versioned configuration or adapter experiment. Both produce artifacts for external evaluation. The shared-intrinsics case follows the configuration path. Experience selection and optional semantic review operate in the host.
Surface
Recorded content
Execution check
Knowledge and experience
Sources, observations and applicability
Source identities and input compatibility
Diagnostics
Named measurements and artifact origins
Valid values and explicit unknown handling
Program structure
Calls, branches and stopping rules
Acyclic graph and stage requirements
Numerical arguments
Registered parameter values
Parameter types and bounds
Configurations and adapters
Camera settings, input bindings and source versions
Source identities and matched reconstruction settings
TABLE I: Intervention surfaces and execution checks.
Fig. 3: Input scenes and camera-center layouts for Dataset-1 U and Dataset-2 U. Each photograph is the middle image in filename order, selected independently of reconstruction scores. Maps show all supplied camera centers, identify the displayed photograph and outline the mesh XY bounds. These bounds are neither a visibility mask nor a ground footprint. The photographs are not overlaid on the maps. Source imagery: UseGeo [ 21 ] .
Comparison
Input and unit
Change
Readout
Camera A/B
Dataset-1 M; one mesh per arm
Per-image/shared intrinsics
Six metrics; proposal provenance; alignment sensitivity
S-U/F-U
Dataset-1 U; one mesh per arm
Estimated/fixed published cameras
Six metrics and image participation
P0/P1/P2
Dataset-1 M; shared historical P0
Restricted program edits
Complete vectors; retained parent
Experience
Dataset-2; three responses per condition
Direct, procedure, procedure plus experience
Nine recommendations and content checks
R0/RF/RJ
Dataset-2 U; one fresh mesh per arm
No added review/full-model/Jev review
Delivery, six metrics and complete elapsed time
TABLE II: Experimental design and measured outcome. M and U identify distinct raster inputs; all rows use development data.
Fig. 4: Camera-strategy control on identical U images and matches. F-U fixes published calibration and poses; S-U estimates cameras from images. Permitted XYZ centers support pair selection and georeferencing. Fresh downstream reconstruction uses common settings. Published cameras retain the upstream adjustment dependence described in Section III-D.
Fig. 5: Camera diagnostics, shared-intrinsics intervention and native-frame evaluation on Dataset-1 M. Focal lengths and center residuals describe the models after the intervention; the single B focal value follows from parameter sharing. Center residuals are paired by the same 224 image names after each model is fitted to the permitted camera centers. The diagonal denotes equal residuals, not an accuracy target. Surface L1 uses the principal evaluation of one reconstructed mesh per setting; no reconstruction-repeat interval is available. The timeline places the hypothesis before matched reconstruction. Figure 9 gives the separate alignment-sensitivity diagnostic.
Metric
Per-image A
Shared B
B − A
Surface L1, m ↓
1.9897
1.8497
-0.1400
Surface RMSE, m ↓
2.3925
2.2614
-0.1311
Best-90% L1, m ↓
1.7101
1.5688
-0.1413
Best-90% RMSE, m ↓
2.0185
1.8767
-0.1418
Precision at 0.20 m, % ↑
6.3999
7.8461
+1.4462 pp
Completeness at 0.20 m, % ↑
2.3646
3.0695
+0.7049 pp
TABLE III: Intrinsic-sharing control on Dataset-1 M. A uses per-image intrinsics and B shared intrinsics; one complete reconstruction per configuration, scored in native coordinates. pp: percentage points.
Metric
Estimated S-U
Fixed F-U
F-U − S-U
Surface L1, m ↓
0.2059
0.1809
-0.0249
Surface RMSE, m ↓
0.3359
0.3241
-0.0118
Best-90% L1, m ↓
0.1352
0.1087
-0.0264
Best-90% RMSE, m ↓
0.1743
0.1465
-0.0279
Precision at 0.20 m, % ↑
69.7435
75.6188
+5.8753 pp
Completeness at 0.20 m, % ↑
66.8045
72.3557
+5.5512 pp
TABLE IV: Camera-strategy control on Dataset-1 U. S-U estimates cameras and F-U fixes publisher cameras on identical undistorted images; one complete reconstruction per strategy. pp: percentage points.
Fig. 6: S-U and F-U mesh surfaces in a common top view and two matching local regions of interest (ROIs). All original triangles are rasterized with a depth buffer, identical view, scale and neutral lighting. The windows retain the same world-coordinate bounds in both products. Background pixels receive no visible triangle at the display resolution; the rendering does not measure completeness or local reference error. No surface repair, smoothing or geometric alignment is applied.
Fig. 7: Shared-intrinsics A/B products in common top views and one local window. Rows show global vertex projections, global distance maps, ROI vertex projections and ROI distance maps. Gray panels project all vertices with common lighting; cyan boxes mark the same window. Distance panels give mean stored 3D nearest-reference distance per 1 m XY cell on a common 0–5 m scale, with larger values saturated. Gray-blue cells indicate mesh samples without full-LiDAR support in the same XY cell, a measure of horizontal occupancy rather than visibility. White marks missing projected vertices or samples, which alone cannot identify surface holes. Samples are not pointwise paired. The native-frame distance samples use seed 20260925; Table III uses seed 20260916.
Fig. 8: Empirical cumulative distributions of native-frame nearest-reference distance for A and B over the whole mesh, inside the window of Figure 7 and outside it. Each full-scene curve uses one million area-uniform samples with seed 20260925; subset counts are shown in the panels. All distances enter the distributions; curves are thinned by rank for display at a resolution of at most 0.101 percentage points. A and B are sampled separately from one fixed mesh per configuration, without pointwise pairing; the curves describe spatial variation, not variation across repeated reconstructions.
Fig. 9: Fitted reference-assisted transforms reverse the native-frame A/B ordering. Bars show validation mean nearest-reference distance in meters for identity, translation, rigid and positive-scale similarity transforms. A uses per-image intrinsics; B shares intrinsics. Each mesh has 6,000 fitting and 12,000 validation samples drawn from alternate 20 m checkerboard cells; adjacent cells remain spatially related. These means describe within-scene alignment sensitivity; the principal scores use unmodified meshes, and reconstruction-repeat uncertainty is unavailable.
Context
Selected F-U
Valid responses
Direct analysis
3/3
3/3
GeoPilot procedure without experience
3/3
3/3
GeoPilot procedure with M/U experience
3/3
3/3
TABLE V: Dataset-2 U recommendation outcomes. Three fresh responses per context on the same task, with the same configured model and tools. Validity requires both configuration and content checks. These are recommendations without geometric scores.
Fig. 10: Delivered Dataset-2 U mesh from R0, reconstructed from all 327 input images using F-U. The mesh contains 9,924,029 triangles. The complete mesh is rendered in an orthographic oblique view, with two windows at fixed fractions of its projected extent and common neutral lighting. Window bounds and scale bars use projected view-plane meters. R0 is shown as the reference condition, independently of its appearance or score. This illustration establishes the delivered product, not a geometric advantage of any review condition.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Action
Parameter
Type
Allowed range
Densify
resolution-level
Integer
0–2
Densify
number-views
Integer
2–20
Densify
number-views-fuse
Integer
2–5
Densify
iters
Integer
1–6
Densify
geometric-iters
Integer
1–4
Densify
fusion-depth-diff-threshold
Number
0.001–0.05
Appendix
Table 16
Level
Available in the research release
Additional requirements or limits
Table recomputation
Complete vectors and original source identities for Tables III–IV and B1–B2
Recompute displayed differences from released measurements; complete original evaluation records remain in the local archive
Geometric rescoring
Evaluation utilities and protocol description
Original meshes, reference data and frozen runtime must be acquired separately
Numerical reconstruction
Selected source, numerical entry points and reproduction instructions
Raw rasters, compatible camera inputs, external binaries and experiment artifacts are required
Agent replay
Curated experience and selected workflow code
Complete historical agent interaction and execution records are not distributed; exact provider model snapshots are unavailable
Appendix
Table 17
Arm or attempt
Delivery status
Artifact validation
Complete six-metric result
Native-A, original attempt
Completed and exported
Validated mesh
Unavailable; evaluation interrupted before completion
Native-B, original attempt
Failed supervision after numerical calls
Original execution remains failed
Absent
Native-B, artifact diagnostic
Recovered product only; no new reconstruction
Separate diagnostic submission valid
Not started
Contemporary P0, original attempt
Failed during dense fusion; no mesh
No valid mesh
Not applicable
Contemporary P0, additional serial attempt
Completed with revised supervisor
Validated mesh
Not started
Appendix
Table 18
Review condition
F-G
F-T
J-G
J-T
Live calls
Nine questions, 4,096-token MiniMax response limit
7/9; 67.10
7/9; 58.17
7/9; 22.79
7/9; 19.81
41
Same nine questions, MiniMax client response limit removed
7/9; 73.51
8/9; 54.54
7/9; 14.85
7/9; 20.13
38
Two previously failed questions, additional execution evidence
1/2; 25.23
2/2; 18.08
1/2; 1.69
1/2; 2.30
8
Nine questions, clarified claim target
8/9; 85.67
7/9; 56.89
7/9; 8.34
7/9; 31.94
38
Same two questions, additional evidence and clarified target
2/2; 25.64
2/2; 7.68
1/2; 1.04
1/2; 1.18
8
Appendix
Table 19
Time or count
R0
RF
RJ
Complete episode, min
89.81
93.35
102.23
First valid delivery observed, min
85.41
89.66
91.76
Reconstruction, min
47.70
52.55
53.18
Geometric evaluation, min
32.30
30.26
32.55
Review and coordination, min
0.00
0.81
8.76
Time inside the review-call routine, s
0.000
4.371
1.494
Appendix
Table 20
Transform
A: per-image intrinsics
B: shared intrinsics
Identity
1.999156
1.848829
Translation
1.030263
1.100730
Rigid
1.011898
1.073042
Similarity
0.959510
1.032006
Appendix
Table 21
Arm
Valid deliveries
Fresh BA attempts
API calls
Provider tokens
Episode time, s
R
4/4
4
0
N/A
14.901
L
4/4
4
20
192,628
327.012
G0
3/4
4
24
236,572
319.271
GE
4/4
4
25
360,230
390.101
Appendix
Table 22
Study or component
Source location and records
P2 proposal and execution
out/geopilot-learning-20260923/proposal-format-replay/ : request.json , response.json , program.json , proposal.json , context.json , candidate.json ; p2-run/ , p2-score-r2/ , comparison-r2.json under the same dated directory
Fig. S1: The executed P0, P2, Native-A and Native-B programs use fixed paths, with parameter changes highlighted. The separate unexecuted diagnostic branch illustrates conditional graph syntax.
Fig. S2: Forward distances query the full LiDAR reference from area-uniform mesh samples; reverse completeness queries the triangle surface from the fixed refined reference. Precision and completeness use 0.20 m. Refined support follows the publisher-MVS-derived 5 m XY rule, rather than a verified visibility mask.
Fig. S3: Shared intrinsics improve all six native-frame measures on Dataset-1 M. Lines connect A (per-image) and B (shared); distance units are meters and threshold measures are percentages at 0.20 m. Axes use local ranges. Each configuration has one reconstruction, so reconstruction-repeat intervals are unavailable. Figure 9 shows the reference-assisted reversal of the mean-distance ordering.
Fig. S4: Fixed published cameras improve all six measures relative to estimated cameras on identical Dataset-1 U pixels. Lines connect S-U and F-U, with distances in meters and threshold measures in percent at 0.20 m. Axes use local ranges. Each configuration has one reconstruction; F-U retains the published cameras’ upstream adjustment dependence. M and U absolute scores concern different inputs.
Fig. S5: Agreement and summed review time in five semantic-review batches. F is MiniMax-M3, J is Jev with MiniMax fallback, and G/T denote generic/task-specific rubrics. Response limits refer to the MiniMax client. Denominators include failed and incomplete responses: nine queries in batches 1, 2 and 4, and two selected diagnostic queries in batches 3 and 5. Blue represents agreement fractions (0–1); orange represents seconds (0–90). Time includes preparation, calls, parsing, fallback and saving, and excludes shared loading, rubric/evidence construction, annotation and reconstruction/scoring. Batches are reported separately because repeated questions and selected follow-ups are correlated.
Fig. S6: Reconstruction and scoring dominate the complete elapsed times of 89.81, 93.35 and 102.23 min for R0, RF and RJ. Each bar represents one independently reconstructed and fully scored Dataset-2 F-U mesh. Stacks separate numerical reconstruction, full scoring, review and coordination intervals, and other host activity. Time inside the review-call routine is included in review and coordination. These single runs with uncontrolled provider, host and reconstruction variability do not establish a stable speed difference.
Reconstructing accurate 3D models of large-scale real-world scenes from unstructured, in-the-wild imagery remains a core challenge in computer vision, especially when the input views have little or no overlap. In such cases, existing reconstruction pipelines often produce multiple disconnected partial reconstructions or erroneously merge non-overlapping regions into overlapping geometry. In this work, we propose a framework that grounds each partial reconstruction to a complete reference model of the scene, enabling globally consistent alignment even in the absence of visual overlap. We obtain reference models from dense, geospatially accurate pseudo-synthetic renderings derived from Google Earth Studio. These renderings provide full scene coverage but differ substantially in appearance from real-world photographs. Our key insight is that, despite this significant domain gap, both domains share the same underlying scene semantics. We represent the reference model using 3D Gaussian Splatting, augmenting each Gaussian with semantic features, and formulate alignment as an inverse feature-based optimization scheme that estimates a global 6DoF pose and scale while keeping the reference model fixed. Furthermore, we introduce the WikiEarth dataset, which registers existing partial 3D reconstructions with pseudo-synthetic reference models. We demonstrate that our approach consistently improves global alignment when initialized with various classical and learning-based pipelines, while mitigating failure modes of state-of-the-art end-to-end models.
Pixel-level cross-view geo-registration aims to align a query image (e.g., drone) to a geo-referenced satellite map so that every query pixel can be mapped to real-world GPS coordinates. Despite strong progress in cross-view geo-localization, existing benchmarks largely provide only GPS labels, limiting evaluation to a single coordinate per image and leaving dense geodetic alignment underexplored. We introduce SkyReg, a dataset and standardized benchmark for pixel-level drone-to-satellite geo-registration, providing dense per-pixel geo-location supervision across diverse settings (orthographic and perspective), scene types (urban, landmark-centric, suburban/rural), and camera configurations. Using SkyReg, we evaluate a broad set of baselines spanning retrieval, feature matching, homography-based alignment, and feed-forward 3D reconstruction. Finally, cross-view pairs from SkyReg, we train a geometry-aware reconstruction pipeline that achieves state-of-the-art results,improving performance by a significant margin.
Qingyang Liu, David G Shatwell, Parth Parag Kulkarni +1
Institute of Artificial Intelligence, University of Central Florida, USA
Recovering metric 3D geometry from images is fundamentally ill-posed due to perspective projection ambiguity. Current geometry models incorporate supplied or predicted intrinsics during inference, but do not use them to supervise model adaptation. To refine these predictions, test-time adaptation (TTA) typically relies on external depth measurements. In this paper, we instead use camera intrinsics as an output-side geometric verifier. Comparing the point cloud's viewing rays with those defined by the known intrinsics yields a direct measure of geometric inconsistency. We propose IntriAdapt, a TTA framework that minimizes this closed-form geometric residual. By optimizing a lightweight LoRA adapter on a frozen geometry foundation model, our method aligns the prediction with the specific camera model without depth sensors or external priors. We further extend this approach to video sequences, demonstrating that adapting a short sequence generalizes to other sequences captured by the same camera. The adapted model processes held-out images in a feed-forward pass, without repeating optimization for each frame. Experiments on ten single-image and seven sequence benchmarks across multiple base models show that IntriAdapt consistently outperforms both unadapted and oracle-intrinsics baselines in metric accuracy and relative geometry.