Polygraph: Large-Scale Multi-Session Trajectory Optimization using Pose Graphs
Authors: Petar Lukovic, Tian Yi Lim, Marc Pollefeys, Pablo Speciale, Hermann Blum, Zuria Bauer
Organizations: ETH Zurich, Switzerland. · Microsoft Spatial AI Lab, Zurich, Switzerland. · University of Bonn and Lamarr Institute for Machine Learning and Artificial Intelligence, Germany.
Robots, vehicles, and wearable devices often traverse the same environment many times. Each pass produces its own trajectory from onboard visual or visual-inertial odometry. These trajectories are locally accurate but drift over time and live in unrelated coordinate frames. Turning many such independent trajectories into one globally consistent set is the large-scale multi-session trajectory optimization problem. Existing solutions largely work offline. All sessions must be available up front, one session is fixed as the reference frame, and a single fused map is then optimized. This paper presents Polygraph, a multi-session trajectory optimization pipeline that processes sessions online, one keyframe and one spatial cluster at a time, and never builds a 3D map. Keyframes are grouped into local clusters. When a cluster closes, it is matched against every other session through learned image retrieval. Candidate pairs are verified with sparse feature matching, monocular metric depth, and robust pose estimation. A two-tier consensus gate admits a cross-session edge into the shared pose graph only once independent evidence corroborates it. A cheap incremental solve keeps the graph current, and a periodic robust batch solve prunes outliers. We evaluate Polygraph on stereo outdoor driving, HoloLens and iPad indoor traversals, legged indoor locomotion with a Boston Dynamics Spot, and large-scale outdoor vehicle and handheld routes. Against the closest prior work on the same GPU model, using its reported timings, Polygraph runs faster, commits fewer loop-closure edges, and matches or improves on accuracy.
Figures & tables
Figure 1: Loop-closure visualisation . Five independent HYDRO sessions recorded with a Boston Dynamics Spot, each in its own color, placed into one shared frame by Polygraph and overlaid on the site’s 3D scan. Purple lines mark confirmed cross-session loop-closure edges. No two of these trajectories start out knowing that they share the same building.
Figure 2: Polygraph pipeline . Each session’s odometry is keyframed and grouped online into spatial clusters. A closed cluster is matched against every other session by image retrieval, candidate pairs are verified by joint pose estimation into one confirmed match, which enters the shared pose graph once the consensus gate admits it. A fast incremental solve runs after every closed cluster, and a periodic full solve prunes outliers.
Figure 3: Joint pose estimation . Turns one candidate cluster pair into one confirmed match. A spanning tree over descriptor similarity selects a few keyframe pairs (purple, one wrong pair in red). Each pair yields its own metric pose, which is lifted through the odometry into the common cluster frame. Estimates that agree within τagree form the consensus set, and the outlier is dropped. The chordal mean of the set is the measurement the gate decides on, and the agreeing pairs enter the graph as factors once the match is admitted.
Figure 4: The consensus gate . Same-session matches commit directly. A cross-session edge whose query session has no live cross-session edge yet must find an agreeing, independent peer in the corroboration pool. Once a session is connected, each edge is checked against its live estimate and queued for later re-checks if it fails.
ORB-SLAM3 odometry
VINS-Fusion odometry
Sequence
Length
Duration
odom
ORB-
Maplab
Maplab
2GO
Ours
Ours
odom
VINS-
Maplab
Maplab
2GO
Ours
Ours
only
SLAM3
BIN
SP
single
multi
only
PGO
BIN
SP
single
multi
KITTI-00
3.72 km
470 s
4.20
4.85
37.51
12.69
1.82
1.28
1.27
13.09
4.20
11.13
11.39
1.58
1.34
1.33
KITTI-02
5.07 km
483 s
8.05
5.64
18.01
44.78
8.04
5.30
5.30
21.02
13.03
23.96
18.29
10.58
7.96
7.96
KITTI-05
2.21 km
288 s
2.05
0.95
7.44
7.98
0.92
0.89
0.89
6.39
3.68
11.37
11.21
1.70
1.88
1.88
KITTI-06
1.23 km
114 s
2.05
0.79
7.09
10.74
0.62
0.72
0.72
3.54
1.68
13.27
14.76
1.48
1.36
1.36
Table I: RMSE ATE on KITTI’s loop sequences and on VBR . Refining ORB-SLAM3 (left) and VINS-Fusion (right) odometry [ 8 ] . Best per odometry source shaded and bold, second best underlined.
Dataset
Setting
Devices
Sess.
Multi-device
KITTI
outdoor, driving
car, stereo
7
no
VBR
driving, handheld
car / handheld
8
no
HYDRO
indoor
Spot, iPad
10
yes
HGE
indoor
HoloLens, iPad
9
yes
EuRoC
indoor, MAV
MAV, stereo(-IMU)
11
yes
Table II: Datasets used throughout Experiments . Multi-device marks a site captured by more than one platform. VBR’s multi-session sites include two campus sessions.
Figure 5: Single-session VBR visualisation . Two handheld and two vehicle. Top shows VINS-Fusion odometry; bottom shows Polygraph, both against ground truth in gray. Shades mark spatial clusters; dots mark clusters joined by a surviving loop closure.
Method
Split (robots)
Joint ATE (m)
RTF
Swarm-SLAM [ 6 ]
2
2.17
–
Ours, w/ RTAB-Map odom.
2
1.16
0.81
Ours, w/ ORB-SLAM3 odom.
2
1.22
0.74
Ours, w/ VINS-Fusion odom.
2
1.47
0.73
DOOR-SLAM [ 5 ]
3
8.00
–
Ours, w/ RTAB-Map odom.
3
1.26
0.95
Table III: KITTI multi-robot experiment . KITTI-00 cut into 2 or 3 equal parts, one robot each, against DOOR-SLAM and Swarm-SLAM on their own protocol, RTAB-Map stereo odometry for both. Best shaded and bold, second underlined, per split. RTF is conservative, run on a shared cluster.
ATE (m)
Anchoring, mixed
Device
odom only
multi-session
mixed
trans (m)
trans (%)
rot (deg)
HYDRO
Spot
Mean
0.299
0.149
0.148
0.181
0.20
0.39
L2
0.727
0.336
0.336
0.362
0.41
0.79
iOS
Mean
0.408
0.312
0.309
0.418
0.47
0.62
L2
0.990
0.722
0.721
0.963
1.08
1.42
HGE
HoloLens
Mean
0.658
0.280
0.281
0.956
0.57
0.84
Table IV: RMSE ATE on HYDRO [ 11 ] (5 Spot and 5 iOS sessions, footprint diagonal 89 m) and HGE [ 10 ] (5 HoloLens and 4 iOS sessions, footprint diagonal 167 m) . Each device on its own odometry, and reference-anchoring error with every session of a site placed into one shared frame. Pooled per device as mean and L2 norm. Best of multi-session and mixed shaded and bold, second best underlined.
Figure 6: Reference anchoring on HGE (top) and HYDRO (bottom) . One fixed transform per connected component applied to every session past its reference. Ground truth in green, Polygraph’s estimate on top. On HGE, (a) is HoloLens alone with two reference groups, (b) is iOS alone, (c) is HoloLens and iOS anchored together. On HYDRO, (a) is Spot alone, (b) is Spot and iOS together, all ten sessions in one component.
Dataset
Cameras
Single (m)
Multi (m)
Cross edges
HGE
1, baseline
0.652
0.280
252
4, best
0.652
0.278
506
HYDRO
1, baseline
0.236
0.149
1484
4, best
0.197
0.146
3672
KITTI
1, baseline
1.946
1.930
18
2, best
1.946
1.930
18
Table V: Camera-pairing ablation . Baseline is single-camera production; best is the best combination available. The better ATE per pair is shaded and bold.
Figure 7: Pose-estimation ablation . Mean wall time against mean ATE across KITTI, HGE, HGE iOS, and HYDRO, both raw values, so KITTI’s larger gaps keep their real weight instead of being normalized down to HGE’s. XLG = XFeat+LighterGlue (production), M3R = MASt3R, M3D = Metric3D depth, M3Rd = MASt3R’s own depth. XLG+M3D ties M3R+M3D on accuracy (2.20 m each) at 21% less wall time. XLG+M3Rd is faster still but 0.07 m worse, mostly one 0.3 m KITTI gap. MASt3R-native is off this plot at 4.26 m, nearly twice everyone else’s.
Method
Front-end
Edges
Trans. (m)
Rot. (deg)
COVINS-G PnP
VINS-Mono
852
0.69 ± 0.60
3.44 ± 3.5
COVINS-G 17-point
VINS-Mono
1823
0.10 ± 0.17
1.09 ± 1.25
COVINS-G PnP
ORB-SLAM3
505
0.23 ± 0.32
1.56 ± 3.7
COVINS-G 17-point
ORB-SLAM3
406
0.07 ± 0.12
1.01 ± 0.9
Ours, stereo-inertial
ORB-SLAM3
43
0.48 ± 0.29
1.61 ± 0.96
Ours, stereo
ORB-SLAM3
62
0.57 ± 0.41
2.31 ± 1.71
Table VI: EuRoC [ 13 ] . Accuracy of committed cross-session edges on MH01–05 before graph optimization, mean and standard deviation against ground truth, and COVINS-G values from their paper [ 33 ] .
Figure 8: Runtime and committed edges against 2GO . Ran on the same RTX 4090, VINS-Fusion odometry, log scale. 2GO values are from their paper; ours is the mean of five runs. ciampino’s speed config trims top_k_clusters / min_kf_pairs vs. production.
Submitted
Trans err (m)
Rot err (deg)
Dataset
On
Off
On
Off
On
Off
Exec Δ
HGE
250
373
0.399
2.574
0.87
12.41
+7.0%
HYDRO
775
839
0.726
1.223
0.90
1.78
+8.6%
Total
1025
1212
0.646
1.639
0.89
5.05
+8.2%
Table VII: Two-tier consensus gate ablation . Six HGE/HYDRO multi-session conditions pooled by dataset. Submitted are distinct cross-session candidates the gate admitted; trans/rot err is per committed edge before graph optimization. Exec Δ is gate-off vs. on wall time, run on an RTX 2080, not comparable in absolute terms to the pinned-4090 numbers elsewhere. Gate-on shaded and bold.
Trajectory optimization computes motion or control sequences that minimize costs while satisfying system dynamics, safety limits, and task constraints.
Tracking a novel object's 6D pose over long horizons currently requires either expensive onboarding or a reconstruction maintained throughout the sequence. This makes current trackers impractical for robotic manipulation and augmented reality, which need trackers that are ready to use and run in real time. We show that a lightweight tracking module can be applied on top of a wide range of correspondence estimation methods to keep drifts bounded while maintaining fast runtime. Our key idea is to avoid point-based optimization in the pose graph and operate only on relative pose constraints, which we weight by a derived uncertainty from the geometric alignment. This makes optimization independent of the number of correspondences while avoiding the direct inclusion of noisy point measurements, leading to fast and robust long-term tracking. Across four real-world benchmarks, our approach achieves tracking accuracy comparable to reconstruction-based trackers with a fraction of the optimization cost. Overall, these results suggest that a compact and reliable pose graph optimization can provide long-horizon consistency at substantially lower computational cost.
Xiaojie Zhang, Tom Fischer, Viktor Larsson +1
University of Technology Nuremberg · Lund University
Continuous 6-DoF pose estimation is essential for autonomous UAV operations. Yet, existing visual odometry and SLAM methods accumulate drift and yield only relative, up-to-scale trajectories. Single-frame geo-localization, in turn, discards temporal continuity and remains too slow for real-time use. We present OrthoTrack, a training-free system that estimates continuous 6-DoF UAV trajectories using only publicly available orthophotos and surface models as a map prior. OrthoTrack matches keyframes against the orthophoto and lifts correspondences to metric 3D via the surface model. It then propagates these map-anchored correspondences to intermediate frames with optical flow, producing absolute, metrically scaled poses at every frame without GPS or post-hoc alignment. We also introduce the MovingDrone Dataset, a large-scale benchmark pairing photorealistic UAV sequences with dense 6-DoF ground truth and co-registered multi-modal geodata including multi-temporal orthophotos. On MovingDrone and real-world benchmarks, OrthoTrack runs in real time on a single GPU. It outperforms all baselines by a large margin, even those receiving oracle scale and alignment. By relying on publicly available geodata, OrthoTrack enables deployment to new regions without site-specific adaptation.
Oussema Dhaouadi, Zuria Bauer, Johannes Michael Meier +3
ETH Zurich · TU Munich · University of Cambridge +2
Operating in previously visited environments is becoming increasingly crucial for autonomous systems, with direct applications in autonomous driving, surveying, and warehouse or household robotics. This repeated exposure to observing the same areas poses significant challenges for mapping and localization across sessions, particularly in deciding when a prior model is sufficient for reliable localization and when new mapping is required. In this work, we propose a novel multi-session framework that builds on map-based localization, in contrast to the common practice of greedily running full SLAM sessions and trying to find correspondences between the resulting maps. The core contribution is a principled, topology-driven mechanism to detect multi-session mapping needs from the pose-graph structure. Specifically, our approach uses spectral connectivity metrics on the joint pose-graph to identify disconnections and weakly constrained regions, and selectively triggers mapping and loop closing only when the pose-graph topology indicates insufficient structural support. The resulting map and pose-graph are seamlessly integrated into the existing model, reducing accumulated error and enhancing global consistency while avoiding redundant remapping. We validate our method on overlapping sequences from datasets and demonstrate its effectiveness in a real-world mine-like environment.
Lorenzo Montano-Olivan, Julio A. Placed, Luis Montano +1
Instituto Tecnológico de Aragón (ITA), María de Luna 7, Zaragoza, Spain