HiP3R: Hierarchical Pose Graphs for Long-Horizon 3D Reconstruction

Zunnan Xu; Shangchen Zhou; Yihang Luo; Chen Change Loy

cs.CVOct 22, 2026

HiP3R: Hierarchical Pose Graphs for Long-Horizon 3D Reconstruction

Authors: Zunnan Xu, Shangchen Zhou, Yihang Luo, Chen Change Loy

Organizations: S-Lab, Nanyang Technological University · Imperial College London · ACE Robotics

Abstract

Feed-forward models accurately estimate camera motion and 3D geometry from short video windows. However, computational and memory costs prevent direct inference on sequences containing thousands of frames. Long videos must therefore be processed as overlapping chunks. This division produces independent local reconstructions with inconsistent coordinate frames and duplicate estimates of shared cameras. Achieving global consistency requires propagating constraints from adjacent overlaps and distant revisits across the sequence. This requirement naturally motivates pose graph optimization. However, the choice of graph level creates a trade-off. Chunk-level graphs efficiently resolve scale drift and close large loops. Yet, their rigid transformations cannot correct internal errors or conflicting frame predictions. Conversely, frame-level optimization offers local flexibility, but scaling it globally is computationally expensive and can break metric scale. HiP3R resolves this trade-off using a hierarchical pose graph. HiP3R first optimizes a robust chunk-level $\mathrm{Sim}(3)$ graph to establish the global layout and scale. It then holds this layout fixed and estimates one shared $\mathrm{SE}(3)$ camera per frame, turning overlapping window predictions into a single trajectory. Experiments across multiple benchmarks demonstrate the effectiveness of HiP3R in improving camera trajectories and 3D reconstruction. Our method applies to both chunk-based and streaming backends without additional training or fine-tuning. The codes are available at https://kkakkkka.github.io/HiP3R.

Figures & tables

Figure 1: Overlapping predictions recover local geometry in inconsistent frames. Coarse layout optimization corrects global placement and scale, but keeps each chunk’s internal cameras. Shared-camera trajectory refinement estimates one trajectory from predictions and within-chunk motion.

Figure 2: Overview of HiP3R . Frozen backends predict overlapping chunks. Stage I builds a chunk-level $\mathrm{Sim}(3)$ graph from overlap and loop constraints to recover layout and scale. Stage II estimates one shared $\mathrm{SE}(3)$ camera per frame under that layout.

Method	Mean ATE $\downarrow$	Mean RPE rot $\downarrow$	Mean RPE trans $\downarrow$
CUT3R ( Wang et al., 2025b )	44.894	4.469	1.4600
TTT3R ( Chen et al., 2026b )	23.775	5.077	0.8077
LoGeR ( Zhang et al., 2026a )	40.691	2.086	1.1035
VGGT ( Wang et al., 2025a )	20.914	3.159	1.6396
ZipMap ( Jin et al., 2026 )	28.230	2.792	0.9454
LingBot-MAP ( Chen et al., 2026a )	8.288	1.818	1.4152

Table 1: Trajectory evaluation across indoor and outdoor benchmarks. Values are dataset-macro averages of ATE (m), RPE rot ( ∘ ), and RPE trans (m). Bold marks the best result.

		ATE $\downarrow$
Backend	Method	Oxford	KITTI	VBR	Mean
VGGT-Omega	Baseline	$9.861$	$77.42$	$41.91$	$43.06$
	VGGT-Long (w/o loops)	$1.777$	$110.3$	$85.50$	$65.86$
	VGGT-Long (w/ loops)	$1.665$	$10.19$	$41.27$	$17.71$
	$+$ HiP3R (Ours)	1.579	9.730	22.44	11.25
VGGT	VGGT-Long (released)	$9.451$	$48.84$	$99.83$	$52.71$

Table 2: Trajectory comparison on KITTI, VBR, and Oxford, alongside the long-horizon mean.

Figure 3: Qualitative comparison between VGGT-Omega chunk placement and HiP3R .

Method	AUC@5 $\uparrow$	AUC@15 $\uparrow$	AUC@30 $\uparrow$	Chamfer $\downarrow$	F1 $\uparrow$
VGGT-Omega	29.77	50.24	63.22	0.0759	65.92
HiP3R (VGGT-Omega)	50.75	67.98	75.21	0.0272	91.26
LingBot-MAP	17.83	38.61	53.19	0.1423	61.13
HiP3R (LingBot-MAP)	22.01	51.35	66.90	0.1042	74.60

Table 3: Pose AUC and dense geometry on both backends.

Configuration	ATE $\downarrow$	Graph Build & Solve (s) $\downarrow$	Final solve (s)
Baseline	21.7751	12232.18	1404.54
$+$ Bounded sampling	21.7569	2004.20	1345.43
$+$ Fit weighting	21.7770	2077.49	1387.14
$+$ Progressive loops	21.5241	1477.59	688.02
Complete	17.4708	753.43	41.28

Table 4: Ablation of point sampling and progressive layout solver on extended sequences. Rows add components cumulatively. Appendix K details all test conditions.

	VGGT-Omega			LingBot-MAP
	ATE	RPE rot	RPE trans	ATE	RPE rot	RPE trans
Layout Only	4.9143	0.6412	0.0817	4.0517	1.1837	0.2126
Single	4.9079	0.6238	0.0695	4.0256	1.1131	0.1930
Unary	4.9067	0.6010	0.0674	4.0192	1.0871	0.1828
Motion	11.6776	0.5877	0.0619	6.5007	1.0464	0.1816
Shared	4.9061	0.5899	0.0622	4.0168	1.0549	0.1824

Table 5: Trajectory comparison on identical predictions and one optimized coarse layout.

Figure 4: Trajectory comparison on four long sequences with identical VGGT-Omega points.

		Tracking Error			Runtime (s)
Dataset	Stage	ATE $\downarrow$	RPE rot ( ∘ ) $\downarrow$	RPE trans $\downarrow$	Inference	Stage solve
KITTI ( $T{=}4541$ )	Baseline	$77.42$	$0.895$	$2.358$	$825.745$	–
	Layout Only	$9.750_{\scriptscriptstyle(-87\%)}$	$0.202_{\scriptscriptstyle(-77\%)}$	$0.131_{\scriptscriptstyle(-94\%)}$	$896.785$	$5.40$
	Full	$\mathbf{9.730}_{\scriptscriptstyle(-0.2\%)}$	$\mathbf{0.154}_{\scriptscriptstyle(-24\%)}$	$\mathbf{0.093}_{\scriptscriptstyle(-29\%)}$	$896.785$	$0.08$
VBR ( $T{\approx}12{,}383$ )	Baseline	$41.91$	$2.117$	$0.256$	$2283.438$	–
	Layout Only	$22.44_{\scriptscriptstyle(-46\%)}$	$0.617_{\scriptscriptstyle(-71\%)}$	$0.075_{\scriptscriptstyle(-71\%)}$	$2511.460$	$200.0$

Table 6: Stage-wise results on identical VGGT-Omega predictions. Layout Only is the direct coarse-layout output; Full adds shared-camera trajectory refinement on the same layout. Oxford and VBR timings are means per sequence. Inference times include all inference required for each output; solver times are reported per stage. Error subscripts give reductions relative to the preceding stage.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Refinement graph	ATE $\downarrow$	RPE rot $\downarrow$	RPE trans $\downarrow$	$\|\mathcal{K}\|$	$t_{\mathrm{f}}$ (s)
Overlap averaging	$9.735$	$0.202$	$0.131$	—	—
Uniform stride $10$	$9.728$	$0.158_{\scriptscriptstyle(-22\%)}$	$0.094_{\scriptscriptstyle(-28\%)}$	$605$	$0.055$
Motion stride $10$	$9.729$	$0.156_{\scriptscriptstyle(-23\%)}$	$0.094_{\scriptscriptstyle(-28\%)}$	$725$	$0.104$
Uniform stride $5$	$9.730$	$0.154_{\scriptscriptstyle(-24\%)}$	$0.093_{\scriptscriptstyle(-29\%)}$	$1059$	$0.075$

Appendix

Table 7: Keyframe subgraph on KITTI after the same coarse

A^{\star}

		Consensus		Uniform $s{=}10$		Uniform $s{=}5$
Scene	$T$	RPE rot	RPE trans	RPE rot	RPE trans	RPE rot	RPE trans
campus_train0	$12{,}042$	$0.204$	$0.129$	$0.122$	$0.118$	$0.119$	$0.117$
spagna_train0	$14{,}141$	$0.542$	$0.048$	$0.535$	$0.035$	$0.535$	$0.034$
downtown2	$2200$	$3.509$	$0.083$	$3.507$	$0.082$	$3.507$	$0.082$

Appendix

Table 8: The same keyframe comparison on additional long sequences.

		Dense $F_{t}$			Uniform $s{=}5$
Scene	$T$	ATE	RPE rot	RPE trans	ATE	RPE rot	RPE trans
Courthouse	$1106$	$2.599$	$10.82$	$0.517$	$2.600$	$10.81$	$0.517$
downtown1	$1427$	$3.223$	$3.940$	$0.0875$	$3.223$	$3.940$	$0.0876$
KITTI	$4541$	$9.731$	$0.144$	$0.086$	$9.730$	$0.154$	$0.093$

Appendix

Table 9: Dense fine graph versus uniform stride

5

on VGGT-Omega predictions and coarse

A^{\star}

Quantity	Value
VGGT-Omega default window	$C{=}60$ , $O{=}30$ frames
LingBot-MAP default window	$C{=}120$ , $O{=}32$ frames, windowed inference
Loop window	$20$ frames per endpoint
SALAD retrieval	similarity $0.85$ , top- $5$ , NMS $25$ frames, minimum chunk separation $1$ , one pair per chunk pair (first hit)
Loop-constraint addition	$4$ configured span-ordered stages, expanded to at most $8$ by adaptive continuation
Keyframe subgraph ( $T>2000$ )	uniform stride $5$ , all chunk endpoints, at least $\min(3,\|\mathcal{O}\|)$ frames in each overlap $\mathcal{O}$ , correction interpolation (Appendix C )

Appendix

Table 10: Solver hyperparameters and optimization settings across backends. We use the public configurations of VGGT-Omega ( Wang et al., 2026b ) , LingBot-MAP ( Chen et al., 2026a ) , and SALAD ( Izquierdo and Civera, 2024 ) .

Dataset	Domain	#Seq.	What it tests
7-Scenes	Indoor RGB-D	38	Textureless rooms, motion blur
ETH3D	Indoor and outdoor		Mixed-scale laser-scanned scenes
Neural RGB-D	Indoor RGB-D		Cluttered rooms, fine geometry
Tanks and Temples	Outdoor structures	6	Large objects and buildings
Oxford Spires	Campus indoor and outdoor	9	Loops and scale change ( $T{\approx}320$ )
KITTI seq00	Driving	1	Long outdoor loop ( $T{=}4541$ )

Appendix

Table 11: Dataset composition and experimental splits.

		Split
Method	Component	Oxford	KITTI seq00	VBR
		( $T{\approx}320$ )	( $T{=}4541$ )	( $T{\approx}12{,}383$ )
HiP3R (Ours)	Local baseline chunk inference ( $t_{\mathrm{g}}^{\mathrm{N}}$ )	$53.3$	$825.7$	$2283.4$
	Loop-window inference ( $t_{\mathrm{loop}}$ )	$1.6$	$71.0$	$228.0$
	CPU coarse + fine solves ( $t_{\mathrm{solve}}$ )	$0.05$	$5.5$	$200.2$
	Component sum	$55.0$	$902.3$	$2711.6$

Appendix

Table 12: Mean pipeline component times (s) on Oxford, KITTI, and VBR. For HiP3R , component sums include chunk and loop inference and coarse and fine solves.

Method	7S	ETH	KITTI	NRGBD	Oxford	T&T	VBR
VGGT-Omega ( Wang et al., 2026b )	0.0370	0.118	77.42	0.0190	9.861	0.518	41.914
$\pi^{3}$ ( Wang et al., 2025c )	0.0574	0.104	68.22	0.0244	12.06	0.467	87.269
GlueMap ( Pan et al., 2026 )	0.0624	0.810	193.5	0.0232	16.65	0.521	105.4
Scal3R ( Xie et al., 2026 )	0.0720	0.236	86.25	0.0471	4.573	0.572	101.535
VGGT-Long ( Deng et al., 2025 )	0.0743	0.204	48.84	0.0374	9.451	0.550	99.832
VGGT-Align ( Zhang et al., 2026b )	0.0731	0.209	69.80	0.0374	8.567	0.537	99.867

Appendix

Table 13: Per-dataset ATE on the seven public datasets (lower is better). NRGBD is Neural RGB-D, 7S is 7-Scenes, T&T is Tanks and Temples. Bold and underline mark the full-table best and second-best values in each column.

Method	7S	ETH	KITTI	NRGBD	Oxford	T&T	VBR
VGGT-Omega ( Wang et al., 2026b )	0.501	0.868	0.895	0.146	5.757	2.458	2.117
$\pi^{3}$ ( Wang et al., 2025c )	0.510	0.729	0.218	0.0999	6.912	2.036	0.636
GlueMap ( Pan et al., 2026 )	0.507	1.755	0.535	0.0984	4.337	2.308	1.388
Scal3R ( Xie et al., 2026 )	0.520	2.036	0.215	0.0957	2.193	1.985	2.589
VGGT-Long ( Deng et al., 2025 )	0.535	1.257	0.264	0.118	5.135	2.027	0.806
VGGT-Align ( Zhang et al., 2026b )	0.536	1.258	2.141	0.118	5.109	2.027	0.808

Appendix

Table 14: Per-dataset RPE rot on the same seven datasets (lower is better). Bold and underline mark the full-table best and second-best values in each column.

Method	7S	ETH	KITTI	NRGBD	Oxford	T&T	VBR
VGGT-Omega ( Wang et al., 2026b )	0.0172	0.101	2.358	0.0109	4.044	0.128	0.2557
$\pi^{3}$ ( Wang et al., 2025c )	0.0158	0.091	0.150	0.0091	1.999	0.106	0.1591
GlueMap ( Pan et al., 2026 )	0.0145	0.583	1.805	0.0079	2.268	0.115	8.599
Scal3R ( Xie et al., 2026 )	0.0176	0.238	0.335	0.0092	0.259	0.107	0.2260
VGGT-Long ( Deng et al., 2025 )	0.0188	0.190	0.221	0.0098	0.997	0.110	0.2292
VGGT-Align ( Zhang et al., 2026b )	0.0188	0.191	14.88	0.0098	1.053	0.110	0.2633

Appendix

Table 15: Per-dataset RPE trans on the same seven datasets (lower is better). Bold and underline mark the full-table best and second-best values in each column.

	Chamfer $\downarrow$		F1 $\uparrow$
Dataset	Baseline	HiP3R	Baseline	HiP3R
ETH3D	0.0339	0.0334	98.92	99.00
Neural RGB-D	0.1310	0.0205	40.53	90.05
7-Scenes	0.0739	0.0269	58.45	87.13

Appendix

Table 16: Dense 3D reconstruction metrics (Chamfer distance and F1 score) on VGGT-Omega versus HiP3R evaluated on datasets with dense 3D ground truth.

	Chamfer $\downarrow$		F1 $\uparrow$
Dataset	Baseline	HiP3R	Baseline	HiP3R
ETH3D	0.1295	0.1295	86.49	86.49
Neural RGB-D	0.1620	0.0675	42.66	65.24
7-Scenes	0.1403	0.1071	54.87	72.01

Appendix

Table 17: Dense 3D reconstruction metrics (Chamfer distance and F1 score) on LingBot-MAP versus HiP3R evaluated on datasets with dense 3D ground truth.

Dataset	VGGT-Omega ( Wang et al., 2026b )			LingBot-MAP ( Chen et al., 2026a )
	$C{/}O$	ATE	vs $60/30$	$C{/}O$	ATE	vs $120/32$
7-Scenes	$\mathbf{90/30}$	$0.0366$	$0.0381$	$\mathbf{120/8}$	$0.0801$	$0.0793$
ETH3D	$60/30$	$0.0546$	$0.0546$	$120/32$	$0.479$	$0.479$
KITTI	$\mathbf{60/16}$	$9.352$	$9.736$	$120/32$	$3.508$	$3.508$
NRGBD	$60/30$	$0.0219$	$0.0219$	$\mathbf{160/32}$	$0.0608$	$0.0776$
Oxford	$60/30$	$1.579$	$1.579$	$120/32$	$1.651$	$1.651$

Appendix

Table 18: Chunk and overlap comparison.

C/O

denotes chunk size and overlap size in frames. The “vs” columns report ATE with the indicated default sizes. Bold marks a change from Table 10 .

		Layout Only			Full
Sequence	$T$	ATE	RPE rot	RPE trans	ATE	RPE rot	RPE trans
KITTI seq00	$4541$	$9.750$	$0.202$	$0.131$	$9.730$	$0.154$	$0.093$
spagna_train0	$14{,}141$	$10.256$	$0.542$	$0.048$	$10.261$	$0.535$	$0.034$
campus_train1	$11{,}671$	$7.381$	$0.230$	$0.076$	$7.387$	$0.157$	$0.045$

Appendix

Table 19: Trajectory accuracy of Layout Only versus Full on the same feed-forward predictions.

Clean loop measurements
Configuration	ATE	RPE rot	RPE trans	Graph Build & Solve (s)	Final solve (s)
Baseline	21.7751	1.31094	0.11296	12232.18	1404.54
$+$ Bounded sampling	21.7569	1.31093	0.11280	2004.20	1345.43
$+$ Fit weighting	21.7770	1.31084	0.11262	2077.49	1387.14
$+$ Progressive loop scheduling	21.5241	1.31077	0.11070	1477.59	688.02
Complete	17.4708	1.30972	0.09834	753.43	41.28

Appendix

Table 20: Ablation of layout-solver components on extended sequences. Total time includes measurement construction and all layout solves. Final solve is the duration of the final least-squares call.

VGGT-Omega			LingBot-MAP
Output	Chamfer $\downarrow$	F1 $\uparrow$	Output	Chamfer $\downarrow$	F1 $\uparrow$
Native	0.0759	65.92	Native	0.1423	61.13
Layout Only	0.0273	91.20	Layout Only	0.1049	74.49
Full	0.0272	91.26	Full	0.1042	74.60

Appendix

Table 21: Dense geometry across optimization stages. F1 is reported in percent.

3D Reconstruction

3D reconstruction recovers a scene’s shape, appearance, and camera motion from images, video, depth, or other sensor measurements.

See all 957 papers

Think Locally, Refine Globally for Memory-Efficient 3D Reconstruction
Sep 18, 2026Jingke Zhou, Chenhang Ma, Zhizhou Zhong +63D Reconstruction Streaming 3D Reconstruction
Zhejiang University · Fudan University · Peking University +1
GeoWeaver: Accurate Long-Sequence 3D Reconstruction via Hierarchical Geometric Assembly
Aug 18, 2026Tinghao Jiang, Sheng Tang, Shengzhe Wei +43D Reconstruction Feed-Forward 3D Reconstruction
Kosmo Research · Shanghai Jiao Tong University · Tsinghua University
PAS3R: Pose-Adaptive Streaming 3D Reconstruction for Long Video Sequences
Mar 22, 2026Lanbo Xu, Liang Guo, Caigui Jiang +1Streaming 3D Reconstruction Monocular 3D Reconstruction
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, China · University of East Anglia, Norwich, NR47TJ, United Kingdom

Sep 18, 2026J/K move · Enter open · S save

Think Locally, Refine Globally for Memory-Efficient 3D Reconstruction

Jingke Zhou, Chenhang Ma, Zhizhou Zhong, Mingkai Liu+5

Zhejiang University · Fudan University · Peking University+1

3D Reconstruction Streaming 3D Reconstruction 3D ViTs Camera Pose Estimation

Open paper AskPDF arXiv

We propose LoG-VGGT, a memory-efficient framework for long-sequence 3D reconstruction that balances local temporal modeling with global camera consistency. Instead of relying on full global attention, our method introduces cross-window attention at a small subset of transformer blocks, enabling effective information propagation across adjacent temporal windows while keeping memory usage bounded. To mitigate long-term pose drift, we further design a global camera consistency refinement module, where camera tokens interact with compact register tokens via cross-attention to enforce scene-level constraints across the entire sequence. This design enables joint optimization of camera representations and significantly improves long-horizon pose stability without incurring the high cost of sequence-wide attention. Extensive experiments demonstrate that LoG-VGGT achieves improved depth accuracy and robust camera pose estimation across multiple long-sequence benchmarks, while delivering competitive streaming reconstruction performance.

Abstract

3D Reconstruction

Explore similar work

Think Locally, Refine Globally for Memory-Efficient 3D Reconstruction

GeoWeaver: Accurate Long-Sequence 3D Reconstruction via Hierarchical Geometric Assembly

PAS3R: Pose-Adaptive Streaming 3D Reconstruction for Long Video Sequences