MERG3R: A Divide-and-Conquer Approach to Large-Scale Neural Visual Geometry

Leo Kaixuan Cheng; Abdus Shaikh; Ruofan Liang; Zhijie Wu; Yushi Guan; Nandita Vijaykumar

cs.CVMar 2, 2026

MERG3R: A Divide-and-Conquer Approach to Large-Scale Neural Visual Geometry

Authors: Leo Kaixuan Cheng, Abdus Shaikh, Ruofan Liang, Zhijie Wu, Yushi Guan, Nandita Vijaykumar

Organizations: University of Toronto · Vector Institute

Abstract

Recent advancements in neural visual geometry, including transformer-based models such as VGGT and Pi3, have achieved impressive accuracy on 3D reconstruction tasks. However, their reliance on full attention makes them fundamentally limited by GPU memory capacity, preventing them from scaling to large, unordered image collections. We introduce MERG3R, a training-free divide-and-conquer framework that enables geometric foundation models to operate far beyond their native memory limits. MERG3R first reorders and partitions unordered images into overlapping, geometrically diverse subsets that can be reconstructed independently. It then merges the resulting local reconstructions through an efficient global alignment and confidence-weighted bundle adjustment procedure, producing a globally consistent 3D model. Our framework is model-agnostic and can be paired with existing neural geometry models. Across large-scale datasets, including 7-Scenes, NRGBD, Tanks & Temples, and Cambridge Landmarks, MERG3R consistently improves reconstruction accuracy, memory efficiency, and scalability, enabling high-quality reconstruction when the dataset exceeds memory capacity limits.

Figures & tables

Method	State-of-art accuracy		Memory scalability	Works for unordered
Method	Ordered	Unordered	Memory scalability	Works for unordered
Mast3R-SFM [ 10 ]	✗	✗	✗	✓
CUT3R [ 36 ]	✗	✗	✓	✓
TTT3R [ 6 ]	✗	✗	✓	✓
VGGT [ 35 ]	✗	✗	✗	✓
VGGT-Long [ 8 ]	✗	✗	✓	✗

Table 1: Conceptual comparison of methods across accuracy, memory scalability, and unordered-image compatibility. Our method can achieve superior or comparable accuracy over other SOTA baselines.

Figure 2: Overview of our large-scale 3D reconstruction pipeline. Given an unordered set of images, we first sort them into a pseudo-video sequence, then split the sequence into multiple interleaved subsets. Each subset is independently processed by a geometric foundation model to produce local pointmaps and poses. The resulting clusters are aligned into a common reference frame and jointly refined via global bundle adjustment, producing a coherent final reconstruction.

Figure 3: Illustration of the image partitioning process. Given an unordered set of $n$ images, we first compute the visual-similarity matrix $\mathbf{M}\in\mathbb{R}^{n\times n}$ and use it to search for a Hamiltonian path (shown in red) to produce the pseudo-video sequence. We then reorder the images using interleaved sampling and divide them into overlapping clusters.

Method	7-Scenes, 500 Images			7-Scenes, 1000 Images
Method	RRA@30 $\uparrow$	RTA@30 $\uparrow$	AUC@30 $\uparrow$	RRA@30 $\uparrow$	RTA@30 $\uparrow$	AUC@30 $\uparrow$
MAStR-SfM [ 10 ]	100	97.30	79.47	OOM	OOM	OOM
CUT3R [ 36 ]	75.89	40.16	38.82	60.47	30.50	14.11
TTT3R [ 6 ]	100	86.55	57.44	97.19	53.69	30.95
VGGT* [ 35 ]	100	96.87	81.13	OOM	OOM	OOM
FastVGGT [ 28 ]	100	96.75	80.59	OOM	OOM	OOM

Table 2: Camera pose estimation results on 7-Scenes.

Method	T&T			Cambridge Landmarks
Method	ATE $\downarrow$	RRE $\downarrow$	RTE $\downarrow$	ATE $\downarrow$	RRE $\downarrow$	RTE $\downarrow$
MAStR-SfM [ 10 ]	0.202	0.521	0.024	7.695	4.426	0.987
CUT3R [ 36 ]	1.575	1.217	0.087	16.645	4.436	1.212
TTT3R [ 6 ]	0.951	2.025	0.090	7.162	4.856	1.419
VGGT* [ 35 ]	0.535	1.498	0.071	0.793	5.728	1.085
FastVGGT [ 28 ]	0.549	1.400	0.072	0.812	4.198	1.095

Table 3: Camera pose estimation on the T&T and Cambridge Landmarks datasets.

Method	RRA@30 $\uparrow$	RTA@30 $\uparrow$	AUC@30 $\uparrow$	Time
GLOMAP [ 24 ]	100	96.67	81.38	10 min 34 s
InstantSfM [ 42 ]	100	95.55	76.32	8 min 56 s
Ours + $\pi^{3}$	100	97.61	83.31	4 min 48 s

Table 4: Camera pose estimation results with classical baselines on 7-Scenes (500 images).

Figure 4: Qualitative comparison of a predicted camera trajectory on a 300-image sequence from the Cambridge Landmarks dataset. Each subplot shows the estimated camera poses (red) overlaid with ground truth trajectories (green). Our method produces accurate and consistent trajectories, effectively handling long sequences with hundreds of frames.

Method	Stride	7-Scenes						NRGBD
		Acc $\downarrow$		Comp $\downarrow$		NC $\uparrow$		Acc $\downarrow$		Comp $\downarrow$		NC $\uparrow$
		Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.
CUT3R [ 36 ]	3/5	0.086	0.054	0.055	0.017	0.564	0.596	0.266	0.178	0.117	0.047	0.600	0.659
TTT3R [ 6 ]	3/5	0.137	0.090	0.072	0.027	0.554	0.581	0.769	0.504	0.089	0.046	0.702	0.796
VGGT ∗ [ 35 ]	3/5	0.019	0.008	0.028	0.011	0.607	0.660	0.028	0.018	0.018	0.007	0.710	0.816
FastVGGT [ 28 ]	3/5	0.014	0.006	0.028	0.010	0.631	0.705	0.024	0.013	0.018	0.010	0.665	0.791

Table 5: Point cloud estimation on 7-Scenes and NRGBD.

Figure 5: Qualitative comparison of 3D reconstructions on short (appox. 300–500 images) and long (approx. 1,000 images) sequences. Our method (Ours + Pi-3) produces sharper and more complete point clouds than CUT3R, TTT3R, and $\pi^{3}$ . Competing methods fail or run out of memory (OOM) on long sequences, while ours remains stable.

Figure 6: Qualitative comparison of 3D reconstructions on Zip-NERF [ 4 ] . For a fair comparison, the input to $\pi^{3}$ is subsampled to the same subset size with our method and the input ordering of sequence is the original one. Our method uses the same subset size as Pi-Long.

Panel of Figure 7 — Figure 7: Our method substantially reduces both runtime and memory consumption for base models processing long inputs.

Method	w/ real video seq.			w/ pseudo seq.
Method	ATE $\downarrow$	RRE $\downarrow$	RTE $\downarrow$	ATE $\downarrow$	RRE $\downarrow$	RTE $\downarrow$
VGGT [ 35 ]	0.521	1.317	0.544	0.522	1.563	0.061
FastVGGT [ 28 ]	0.525	1.364	0.056	0.526	1.598	0.079
$\pi^{3}$ [ 38 ]	0.077	0.196	0.014	0.077	0.178	0.013

Table 6: Effect of the sequential input vs. unordered input.

Splitting Method	ATE $\downarrow$	RRE $\downarrow$	RTE $\downarrow$
Graph clustering	0.319	1.280	0.060
Sliding window	0.106	0.492	0.024
Ours	0.077	0.178	0.013

Table 7: Effect of the image set splitting strategy for camera pose accuracy.

Subset Size						Overlap Size
Split Size	ATE $\downarrow$	RRE $\downarrow$	RTE $\downarrow$	Time (s)	Mem. (GB)	Overlap	ATE $\downarrow$	RRE $\downarrow$	RTE $\downarrow$
25	0.084	0.674	0.026	127.41	10.95	3	0.075	0.198	0.017
50	0.078	0.415	0.016	127.20	10.63	5	0.076	0.175	0.013
75	0.076	0.180	0.014	132.43	10.98	7	0.076	0.171	0.012
100	0.076	0.177	0.013	136.51	11.63	10	0.076	0.167	0.012
125	0.075	0.192	0.016	139.63	12.50	15	0.076	0.162	0.011

Table 8: Effect of subset size (left) and overlap size (right).

Variant	ATE $\downarrow$	RRE $\downarrow$	RTE $\downarrow$	Time	Mem.
w/o BA	0.551	2.284	0.119	62.86	12.58
BA w/ VGGT Tracking [ 35 ]	0.602	2.659	0.155	137.68	33.22
BA w/ LightGlue (pseudo video) [ 19 ]	0.526	1.572	0.061	140.53	12.58
BA w/ LightGlue (graph) [ 19 ]	0.522	1.563	0.060	141.08	12.68

Table 9: Effect of the bundle adjustment method.

Figure 8: Qualitative examples of 3D reconstructions on various indoor and outdoor scenes from Tanks & Temples [ 16 ] , 7-Scenes [ 29 ] and UrbanScene3D [ 18 ] . MERG3R produces high-quality, detailed reconstructions that preserve fine geometric structure and maintain global consistency.

Panel of Figure 9 — Figure 9: Qualitative examples of 3D reconstructions on dynamical scene from internet videos.

	ATE $\downarrow$							RRE $\downarrow$							RTE $\downarrow$
Scene	CUT3R [ 36 ]	TT3R [ 6 ]	MAST3R-SfM [ 10 ]	VGGT* [ 35 ]	Pi3 [ 38 ]	VGGT* + Ours	Pi3 + Ours	CUT3R [ 36 ]	TT3R [ 6 ]	MAST3R-SfM [ 10 ]	VGGT* [ 35 ]	Pi3 [ 38 ]	VGGT* + Ours	Pi3 + Ours	CUT3R [ 36 ]	TT3R [ 6 ]	MAST3R-SfM [ 10 ]	VGGT* [ 35 ]	Pi3 [ 38 ]	VGGT* + Ours	Pi3 + Ours
Barn	1.246	0.827	0.093	0.074	0.048	0.035	0.030	0.793	0.773	0.268	0.524	0.234	0.440	0.179	0.058	0.064	0.009	0.038	0.015	0.026	0.006
Caterpillar	0.974	0.277	0.036	0.051	0.035	0.021	0.023	0.987	0.931	0.658	0.299	0.195	0.104	0.097	0.069	0.058	0.012	0.028	0.018	0.007	0.005
Church	2.288	1.176	0.983	2.934	0.358	3.002	0.330	1.684	5.337	1.638	7.378	0.396	8.326	0.380	0.124	0.172	0.101	0.274	0.057	0.298	0.046
Ignatius	0.663	0.188	0.024	0.043	0.035	0.020	0.021	1.043	0.785	0.197	0.210	0.156	0.144	0.117	0.091	0.069	0.008	0.025	0.020	0.010	0.008
Meeting Room	1.819	0.646	0.046	0.071	0.035	0.039	0.037	0.987	0.884	0.163	0.340	0.201	0.230	0.166	0.102	0.121	0.008	0.036	0.021	0.016	0.009

Table 10: Per-scene camera pose evaluation results (ATE, RRE, RTE) across methods for the Tanks & Temples dataset.

	RRA@30 $\uparrow$							RTA@30 $\uparrow$							AUC@30 $\uparrow$
Scene	CUT3R [ 36 ]	TT3R [ 6 ]	MAST3R-SfM [ 10 ]	VGGT* [ 35 ]	Pi3 [ 38 ]	VGGT* + Ours	Pi3 + Ours	CUT3R [ 36 ]	TT3R [ 6 ]	MAST3R-SfM [ 10 ]	VGGT* [ 35 ]	Pi3 [ 38 ]	VGGT* + Ours	Pi3 + Ours	CUT3R [ 36 ]	TT3R [ 6 ]	MAST3R-SfM [ 10 ]	VGGT* [ 35 ]	Pi3 [ 38 ]	VGGT* + Ours	Pi3 + Ours
Barn	77.64	100	100	100	100	100	100	88.85	89.71	99.99	99.88	99.96	99.93	99.99	40.33	63.90	89.23	93.86	95.82	95.44	95.82
Caterpillar	87.97	100	99.48	100	100	100	100	89.51	98.66	99.70	99.91	99.77	99.99	100	48.15	90.89	92.75	95.36	95.85	96.48	96.5
Church	44.33	80.38	79.12	56.82	100	58.21	100	52.13	69.56	78.01	54.60	95.03	61.49	95.46	21.65	46.04	78.01	34.40	86.18	51.79	87.33
Ignatius	99.44	100	100	100	100	100	100	1.78	99.38	99.99	99.97	99.97	99.99	100	61.09	88.08	99.99	95.57	95.98	96.47	96.41
Meeting Room	95.09	100	100	100	100	100	100	69.00	92.98	99.95	99.83	99.96	99.94	99.95	39.76	68.96	94.73	93.51	95.72	95.32	95.27

Table 11: Per-scene camera pose evaluation results (RRA, RTA, AUC) across methods for the Tanks & Temples dataset.

Figure 10: Qualitative comparison of 3D reconstructions on large scale dataset. MERG3R achieves consistently better performance on the Zip-NeRF [ 4 ] scenes (Berlin). The input sequence is the original ordering before splitting.

Figure 11: Visualization results for each subset. Our full reconstruction faithfully recovers the entire house, while each individual subset captures only partial and fragmented portions of the scene.

Panel of Figure 12 — Figure 12: A sample sequence from Zip-NeRF dataset with randomly inserted frames from other scenes (above). The reordered sequence (bottom) places all inserted frames at the end.

Figure 13: Histograms of two pair types (valid–valid vs. valid–distractor). The clear separation between the distributions demonstrates the robustness of DINO similarity for detecting outlier frames.

Figure 14: Qualitative pose estimation results for all scenes in the Tanks & Temples dataset.

Explore similar work

VGG-T $^3$ : Offline Feed-Forward 3D Reconstruction at Scale
Feb 26, 2026Sven Elflein, Ruilong Li, Sérgio Agostinho +43D Reconstruction Feed-Forward 3D Reconstruction
NVIDIA · Vector Institute · University of Toronto
HD-VGGT: High-Resolution Visual Geometry Transformer
Mar 28, 2026Tianrun Chen, Yuanqi Hu, Yidong Han +113D Reconstruction 3D ViTs
KOKONI 3D, Moxin Technology · Zhejiang University · Huzhou University +4
TurboVGGT: Fast Visual Geometry Reconstruction with Adaptive Alternating Attention
May 14, 2026David Huang, Guile Wu, Chengjie Huang +23D ViTs Multi-View 3D Reconstruction
Huawei Noah’s Ark Lab · University of Toronto · Foundation Model Department, Huawei

Feb 26, 2026J/K move · Enter open · S save

VGG-T $^3$ : Offline Feed-Forward 3D Reconstruction at Scale

Sven Elflein, Ruilong Li, Sérgio Agostinho, Zan Gojcic+3

NVIDIA · Vector Institute · University of Toronto

3D Reconstruction Feed-Forward 3D Reconstruction 3D Scene Representation

Open paper AskPDF arXiv

We present a scalable 3D reconstruction model that addresses a critical limitation in offline feed-forward methods: their computational and memory requirements grow quadratically w.r.t. the number of input images. Our approach is built on the key insight that this bottleneck stems from the varying-length Key-Value (KV) space representation of scene geometry, which we distill into a fixed-size Multi-Layer Perceptron (MLP) via test-time training. VGG-T

^3

(Visual Geometry Grounded Test Time Training) scales linearly w.r.t. the number of input views, similar to online models, and reconstructs a

1k

image collection in just

54

seconds, achieving a

11.6\times

speed-up over baselines that rely on softmax attention. Since our method retains global scene aggregation capability, our point map reconstruction error outperforming other linear-time methods by large margins. Finally, we demonstrate visual localization capabilities of our model by querying the scene representation with unseen images.

Abstract

Explore similar work

VGG-T3^33: Offline Feed-Forward 3D Reconstruction at Scale

HD-VGGT: High-Resolution Visual Geometry Transformer

TurboVGGT: Fast Visual Geometry Reconstruction with Adaptive Alternating Attention

VGG-T $^3$ : Offline Feed-Forward 3D Reconstruction at Scale