Learning 3D Reconstruction with Priors in Test Time

Lei Zhou; Haoyu Wu; Akshat Dave; Dimitris Samaras

cs.CVApr 4, 2026

Learning 3D Reconstruction with Priors in Test Time

Authors: Lei Zhou, Haoyu Wu, Akshat Dave, Dimitris Samaras

Organizations: Stony Brook University

Abstract

We introduce a test-time framework for multiview Transformers (MVTs) that incorporates priors (e.g., camera poses, intrinsics, and depth) to improve 3D tasks without retraining or modifying pre-trained image-only networks. Rather than feeding priors into the architecture, we cast them as constraints on the predictions and optimize the network at inference time. The optimization loss consists of a self-supervised objective and prior penalty terms. The self-supervised objective captures the compatibility among multi-view predictions and is implemented using photometric or geometric loss between renderings from other views and each view itself. Any available priors are converted into penalty terms on the corresponding output modalities. Across a series of 3D vision benchmarks, including point map estimation and camera pose estimation, our method consistently improves performance over base MVTs by a large margin. On the ETH3D, 7-Scenes, and NRGBD datasets, our method reduces the point-map distance error by more than half compared with the base image-only models. Our method also outperforms retrained prior-aware feed-forward methods, demonstrating the effectiveness of our test-time constrained optimization (TCO) framework for incorporating priors into 3D vision tasks.

Figures & tables

Figure 1: Method Overview. (left) Multi-view Transformers (MVTs) take a set of RGB images as input and output depth maps, camera poses, and intrinsics. (middle) Given camera priors, MapAnything [ 22 ] and Pow3R [ 19 ] feed them into the network as additional input modalities, which requires retraining a modified MVT. (right) Our method, Test-time Constrained Optimization (TCO), treats the priors as constraints on the MVT’s predictions and optimizes the network with LoRA using both prior penalty terms and a self-supervised objective, namely MVT prediction compatibility, at inference time.

Method	Compat	Pose	Intr	Acc. $\downarrow$		Comp. $\downarrow$		N.C. $\uparrow$
Method	Compat	Pose	Intr	Mean	Med.	Mean	Med.	Mean	Med.
Fast3R [ 48 ]	✗	✗	✗	0.832	0.691	0.978	0.683	0.667	0.766
CUT3R [ 44 ]	✗	✗	✗	0.617	0.525	0.747	0.579	0.754	0.848
FLARE [ 53 ]	✗	✗	✗	0.464	0.338	0.664	0.395	0.744	0.864
Pow3R ( asym. ) [ 19 ]	✗	✓	✓	0.230	0.146	0.188	0.119	0.865	0.968
MapAnything [ 22 ]	✗	✓	✓	0.167	0.100	0.131	0.066	0.866	0.968

Table 1: Point Map Estimation on ETH3D. TCO outperforms both base image-only MVTs and prior-aware feed-forward methods, achieving the best overall performance. Camera poses are the most important prior for improving 3D reconstruction on ETH3D, and combining them with intrinsics yields further gains.

Method	Compat	Pose	Intr	Acc. $\downarrow$		Comp. $\downarrow$		N.C. $\uparrow$
Method	Compat	Pose	Intr	Mean	Med.	Mean	Med.	Mean	Med.
Fast3R [ 48 ]	✗	✗	✗	3.340	1.919	2.929	1.125	0.671	0.755
CUT3R [ 44 ]	✗	✗	✗	4.742	2.600	3.400	1.316	0.679	0.764
FLARE [ 53 ]	✗	✗	✗	2.541	1.468	3.174	1.420	0.684	0.774
Pow3R [ 19 ]	✗	✓	✓	1.234	0.620	2.357	0.833	0.680	0.765
MapAnything [ 22 ]	✗	✓	✓	2.731	1.570	2.209	0.892	0.681	0.767

Table 2: Point Map Estimation on DTU. TCO outperforms both the base image-only models and the prior-aware feed-forward methods. The proposed prediction compatibility loss is essential for incorporating priors as a regularization signal from distillation priors into the network.

Method	Compat	Pose	Intr	Acc. $\downarrow$		Comp. $\downarrow$		NC. $\uparrow$
Method	Compat	Pose	Intr	Mean	Med.	Mean	Med.	Mean	Med.
Fast3R [ 48 ]	✗	✗	✗	0.096	0.065	0.145	0.093	0.672	0.760
CUT3R [ 44 ]	✗	✗	✗	0.094	0.051	0.101	0.050	0.703	0.804
FLARE [ 53 ]	✗	✗	✗	0.085	0.058	0.142	0.104	0.695	0.779
Pow3R [ 19 ]	✗	✓	✓	0.034	0.019	0.041	0.022	0.763	0.880
MapAnything [ 22 ]	✗	✓	✓	0.026	0.015	0.032	0.018	0.773	0.889

Table 3: Point Map Estimation on 7-Scenes (sparse view). Our method outperforms both the base image-only models and the prior-aware feed-forward methods.

Method	Compat	Pose	Intr	Acc. $\downarrow$		Comp. $\downarrow$		NC. $\uparrow$
Method	Compat	Pose	Intr	Mean	Med.	Mean	Med.	Mean	Med.
Fast3R [ 48 ]	✗	✗	✗	0.135	0.091	0.163	0.104	0.759	0.877
CUT3R [ 44 ]	✗	✗	✗	0.104	0.041	0.079	0.031	0.822	0.968
FLARE [ 53 ]	✗	✗	✗	0.053	0.024	0.051	0.025	0.877	0.988
Pow3R [ 19 ]	✗	✓	✓	0.062	0.029	0.062	0.029	0.844	0.983
MapAnything [ 22 ]	✗	✓	✓	0.040	0.024	0.041	0.020	0.875	0.985

Table 4: Point Map Estimation on NRGBD (sparse view). Our method outperforms both the base image-only models and the prior-aware feed-forward methods.

Figure 2: Qualitative Comparison. We compare TCO-VGGT with the base image-only model VGGT and the prior-aware feed-forward methods Pow3R and MapAnything. Overall, TCO-VGGT effectively corrects structural errors in image-only reconstructions by incorporating camera priors. Red , orange , and green circles highlight regions that are wrongly reconstructed, partially corrected, and correctly reconstructed, respectively. In the first row, TCO-VGGT corrects the inaccurate relative positions of two walls in the VGGT reconstruction. The same phenomenon is observed in the second and third rows, where the inaccurate scene structures are corrected. In the last row, TCO-VGGT reconstructs the hand as a whole, whereas VGGT reconstructs it as separate parts. MapAnything reconstructs the hand with blurry boundaries. Additional fine-grained qualitative comparisons with MapAnything and Pow3R are provided in the Supplementary Material.

Stride	Method	Compat	Depth	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$
	Pow3R [ 19 ]	✗	✓	0.0172	0.0258	0.4614
1	VGGT [ 43 ]	✗	✗	0.0094	0.0160	0.3781
	TCO-VGGT	✓	✓	0.0072	0.0139	0.3731
	Pow3R [ 19 ]	✗	✓	0.0201	0.0335	0.7114
4	VGGT [ 43 ]	✗	✗	0.0127	0.0215	0.5706
	TCO-VGGT	✓	✓	0.0106	0.0190	0.6026

Table 5: Camera Pose Estimation on ScanNet with different keyframe strides. Our method outperforms both the base image-only models and the prior-aware feed-forward methods.

Fine-tuning strategy
Setting	Acc. $\downarrow$		Comp. $\downarrow$		N.C. $\uparrow$
	Mean	Med.	Mean	Med.	Mean	Med.
Finetune decoder (Ours)	0.114	0.060	0.116	0.054	0.901	0.984
+ Camera head	0.158	0.107	0.191	0.113	0.864	0.966
+ Depth head	0.162	0.081	0.098	0.054	0.850	0.954
+ Camera + depth head	0.183	0.088	0.195	0.116	0.817	0.937

Table 6: Ablations on ETH3D. We study the fine-tuning strategy and the 2DGS heuristics used in the compatibility objective.

Figure 3: Test-time scaling curve of our method on ETH3D.

Rendering loss
Setting	Acc. $\downarrow$		Comp. $\downarrow$		N.C. $\uparrow$
	Mean	Med.	Mean	Med.	Mean	Med.
VGGT (base)	0.280	0.185	0.305	0.182	0.853	0.950
Radius scale = 0.5 (def.)	0.114	0.060	0.116	0.054	0.901	0.984
Radius scale = 0.05	0.122	0.066	0.124	0.058	0.895	0.983
Radius scale = 5	0.143	0.072	0.207	0.123	0.871	0.972

Table 7: Ablation: Target view rendering loss and LoRA rank (ETH3D). Acc., Comp., and N.C. are reported (lower is better for Acc./Comp., higher is better for N.C.).

Rot pert. (deg)	Trans pert. (%)	Focal pert. (%)	Acc. $\downarrow$	Comp. $\downarrow$	N.C. $\uparrow$
VGGT (base)			0.280	0.305	0.853
0	0	0	0.1765	0.1816	0.8822
1	0	0	0.1755	0.1755	0.8791
3	0	0	0.2111	0.2108	0.8669
5	0	0	0.2344	0.2143	0.8569
0	1	0	0.1702	0.1700	0.8790

Table 8: Noise robustness (ETH3D). We report mean Acc., mean Comp., and mean N.C. (lower is better for Acc./Comp., higher is better for N.C.).

Figure 4: Fine-grained Qualitative Results. We compare TCO-VGGT with prior-aware feed-forward methods, including Pow3R and MapAnything. In each grid cell, the predicted geometry is overlaid with the ground truth geometry, whose points are shown in green . Discrepancies between the predicted and ground-truth geometries are highlighted by red double arrows, whose lengths indicate the magnitude of the errors. TCO-VGGT exhibits much smaller discrepancies than MapAnything and Pow3R in both scene structure and boundary regions.

TTT Steps	Render Loss	Avg (s)	Std (s)	Min (s)	Max (s)
0	✗	1.301	0.198	1.104	1.499
10	✗	43.630	0.218	43.412	43.848
10	✓	45.598	0.440	45.158	46.037
40	✓	161.547	1.424	160.123	162.971

Table 9: Test-time training steps vs. time (10 views). The reported times are wall-clock seconds per scene. The image resolution is 392(H)

\times

518(W). The base model is VGGT. The GPU is NVIDIA TITAN RTX.

Figure 5: 2DGS Rendering Visualization. We visualize the 2DGS rendering process for one scene from 7-Scenes. As shown in the Rendered Image row, our 2DGS heuristic parameterization produces rendered images that closely match the ground-truth images. We also compare the depth maps and normal maps rendered from 2DGS with the corresponding ground-truth depth and normal maps, i.e., those directly predicted from the MVT views.

Multi-View 3D Reconstruction

Multi-View 3D Reconstruction recovers scene geometry and appearance from images captured at multiple viewpoints.

See all 277 papers

Déjà View: Looping Transformers for Multi-View 3D Reconstruction
May 28, 2026Alessandro Burzio, Tobias Fischer, Sven Elflein +9Multi-View 3D Reconstruction Recurrent Transformers
NVIDIA · University of Modena and Reggio Emilia, AImageLab · ETH Zürich +1
Motion- and Gravity-Aware Feedforward 3D Reconstruction
Oct 15, 2026Shiyao Qian, Yuan Ren, Dongfeng BaiMulti-View 3D Reconstruction Feed-Forward 3D Reconstruction
Huawei Noah’s Ark Lab, Markham, ON, Canada. · University of Toronto, Toronto, ON, Canada.
Lite3R: A Model-Agnostic Framework for Efficient Feed-Forward 3D Reconstruction
May 12, 2026Haoyu Zhang, Zeyu Zhang, Zedong Zhou +2Quantization-Aware Training 3D Reconstruction
1Peking University · 2La Trobe University

May 28, 2026J/K move · Enter open · S save

Déjà View: Looping Transformers for Multi-View 3D Reconstruction

Alessandro Burzio, Tobias Fischer, Sven Elflein, Qunjie Zhou+8

NVIDIA · University of Modena and Reggio Emilia, AImageLab · ETH Zürich+1

Multi-View 3D Reconstruction Recurrent Transformers Iterative Refinement Feed-Forward 3D Reconstruction

Open paper AskPDF arXiv

Recent feed-forward 3D reconstruction transformers have scaled to over a billion parameters, following the broader trend of increasing model capacity in computer vision. Yet emerging evidence suggests that contiguous transformer layers often behave like repeated applications of similar operations, and multi-view reconstruction transformers refine their predictions progressively across decoder depth. We posit that model depth partially buys iteration, paid for inefficiently in unique parameters, and instead make that iteration explicit in architecture. Our model, DéjàView, applies a single looped transformer block recurrently to per-view features for K refinement steps. Trained once, it exposes K as an inference-time compute knob, matching or outperforming substantially larger feed-forward baselines across five reconstruction benchmarks spanning indoor, outdoor, object-centric, and driving scenes, while using a fraction of their parameters and comparable or lower compute. Importantly, the same looped block formulation outperforms an otherwise identical variant with independent per-step parameters under matched training data and compute, suggesting that explicit iteration is not merely a compute-efficient substitute for capacity but a stronger inductive bias for multi-view 3D reconstruction.

Abstract

Multi-View 3D Reconstruction

Explore similar work

Déjà View: Looping Transformers for Multi-View 3D Reconstruction

Motion- and Gravity-Aware Feedforward 3D Reconstruction

Lite3R: A Model-Agnostic Framework for Efficient Feed-Forward 3D Reconstruction