Diff3R: Feed-forward 3D Gaussian Splatting with Uncertainty-aware Differentiable Optimization

Yueh-Cheng Liu; Jozef Hladký; Matthias Nießner; Angela Dai

cs.CVApr 1, 2026

Diff3R: Feed-forward 3D Gaussian Splatting with Uncertainty-aware Differentiable Optimization

Authors: Yueh-Cheng Liu, Jozef Hladký, Matthias Nießner, Angela Dai

Organizations: Technical University of Munich · Computing Systems Lab, Huawei Technologies, Switzerland

Abstract

Recent advances in 3D Gaussian Splatting (3DGS) present two main directions: feed-forward models offer fast inference in sparse-view settings, while per-scene optimization yields high-quality renderings but is computationally expensive. To combine the benefits of both, we introduce Diff3R, a novel framework that explicitly bridges feed-forward prediction and test-time optimization. By incorporating a differentiable 3DGS optimization layer directly into the training loop, our network learns to predict an optimal initialization for test-time optimization rather than a conventional zero-shot result. To overcome the computational cost of backpropagating through the optimization steps, we propose computing gradients via the Implicit Function Theorem and a scalable, matrix-free PCG solver tailored for 3DGS optimization. Additionally, we incorporate a data-driven uncertainty model into the optimization process by adaptively controlling how much the parameters are allowed to change during optimization. This approach effectively mitigates overfitting in under-constrained regions and increases robustness against input outliers. Since our proposed optimization layer is model-agnostic, we show that it can be seamlessly integrated into existing feed-forward 3DGS architectures for both pose-given and pose-free methods, providing improvements for test-time optimization.

Figures & tables

Figure 1: Optimization-Aware 3D Gaussian Splatting. (a) Standard feed-forward 3DGS models directly predict Gaussian parameters, frequently resulting in blurry novel views. Applying test-time optimization (TTO) to these predictions in sparse-view settings is highly unconstrained, which makes it easily overfits to the input views, trapping the solution in poor local minima with severe visual artifacts. (b) By training through the optimization process via implicit gradients, our method learns an optimization-aware initialization, tailored for 3DGS post-optimization.

Figure 2: Overview of our Uncertainty-Aware Differentiable 3DGS Framework. Given a sparse set of context views (with optional camera parameters), our feed-forward network predicts an initial set of 3D Gaussian parameters ( $\Theta_{0}$ ). Our proposed differentiable optimization layer refines these parameters via gradient descent to yield the optimized Gaussians ( $\Theta^{*}$ ). To train the network end-to-end, we introduce an efficient analytical solution for the backward pass using implicit gradients and a matrix-free PCG solver. Additionally, to make the optimization more robust in sparse-view settings, we predict learnable uncertainty weights ( $\boldsymbol{\Lambda}$ ). These weights act as an adaptive proximal bound on the optimization trajectory, preventing the model from overfitting to the context views.

	PSNR			SSIM		LPIPS
	Before	After		Before	After	Before	After
PixelSplat [ 3 ]	25.856	26.144	( $+$ 0.288)	0.848	0.854	0.150	0.145
MVSplat [ 4 ]	26.654	26.649	( $-$ 0.005)	0.864	0.866	0.132	0.131
DepthSplat [ 41 ]	27.665	27.694	( $+$ 0.029)	0.886	0.888	0.117	0.116
Ours	27.675	27.949	( $+$ 0.274)	0.887	0.891	0.116	0.112

Table 1: Novel view synthesis on RE10K with known poses. We report PSNR, SSIM, and LPIPS before and after test-time optimization (TTO) with two input views. While DepthSplat and ours achieve similar zero-shot performance, our method exhibits substantially larger gains after optimization (+0.27 dB vs. +0.03 dB) and attains the highest final scores, demonstrating that our optimization-aware initialization effectively balances prior preservation with adaptation flexibility.

Figure 3: Qualitative results on RE10K dataset with pose-given setting. We compare ours with feed-forward 3DGS methods such as PixelSplat [ 3 ] , MVSplat [ 4 ] , and DepthSplat [ 41 ] with two input views. As shown in the zoom-in regions, ours generates sharper result, and is more robust to exposure noises in the input views during optimization thanks to the adaptive regularization.

	PSNR			SSIM		LPIPS
	Before	After		Before	After	Before	After
AnySplat [ 11 ]	17.762	17.546	( $-$ 0.216)	0.603	0.585	0.325	0.328
DA3 [ 19 ]	21.981	22.582	( $+$ 0.601)	0.741	0.764	0.223	0.205
Ours	22.043	22.806	( $+$ 0.763)	0.742	0.767	0.221	0.202

Table 2: Pose-free feed-forward 3DGS on RE10K. Comparison of feed-forward methods using four input views without camera information. We report PSNR, SSIM, and LPIPS before and after test-time optimization (TTO). Notably, AnySplat suffers from catastrophic overfitting (PSNR degrades by 0.2 dB) due to poor initialization, while our method achieves the largest performance gains (+0.76 dB vs. +0.60 dB for DA3) and attains the best final scores across all metrics.

	ScanNet++ (DSLR)						ScanNet++ (iPhone)
	PSNR		SSIM		LPIPS			PSNR		SSIM		LPIPS
	Before	After	Before	After	Before	After		Before	After	Before	After	Before	After
AnySplat [ 11 ]	18.84	18.66	0.675	0.654	0.307	0.312		18.21	18.08	0.671	0.655	0.304	0.299
DA3 [ 19 ]	23.54	24.29	0.820	0.844	0.193	0.174		23.23	24.43	0.840	0.878	0.171	0.152
Ours	23.78	24.64	0.824	0.850	0.182	0.162		23.15	24.57	0.840	0.881	0.167	0.145

Table 3: Pose-free evaluation on ScanNet++ (DSLR & iPhone) with four input views. Results on the official validation split using four unposed input views. We compare against AnySplat and Depth Anything v3 (DA3), reporting metrics before and after TTO. Our method consistently outperforms baselines after optimization across both capture modalities, achieving particularly strong gains on the challenging DSLR split (+0.35 dB PSNR against DA3).

Figure 4: Qualitative results on ScanNet++. We compare our pose-free methods against Depth Anything v3 (DA3) [ 19 ] after optimization on ScanNet++ with four input views. Our method is able to reduce the artifacts during optimization and provides sharper renderings.

		PSNR			SSIM		LPIPS
		Before	After		Before	After	Before	After
(a)	DepthSplat [ 41 ]	27.665	27.694	( $+$ 0.029)	0.886	0.888	0.117	0.116
(b)	Ours w/o $\mathcal{L}_{\text{proximal}}$	27.675	27.828	( $+$ 0.153)	0.887	0.890	0.116	0.115
(c)	Ours w/o $\Lambda$	27.630	27.746	( $+$ 0.116)	0.888	0.889	0.116	0.115
(d)	Ours	27.675	27.949	( $+$ 0.274)	0.887	0.891	0.116	0.112

Table 4: Ablation study on Diff3R. We compare our full method and other variations on RE10K, pose-given, two-view settings. Without proximal term, the method is still benefiting from learning a better initialization but becomes worse due to unconstrained optimization. On the other hand, without the learnable parameter regularization, the global bound makes the optimization too conservative.

Figure 5: Visualization of learned uncertainty weights. We visualize the regularization weights applied to the mean of the per-pixel Gaussians ( $\Lambda_{\text{mean}}$ ). Notably, the network intrinsically learns to predict higher anchoring weights for regions lacking multi-view constraints ( e.g . , the left third of Input 1, which is not visible in Input 2) without explicit supervision. This helps to prevent overfitting during the test-time optimization process.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 6: More qualitative results on RE10K. Our predicted uncertainty weights implicitly learn to penalize changes ( i.e . , higher values) in non-overlapping regions during post-optimization. Crucially, we show that Diff3R learns a more effective initialization: while the initial predictions may yield worse zero-shot renderings, they result in higher-quality outputs following optimization.

Explore similar work

ForeSplat: Optimization-Aware Foresight for Feed-Forward 3D Gaussian Splatting
May 21, 2026Yuke Li, Weihang Liu, Cheng Zhang +83D Gaussian Splatting Meta-Learning
ShanghaiTech University · GGU Technology Co., Ltd · Stereye
GIFSplat: Generative Prior-Guided Iterative Feed-Forward 3D Gaussian Splatting from Sparse Views
Feb 26, 2026Tianyu Chen, Wei Xiang, Kang Han +43D Reconstruction Sparse-View 3D Reconstruction
La Trobe University, Melbourne, VIC 3086, Australia · Cisco Research, San Jose, CA, USA
SparseSplat: Towards Applicable Feed-Forward 3D Gaussian Splatting with Pixel-Unaligned Prediction
Apr 3, 2026Zicheng Zhang, Xiangting Meng, Ke Wu +13D Gaussian Splatting Feed-Forward 3D Reconstruction
Fudan University · ShanghaiTech University

May 21, 2026J/K move · Enter open · S save

ForeSplat: Optimization-Aware Foresight for Feed-Forward 3D Gaussian Splatting

Yuke Li, Weihang Liu, Cheng Zhang, Yuefeng Zhang+7

ShanghaiTech University · GGU Technology Co., Ltd · Stereye

3D Gaussian Splatting Meta-Learning 3D Reconstruction Feed-Forward 3D Reconstruction

Open paper AskPDF arXiv

Feed-forward 3D Gaussian Splatting models offer fast single-pass reconstruction,but scaling them to match per-scene optimization quality is fundamentally hindered by the scarcity of large-scale 3D annotations. A practical compromise is predict-then-refine,where post-prediction optimization compensates for the limited capacity of the feed-forward network. However,standard feed-forward 3DGS is trained solely for zero-step rendering error,ignoring whether its output constitutes a good initialization for the downstream optimizer. We present ForeSplat,an optimization-aware training framework that equips feed-forward 3DGS models to produce initializations explicitly designed for rapid,effective refinement. By offloading part of the scene-modeling burden to the optimizer,ForeSplat substantially reduces the capacity pressure on the feed-forward model,making high-quality reconstruction feasible even with compact networks. At its core is MetaGrad,a lightweight multi-anchor meta-gradient training rule that bypasses costly higher-order differentiation through the 3DGS optimizer. MetaGrad unrolls a short inner-loop refinement trajectory,samples anchor states,and back-propagates aggregated first-order gradients to the prediction head as a surrogate optimization-aware signal. This fine-tuning adds no inference cost and enables high-quality reconstruction within seconds after a few refinement steps. We instantiate ForeSplat on diverse backbones,including AnySplat,Pi3X,and a distilled variant tailored for edge deployment. Across all tested architectures,a ForeSplat-trained initialization converges in fewer refinement steps and reaches a higher peak reconstruction quality than its vanilla counterpart,even fully converged. The framework consistently bridges the gap between amortized prediction and per-scene optimization,establishing a practical path toward lightweight,high-fidelity 3D reconstruction.

Abstract

Explore similar work

ForeSplat: Optimization-Aware Foresight for Feed-Forward 3D Gaussian Splatting

GIFSplat: Generative Prior-Guided Iterative Feed-Forward 3D Gaussian Splatting from Sparse Views

SparseSplat: Towards Applicable Feed-Forward 3D Gaussian Splatting with Pixel-Unaligned Prediction