Splat2Real: Novel-view Scaling for Physical AI with 3D Gaussian Splatting

Hansol Lim; Jongseong Brad Choi

cs.CVMar 11, 2026

Splat2Real: Novel-view Scaling for Physical AI with 3D Gaussian Splatting

Authors: Hansol Lim, Jongseong Brad Choi

Organizations: Department of Mechanical Engineering, State University of New York, Korea, 21985, Republic of Korea. · Department of Mechanical Engineering, State University of New York, Stony Brook, Stony Brook, NY 11794, USA.

Abstract

Physical AI faces viewpoint shift between training and deployment, and novel-view robustness is essential for monocular RGB-to-3D perception. We cast Real2Render2Real monocular depth pretraining as imitation-learning-style supervision from a digital twin oracle: a student depth network imitates expert metric depth/visibility rendered from a scene mesh, while 3DGS supplies scalable novel-view observations. We present Splat2Real, centered on novel-view scaling: performance depends more on which views are added than on raw view count. We introduce CN-Coverage, a coverage+novelty curriculum that greedily selects views by geometry gain and an extrapolation penalty, plus a quality-aware guardrail fallback for low-reliability teachers. Across 20 TUM RGB-D sequences with step-matched budgets (N=0 to 2000 additional rendered views, with N unique <= 500 and resampling for larger budgets), naive scaling is unstable; CN-Coverage mitigates worst-case regressions relative to Robot/Coverage policies, and GOL-Gated CN-Coverage provides the strongest medium-high-budget stability with the lowest high-novelty tail error. Downstream control-proxy results versus N provides embodied-relevance evidence by shifting safety/progress trade-offs under viewpoint shift.

Figures & tables

Fig. 1: Splat2Real pipeline. Real captures build a 3DGS observation teacher for high-throughput novel-view RGB rendering; simulator-style mesh rendering provides aligned metric oracle labels. A scaling policy (Random/Robot/Coverage/CN-Coverage) selects viewpoint budget $N$ . Gated/composited fallback is used as a secondary safety layer.

Metric	Value	Split/frames
Median abs depth error (m)	0.028 $\pm$ 0.019	held-out real frames
Mean abs depth error (m)	0.088 $\pm$ 0.111	held-out real frames
Valid-pixel fraction	0.875 $\pm$ 0.171	held-out real frames

TABLE I: Mesh-oracle fidelity.

Fig. 2: Step-matched scaling: metric AbsRel vs budget

N

for Random, Robot, Coverage, CN-Coverage, CN-Coverage (MeshHist), and GOL-Gated CN-Coverage. Error bars denote 95% CI across sequences; connecting lines are guides to the eye. Budgets are discrete and plotted on categorical x-axis positions. The dashed vertical marker at

N{=}500

denotes the unique-view cap; for

N{>}500

, training samples with replacement from the selected set (resampling-based count scaling).

N	Random	Robot	Coverage	CN-Coverage	CN-Coverage (MeshHist)	GOL-Gated CN-Coverage
0	0.36 $\pm$ 0.05	0.35 $\pm$ 0.06	0.34 $\pm$ 0.04	0.35 $\pm$ 0.05	0.30 $\pm$ 0.07	0.31 $\pm$ 0.07
25	0.36 $\pm$ 0.07	0.35 $\pm$ 0.08	0.36 $\pm$ 0.07	0.35 $\pm$ 0.08	0.31 $\pm$ 0.05	0.28 $\pm$ 0.05
50	0.36 $\pm$ 0.09	0.37 $\pm$ 0.09	0.37 $\pm$ 0.10	0.38 $\pm$ 0.07	0.33 $\pm$ 0.08	0.40 $\pm$ 0.12
100	0.35 $\pm$ 0.06	0.37 $\pm$ 0.09	0.36 $\pm$ 0.08	0.40 $\pm$ 0.10	0.34 $\pm$ 0.07	0.36 $\pm$ 0.09
200	0.38 $\pm$ 0.07	0.39 $\pm$ 0.10	0.39 $\pm$ 0.06	0.40 $\pm$ 0.12	0.38 $\pm$ 0.11	0.31 $\pm$ 0.04
500	0.36 $\pm$ 0.08	0.44 $\pm$ 0.12	0.40 $\pm$ 0.11	0.36 $\pm$ 0.09	0.33 $\pm$ 0.09	0.31 $\pm$ 0.06

TABLE II: Novel-view scaling.

Method	Mean	Worst	Range
GOL-Gated CN-Coverage	0.317	0.328	0.020
CN-Coverage (MeshHist)	0.356	0.390	0.066
Random	0.367	0.382	0.028
CN-Coverage	0.374	0.399	0.044
Coverage	0.397	0.416	0.035
Robot	0.446	0.482	0.096

TABLE III: Scaling stability summary.

Method	$N$	400-step	2000-step	$\Delta$
GOL-Gated CN-Coverage	200	0.322 $\pm$ 0.087	0.253 $\pm$ 0.046	-0.069
Random	200	0.369 $\pm$ 0.092	0.314 $\pm$ 0.046	-0.055
GOL-Gated CN-Coverage	2000	0.356 $\pm$ 0.082	0.263 $\pm$ 0.051	-0.093
Random	2000	0.362 $\pm$ 0.092	0.316 $\pm$ 0.042	-0.046

TABLE IV: Convergence-budget control.

Fig. 3: AbsRel versus surface-coverage fraction. Each point is one (sampler, $N$ ) setting.

Mode	Scenes	Tail@0	Tail@2000	$\Delta$ Tail
CN-Coverage	20	0.367 $\pm$ 0.063	0.387 $\pm$ 0.116	0.020 $\pm$ 0.082
CN-Coverage (MeshHist)	20	0.330 $\pm$ 0.088	0.408 $\pm$ 0.136	0.079 $\pm$ 0.062
Coverage	20	0.350 $\pm$ 0.053	0.419 $\pm$ 0.132	0.069 $\pm$ 0.109
GOL-Gated CN-Coverage	20	0.319 $\pm$ 0.072	0.348 $\pm$ 0.115	0.030 $\pm$ 0.083
Random	20	0.377 $\pm$ 0.060	0.387 $\pm$ 0.117	0.010 $\pm$ 0.082
Robot	20	0.385 $\pm$ 0.091	0.462 $\pm$ 0.154	0.077 $\pm$ 0.098

TABLE V: Tail summary.

Fig. 4: Error versus pose-novelty bins at low/high

N

in vertical subpanels: (a)

N=0

, (b)

N=2000

. The x-axis uses per-sequence novelty quantile bins 1–5 (1 = least novel, 5 = most novel), so all 20 sequences contribute to each panel. Shaded regions denote 95% CI across sequences.

Fig. 5: Teacher-quality interaction scatter at multiple budgets, where

\Delta

AbsRel

=

AbsRel(GS)

-

AbsRel(mesh+Hist) and negative is better. Dashed line marks gate threshold

q_{s}=1.0

. Correlations (Pearson/Spearman) are

N{=}0

: 0.06/

-0.07

N{=}500

: 0.21/0.08,

N{=}2000

-0.23

-0.11

Setting	Value
Total scenes	20
High-quality scenes (q_s ¿= 1)	10
Low-quality scenes (q_s ¡ 1)	10
High-quality rate	0.500
PSNR threshold	10.000
SSIM threshold	0.200

TABLE VI: Gating protocol.

Fig. 6: Success-collision Pareto across budgets

N

(each point annotated by

N

Mode	N	Succ	Col/100	Col/fail	Path
CN-Coverage	0	0.226 $\pm$ 0.026	0.900 $\pm$ 0.341	0.140 $\pm$ 0.053	0.281 $\pm$ 0.026
CN-Coverage	200	0.523 $\pm$ 0.031	22.892 $\pm$ 2.225	5.759 $\pm$ 0.417	0.647 $\pm$ 0.029
CN-Coverage	500	0.514 $\pm$ 0.031	21.525 $\pm$ 2.104	5.315 $\pm$ 0.394	0.625 $\pm$ 0.029
CN-Coverage	1000	0.386 $\pm$ 0.030	11.667 $\pm$ 1.734	2.280 $\pm$ 0.320	0.429 $\pm$ 0.029
Coverage	0	0.226 $\pm$ 0.026	0.900 $\pm$ 0.341	0.140 $\pm$ 0.053	0.281 $\pm$ 0.026
Coverage	200	0.280 $\pm$ 0.028	4.458 $\pm$ 1.037	0.743 $\pm$ 0.171	0.291 $\pm$ 0.027

TABLE VII: Downstream control proxy.

Condition	AbsRel	RMSE	SIlog	Scale	d1
real (upper)	0.221 $\pm$ 0.086	0.505 $\pm$ 0.171	21.556	0.163	0.605 $\pm$ 0.160
mesh	0.283 $\pm$ 0.104	0.583 $\pm$ 0.213	24.000	0.226	0.506 $\pm$ 0.160
mesh+DR	0.308 $\pm$ 0.098	0.621 $\pm$ 0.213	25.575	0.255	0.458 $\pm$ 0.180
mesh+Hist	0.281 $\pm$ 0.149	0.527 $\pm$ 0.150	22.192	0.221	0.560 $\pm$ 0.187
mesh+AdaIN	0.279 $\pm$ 0.115	0.563 $\pm$ 0.155	24.724	0.220	0.523 $\pm$ 0.145
3DGS	0.315 $\pm$ 0.073	0.726 $\pm$ 0.239	27.978	0.257	0.379 $\pm$ 0.168

TABLE A.1: Sim2Real results.

N	Comp	Scenes	MedD	MeanD	p
200	Best non-guardrail	20	0.030	-0.076	0.869
200	Coverage	20	-0.047	-0.081	0.00558
200	Random	20	-0.042	-0.074	0.00944
2000	Best non-guardrail	20	-0.003	-0.032	0.498
2000	Coverage	20	-0.023	-0.058	0.0121
2000	Random	20	-0.020	-0.031	0.0826

TABLE A.2: Paired Wilcoxon tests.

Mode	NPoints	Pearson	Spearman
All	48	0.182	0.182
Random	8	0.296	0.244
Robot	8	0.909	0.952
Coverage	8	0.899	0.903
CN-Coverage	8	0.452	0.390
CN-Coverage (MeshHist)	8	0.658	0.537

TABLE A.3: Coverage correlation.

N	BestSampler	BestAbsRel	Coverage
0	CN-Coverage (MeshHist)	0.298 $\pm$ 0.066	0.771
25	GOL-Gated CN-Coverage	0.283 $\pm$ 0.045	0.823
50	CN-Coverage (MeshHist)	0.331 $\pm$ 0.084	0.860
100	CN-Coverage (MeshHist)	0.342 $\pm$ 0.066	0.905
200	GOL-Gated CN-Coverage	0.308 $\pm$ 0.036	0.946
500	GOL-Gated CN-Coverage	0.309 $\pm$ 0.058	0.971

TABLE A.4: Scaling summary.

Method	AbsRel	RMSE	d1
mesh+Hist	0.302	0.759	0.354
3DGS	0.452	1.107	0.152
GOL-Gated	0.363	0.882	0.263

TABLE A.5: Holdout check.

N	Bucket	Qual.	Scenes	Mean $\pm$ std	Median	Frac( $\Delta<0$ )
0	High-q	$q_{s}\geq 1$	10	0.034 $\pm$ 0.071	0.034	0.10
0	Low-q	$q_{s}<1$	10	0.069 $\pm$ 0.166	0.087	0.20
500	High-q	$q_{s}\geq 1$	10	0.069 $\pm$ 0.091	0.044	0.10
500	Low-q	$q_{s}<1$	10	0.007 $\pm$ 0.105	0.019	0.50
2000	High-q	$q_{s}\geq 1$	10	-0.096 $\pm$ 0.147	-0.021	0.60
2000	Low-q	$q_{s}<1$	10	0.025 $\pm$ 0.107	0.020	0.40

TABLE A.6: Teacher-quality buckets.

Domain	Frames	LPIPS	PSNR	SSIM
3DGS	720	0.677 $\pm$ 0.075	10.855 $\pm$ 2.690	0.373 $\pm$ 0.151
mesh(vertex)	720	0.522 $\pm$ 0.105	11.374 $\pm$ 3.393	0.442 $\pm$ 0.153
mesh+Hist	720	0.499 $\pm$ 0.088	13.524 $\pm$ 3.166	0.506 $\pm$ 0.187
mesh-shaded(sim)	720	0.767 $\pm$ 0.053	5.863 $\pm$ 1.342	0.341 $\pm$ 0.161

TABLE A.7: Domain gap.

Component	Value	Units
GPU	NVIDIA GeForce RTX 5090	–
Teacher build time (n=19)	2.529 $\pm$ 0.392	s/sequence
3DGS setup	non-iterative init; voxel $=0.020$ m, points $=45000$ , iters $=0$	–
GS render throughput	219.89	views/s
Student training time (reference, 6 epochs)	36.22	s
Student inference speed (320x240)	552.56	FPS

TABLE A.8: Runtime.

Scenes	MedianErr (m)	MeanErr (m)	ValidFrac
20	0.028 $\pm$ 0.019	0.088 $\pm$ 0.111	0.875 $\pm$ 0.171

TABLE A.9: Mesh fidelity.

Name	Sampler	Observation source	Mixing rule
Random / Robot / Coverage	corresponding pose policy	3DGS RGB observations	none
CN-Coverage	coverage + bounded-novelty greedy policy	3DGS RGB observations	none
CN-Coverage (MeshHist)	CN-Coverage	mesh RGB with per-scene LAB Reinhard transfer to train real-RGB statistics	none
mesh+AdaIN	Coverage (reference schedule only)	mesh RGB with AdaIN style transfer using a sampled train-split real frame as style reference	none
GOL-Gated CN-Coverage	CN-Coverage	3DGS or mesh+Hist RGB	Bernoulli scene-level mixing with probability $g(q_{s})$
GOL-Composite	Coverage (reference schedule only)	3DGS and mesh+Hist RGB	per-pixel confidence blend $c\cdot$ 3DGS $+(1-c)\cdot$ mesh+Hist

TABLE B.1: Method matrix.

Setting	Value
Student model	DepthUNet (ch=32), single depth head
Input resolution	320x240 RGB
Optimizer	AdamW
Batch size	12
Learning rate	5.0e-05
Weight decay	1.0e-04

TABLE B.2: Training details.

Sampler	Pose rule
Random	train anchor + trans jitter N(0,[0.12,0.12,0.05] m) + yaw jitter N(0,10 deg)
Robot	arc offsets: radius U[0.15,0.35] m, heading U[-45,45] deg, z jitter +/-0.05 m
Coverage	greedy max new covered voxels
CN-Coverage	greedy max coverage-gain * exp(-d/sigma)
Candidate pool $\Omega$	benchmark default: random+robot candidates, $\|\Omega\|=1000$ per scene
Reuse	$\Omega$ fixed per scene/seed; shared across methods and all budgets $N$

TABLE B.3: Pose-sampler protocol.

Setting	Value
Coverage set V(T)	visible voxels from mesh-depth backprojection (z-buffer visibility)
Visibility label $V_{\mathrm{gt}}(T)$	supervision mask from mesh ray-casting; distinct from coverage set $V(T)$
Voxel size	0.100 m
Depth stride	12
$\|\Omega\|$ / scene	1000 $\pm$ 0
Unique selected/scene	500

TABLE B.4: Coverage and selection cost.

Seed	AbsRel
7	0.397 $\pm$ 0.119
11	0.339 $\pm$ 0.086
19	0.328 $\pm$ 0.079
Across-seed mean ± std	0.354 $\pm$ 0.037

TABLE C.1: Seed stability.

Scene	$\Delta$ AbsRel	$q_{s}$	Cov@50	NovQ80
f1-xyz	0.220	1.253	0.872	0.218
f3-struct-notex-near	0.187	0.761	0.931	1.368
f1-desk	0.119	1.158	0.862	0.336

TABLE C.2: N=50 regressions.

Method	$N$	voxel (m)	AbsRel	scenes
GOL-Gated CN-Coverage	200	0.010	0.310 $\pm$ 0.071	20
GOL-Gated CN-Coverage	200	0.020	0.307 $\pm$ 0.067	20
GOL-Gated CN-Coverage	200	0.050	0.308 $\pm$ 0.068	20

TABLE C.3: Voxel sensitivity.

Method	$N$	$\sigma$	AbsRel	scenes
GOL-Gated CN-Coverage	200	0.170	0.305 $\pm$ 0.063	20
GOL-Gated CN-Coverage	200	0.350	0.313 $\pm$ 0.038	20
GOL-Gated CN-Coverage	200	0.700	0.315 $\pm$ 0.088	20
GOL-Gated CN-Coverage	2000	0.170	0.332 $\pm$ 0.086	20
GOL-Gated CN-Coverage	2000	0.350	0.354 $\pm$ 0.095	20
GOL-Gated CN-Coverage	2000	0.700	0.355 $\pm$ 0.092	20

TABLE C.4:

\sigma

sensitivity.

Metric	Value
Spearman correlation	0.221
Pearson correlation	0.028
Scenes	20

TABLE C.5: Geometry proxy correlation.

Method	N	Steps	Regime	AbsRel	Scenes
Random	0	400	Step-matched	0.357 $\pm$ 0.048	20
Random	200	400	Step-matched	0.382 $\pm$ 0.069	20
Random	200	437	Exp-match	0.397 $\pm$ 0.083	20
Random	2000	400	Step-matched	0.355 $\pm$ 0.080	20
Random	2000	770	Exp-match	0.345 $\pm$ 0.076	20
GOL-Gated CN-Coverage	0	400	Step-matched	0.312 $\pm$ 0.069	20

TABLE C.6: Exposure-matched control.

N	Method	AbsRel	Scenes
200	Coverage	0.389 $\pm$ 0.058	20
200	CN-Coverage	0.399 $\pm$ 0.120	20
200	GOL-Gated CN-Coverage	0.308 $\pm$ 0.036	20
200	StochGreedy-Cov	0.395 $\pm$ 0.057	20
2000	Coverage	0.381 $\pm$ 0.095	20
2000	CN-Coverage	0.355 $\pm$ 0.090	20

TABLE C.7: Stochastic-greedy baseline.

Fig. D.1: Per-sequence paired

\Delta

AbsRel distributions at

N=\{200,2000\}

Fig. D.2: N=50 failure-regime diagnosis: per-sequence $\Delta$ AbsRel versus teacher quality.

Fig. D.3: Geometry-aware teacher-quality proxy correlation at

N=200

N	Comparator	MeanD	MedD	W/L/T	p
200	CN-Coverage	-0.091	-0.010	11/9/0	0.368
200	CN-Coverage (MeshHist)	-0.076	0.030	9/11/0	0.869
2000	CN-Coverage	-0.032	-0.003	11/9/0	0.498
2000	CN-Coverage (MeshHist)	-0.067	-0.009	12/8/0	0.097

TABLE E.1: Guardrail comparisons.

N	Comparator	MeanD	MedD	W/L/T	p
0	CN-Coverage	-0.048	-0.066	17/3/0	0.005
0	CN-Coverage (MeshHist)	-0.011	0.003	10/10/0	0.674
2000	CN-Coverage	-0.039	-0.005	12/8/0	0.216
2000	CN-Coverage (MeshHist)	-0.060	-0.016	15/5/0	0.012

TABLE E.2: Tail comparisons.

Active View Selection

Active view selection chooses camera or sensor viewpoints that reveal the most useful new evidence for perception, reconstruction, or action.

See all 35 papers

Efficient Camera Pose Augmentation for View Generalization in Robotic Policy Learning
Mar 31, 2026Sen Wang, Huaiyi Dong, Jingyi Tian +7Robot Policy Generalization Robot Policy Learning
Xi’an Jiaotong University, Noah’s Ark Lab
View-Structured Conformal Prediction for 3D Gaussian Splatting
Sep 9, 2026Junzheng Chu, Bin Pan, Zhenwei Shi3D Gaussian Splatting Conformal Prediction
School of Statistics and Data Science, LEBPS, KLMDASR, LPMC, AAIS, NITFID, Nankai University · Department of Aerospace Intelligent Science and Technology, School of Astronautics, Beihang University
ReconSplat: Generalizable 3D Scene Reconstruction Beyond Observed Views
Aug 28, 2026Giuseppe Stracquadanio, Kevin Raj, Julia Grabinski +1Feed-Forward 3D Reconstruction Novel View Synthesis
TU Darmstadt · Zuse School ELIZA

Mar 31, 2026J/K move · Enter open · S save

Efficient Camera Pose Augmentation for View Generalization in Robotic Policy Learning

Sen Wang, Huaiyi Dong, Jingyi Tian, Jiayi Li+6

Xi’an Jiaotong University, Noah’s Ark Lab

Robot Policy Generalization Robot Policy Learning Visuomotor Policy Learning 3D Gaussian Splatting

Open paper AskPDF arXiv

Prevailing 2D-centric visuomotor policies exhibit a pronounced deficiency in novel view generalization, as their reliance on static observations hinders consistent action mapping across unseen views. In response, we introduce GenSplat, a feed-forward 3D Gaussian Splatting framework that facilitates view-generalized policy learning through novel view rendering. GenSplat employs a permutation-equivariant architecture to reconstruct high-fidelity 3D scenes from sparse, uncalibrated inputs in a single forward pass. To ensure structural integrity, we design a 3D-prior distillation strategy that regularizes the 3DGS optimization, preventing the geometric collapse typical of purely photometric supervision. By rendering diverse synthetic views from these stable 3D representations, we systematically augment the observational manifold during training. This augmentation forces the policy to ground its decisions in underlying 3D structures, thereby ensuring robust execution under severe spatial perturbations where baselines severely degrade.

Abstract

Active View Selection

Explore similar work

Efficient Camera Pose Augmentation for View Generalization in Robotic Policy Learning

View-Structured Conformal Prediction for 3D Gaussian Splatting

ReconSplat: Generalizable 3D Scene Reconstruction Beyond Observed Views