Human perceive the 3D world through 2D observations from limited viewpoints. While recent feed-forward generalizable 3D reconstruction models excel at recovering 3D structures from sparse images, their representations are often confined to observed regions, leaving unseen geometry un-modeled. This raises a key, fundamental challenge: Can we infer a complete 3D structure from partial 2D observations? We present RnG (Reconstruction and Generation), a novel feed-forward Transformer that unifies these two tasks by predicting an implicit, complete 3D representation. At the core of RnG, we propose a reconstruction-guided causal attention mechanism that separates reconstruction and generation at the attention level, and treats the KV-cache as an implicit 3D representation. Then, arbitrary poses can efficiently query this cache to render high-fidelity, novel-view RGBD outputs. As a result, RnG not only accurately reconstructs visible geometry but also generates plausible, coherent unseen geometry and appearance. Our method achieves state-of-the-art performance in both generalizable 3D reconstruction and novel view generation, while operating efficiently enough for real-time interactive applications. Project page: https://npucvr.github.io/RnG
Figures & tables
Task
3D reconstruction
Novel view synthesis
Unified model
Method
VGGT
DUSt3R
LVSM
LGM
RayZer
Matrix3D
RnG
[ 50 ]
[ 14 ]
[ 18 ]
[ 44 ]
[ 17 ]
[ 30 ]
Ours
unposed infer
✓
✓
✗
✗
✓
✓
✓
camera control
N/A
N/A
✓
✓
✗
✓
✓
generate unseen
✗
✗
✓
✗
✓
✓
✓
explicit 3D
✓
✓
✗
✓
✗
✓
✓
Table 1: Comparison between representative 3D reconstruction and novel view synthesis methods.
Figure 2: The Network Architecture of RnG. (a) Source view images are first tokenized using the DINO vision transformer; the Plücker ray map representing the target view point goes through a linear layer. After adding camera tokens for each view, all tokens will then alternately attend to global- and frame-level attention blocks. Finally, camera tokens from input views are used to estimate camera poses, while a point head and an RGB head process ray tokens from the target view, providing geometry and appearance estimations. (b) In inference, the model can cache K/V token from source views, synthesizing novel view geometry and geometry at a higher speed.
Figure 3: The reconstruction-guided causal attention. (a) During training, we decouple reconstruction and generation at the attention level inside global attention blocks. At inference time, the attention process is split into two steps: (b) source-view key value tokens are cached as an implicit 3D representation; (c) the KV-cache is queried by target view poses to generate novel views.
Reconstruction
Generation
Complete
Pose
Source View Depth
Novel View Depth
Novel View Synthesis
3D
RA@5 ↑
RT@5 ↑
AUC@30 ↑
Rel ↓
a1 ↑
Rel ↓
a1 ↑
PSNR ↑
SSIM ↑
LPIPS ↓
CD ↓
posed
LVSM [ 18 ]
camera poses are known
—
—
—
—
27.522
0.902
0.0895
—
LGM [ 44 ]
13.704
85.646
16.804
82.205
16.677
0.751
0.2248
0.1130
Matrix3D [ 30 ]
8.782
95.383
8.897
93.858
19.941
0.803
0.1684
0.0580
pixelSplat [ 4 ]
33.594
33.241
39.819
20.723
12.904
0.718
0.4514
0.4898
Table 2: Quantitative comparison. We evaluate the reconstruction and generation ability of all models on the GSO dataset. ‘—’ means that the model is not capable of delivering that result.
Figure 4: Visual comparison of novel view synthesis. Though RnG does not require accurate pose as input, it provides comparable visual quality with state-of-the-art pose-dependent methods like LVSM. Our model can hallucinate unseen regions with high 3D consistency.
Figure 6
Figure 7: Generalize to other number of input views. Although our model is not trained to handle other number of input views, it still shows strong generalization ability to other number of source images. RnG even works reasonably when given only 1 image.
Reconstruction
Generation
Pose
Source View Depth
Novel View Depth
Novel View Synthesis
Model
RA@5 ↑
RT@5 ↑
AUC@30 ↑
Rel ↓
a1 ↑
Rel ↓
a1 ↑
PSNR ↑
SSIM ↑
LPIPS ↓
LVSM-100K
not applicable
27.522
0.902
0.0895
LVSM-40K
24.619
0.864
0.1544
Ours-40K
85.146
86.019
86.942
0.584
99.929
0.717
99.850
26.276
0.891
0.0975
Ours-15K-scratch
8.252
17.184
33.654
2.331
99.475
2.331
99.364
20.784
0.819
0.2035
Table 3: Ablation studies. We studies the training efficiency of RnG by comparing with LVSM and effectiveness of the model architecture.
peak memory
FLOPS
time
Matrix3D [ 30 ]
11.31 G
2228 T
27 s
RnG w/o KV-Cache
8.12 G
12.26 T
213 ms
RnG w/ KV-Cache
7.10 G
2.29 T
85 ms
Table 4: Efficiency comparison of models with and without KV-Cache for inferring a single novel-view appearance and geometry.
Figure 8: The camera normalization process. The first camera pose will be normalized on a unit sphere and transformed to a fixed position. This introduces additional roll angles to other cameras.
Figure 9: The user interface to visualize novel view images and accumulate complete 3D from multiple views.
Reconstruction
Generation
# input
Pose
Input View Depth
Novel View Depth
Novel View Synthesis
views
RA@5 ↑
RT@5 ↑
AUC@30 ↑
Rel ↓
a1 ↑
Rel ↓
a1 ↑
PSNR ↑
SSIM ↑
LPIPS ↓
8
85.922
86.893
87.942
0.535
99.927
0.593
99.875
27.608
0.903
0.0857
6
85.825
86.505
87.741
0.539
99.935
0.622
99.876
27.235
0.899
0.0891
4
85.146
86.019
86.942
0.584
99.929
0.717
99.850
26.276
0.891
0.0975
3
85.243
86.311
86.136
0.662
99.931
0.848
99.820
25.291
0.881
0.1063
Table 5: Generalize to other number of input views. Although our model is not trained to handle other number of input views, it still shows strong generalization ability to other number of source images.
Figure 10: Qualitative results on CO3D.
Figure 11: RnG’s novel view synthesis results on multi-objects.
Figure 12: The evaluation protocol when models are given different number of input views. Each row represents an input configuration.
Figure 13: RnG performs well even when only a single image is given.
Figure 14: When generating novel views from only one source image is too hard, RnG performs reasonably bad.
Figure 15: RnG performs better when ambiguities are resolved.