Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective
Authors: Weijie Wang, Qihang Cao, Sensen Gao, Donny Y. Chen, Haofei Xu, Wenjing Bian, Songyou Peng, Tat-Jen Cham, +5 more
Organizations: Zhejiang University, China · Nanyang Technological University, Singapore · Monash University, Australia · ETH Zurich, Switzerland · University of Tübingen, Tübingen AI Center, Germany
Reconstructing 3D representations from 2D inputs is a fundamental task in computer vision and graphics, serving as a cornerstone for understanding and interacting with the physical world. While traditional methods achieve high fidelity, they are limited by slow per-scene optimization or category-specific training, which hinders their practical deployment and scalability. Hence, generalizable feed-forward 3D reconstruction has witnessed rapid development in recent years. By learning a model that maps images directly to 3D representations in a single forward pass, these methods enable efficient reconstruction and robust cross-scene generalization. Our survey is motivated by a critical observation: despite the diverse geometric output representations, ranging from implicit fields to explicit primitives, existing feed-forward approaches share similar high-level architectural patterns, such as image feature extraction backbones, multi-view information fusion mechanisms, and geometry-aware design principles. Consequently, we abstract away from these representation differences and instead focus on model design, proposing a novel taxonomy centered on model design strategies that are agnostic to the output format. Our proposed taxonomy organizes the research directions into five key problems that drive recent research development: feature enhancement, geometry awareness, model efficiency, augmentation strategies and temporal-aware models. To support this taxonomy with empirical grounding and standardized evaluation, we further comprehensively review related benchmarks and datasets, and extensively discuss and categorize real-world applications based on feed-forward 3D models. Finally, we outline future directions to address open challenges such as scalability, evaluation standards, and world modeling.
Figures & tables
Figure 1: Outline of the survey. The paper begins with an overview of core 3D representations (NeRF [ 1 ] , 3D Gaussian Splatting [ 2 ] , pointmaps [ 3 ] , and others [ 4 , 5 , 6 , 7 , 8 ] ) and how feed-forward networks generate them. It then analyzes methods along five challenge-driven axes: feature enhancement, geometry awareness, model efficiency, data or visual augmentation, and temporal modeling, followed by reviews of datasets and benchmarks, and practical applications ( e.g. , autonomous driving and robotics). Finally, based on a statistical summary of the surveyed works, we outline recent trends and discuss future directions for feed-forward 3D scene modeling.
Figure 3: Encoder taxonomy in recent feed-forward 3D reconstruction models. Common backbones include ViT [ 87 ] , ResNet [ 88 ] , U-Net [ 89 ] , and Mamba [ 90 ] /Mamba2 [ 91 ] , often initialized with or augmented by large-scale pre-trained priors ( e.g. , CroCo [ 92 ] , DINO [ 93 ] /DINOv2 [ 94 ] , CLIP [ 95 ] , UniMatch [ 96 ] , diffusion models [ 97 ] , or VAEs [ 98 ] ) to inject visual/geometric knowledge learned from 2D data. Representative model instances are discussed in § 4.1.3 .
Figure 4: Overview of research directions for geometry-aware improvement.
Figure 5: Efficiency comparison of different novel view synthesis (NVS) methods across varying numbers of input views (12, 24, and 36). Subplots (a)–(c) illustrate memory consumption, the number of Gaussians, and inference time, respectively. Color encodes the reconstruction method, while hatch patterns indicate the number of input views. All values were measured by the survey authors on RealEstate10K under a unified profiling setting on a single NVIDIA A100 GPU, with input images resized to 256×256 , using the publicly released code and checkpoints for every method.
Figure 6: Comparison between data and visual augmentation. (a) Data Augmentation: Combines synthetic and real data to train the feature extractor and 3D predictor. (b) Visual Augmentation: Refines rendered scenes using image or video diffusion to improve visual realism. Notably, modifying 3D parameters is optional.
Figure 7: Comparison of temporal-aware models. (a) Online Streaming models consume streaming observations and maintain a persistent global state for causal, step-by-step reasoning over time. (b) Offline Processing models jointly use a clip or window, allowing information from later frames to inform earlier predictions. (c) Interactive Modeling models predict not only geometry but also material and physics properties to support interactions. (d) Specialized Tasks models ( e.g. , dynamic removal or joint reconstruct-and-track) build on feed-forward backbones but optimize for specific objectives rather than generic full-scene 4D reconstruction.
Datasets
#Scenes
Type
Source
Train
Test
Geometry-Oriented
DTU [ 21 ]
124
R
MVSNeRF, GeoNeRF
DUSt3R, MASt3R, VGGT
GSO [ 337 ]
1,030
R
IBRNet, GS - LRM, NeuRay
LRM, GRM, Gamma
ABO [ 338 ]
50K
R
–
GS - LRM, LVSM
OmniObject3D [ 339 ]
6,000
S
AGG
MeshFormer
Objaverse [ 340 ]
818K
S
VGGT, LGM, LRM
LRM
Table 1: Representative datasets for feed-forward 3D reconstruction. These datasets are categorized by their primary purpose as Visual-Oriented, Geometry-Oriented, or Mixed. Each entry includes statistics such as dataset scale, source type, and scene category, with representative methods listed for training or evaluation. Within each section, datasets are ordered by category and release year. Symbols: = Objects, = Indoor Scenes, = Outdoor Scenes, + = Mixed Scenes. Source: R = Real, S = Synthetic.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
pixelNeRF [ 99 ]
19.31
0.789
0.382
SRF [ 166 ]
22.12
0.845
0.292
IBRNet [ 100 ]
26.04
0.917
0.190
MVSNeRF [ 162 ]
26.63
0.931
0.168
NeuRay [ 101 ]
26.47
0.875
0.158
GeoNeRF [ 163 ]
26.76
0.893
0.150
Table 2: Results on the DTU 3-view NVS Benchmark. Feed-forward 3D reconstruction methods evaluated on the benchmark are listed along with their reported results under the small-baseline setting.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
IBRNet [ 100 ]
15.99
0.484
0.532
pixelNeRF [ 99 ]
20.43
0.589
0.550
GPNR [ 83 ]
24.11
0.793
0.255
DBARF [ 249 ]
14.79
0.570
0.490
FlowCAM [ 386 ]
18.24
0.455
0.597
AttnRend [ 121 ]
24.78
0.820
0.213
Table 3: Results on the RealEstate10K 2-view NVS Benchmark. Feed-forward 3D reconstruction methods evaluated on the benchmark are listed along with their reported results under the small-baseline setting.
Method
7-Scenes
NRGBD
ETH3D
Acc. ↓
Comp. ↓
NC ↑
Acc. ↓
Comp. ↓
NC ↑
Acc. ↓
Comp. ↓
NC ↑
Overall ↓
DUSt3R [ 3 ]
0.786
1.748
0.597
0.453
2.520
0.515
0.496
9.567
0.563
5.032
MASt3R [ 26 ]
0.330
0.754
0.557
0.749
1.906
0.511
0.739
8.552
0.507
4.645
Spann3R [ 139 ]
0.672
1.648
0.591
0.496
2.555
0.503
0.874
9.339
0.532
5.107
Fast3R [ 390 ]
0.864
1.819
0.564
0.478
2.535
0.490
0.872
9.392
0.535
5.132
π 3 [ 190 ]
0.927
1.392
0.520
0.355
1.514
0.522
0.757
8.436
0.548
4.596
Table 4: Pointmap Estimation on 7-Scenes [ 344 ] , NRGBD [ 389 ] , and ETH3D [ 359 ] . Metrics: Acc. ↓ (pred → GT nearest-point distance, smaller is better), Comp. ↓ (GT → pred nearest-point distance, smaller is better), NC ↑ (normal consistency, larger is better), Overall ↓ = (Acc.+Comp.)/2. The top part reports results under the sparse-view setting using 3 input views, while the bottom part reports results under the dense-view setting using 10 input views.
Figure 8: Illustration of dataset types. The upper part presents geometry-oriented datasets, covering objects, indoor scenes, and outdoor scenes. The lower part presents visual-oriented datasets, including objects, synthetic indoor scenes, and mixed indoor and outdoor datasets.
Figure 9: Visualization of application scenarios for feed-forward 3D reconstruction. Adapted from prior work [ 408 , 409 , 410 , 411 , 412 , 413 , 403 , 414 , 415 , 416 ] .