Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective
Authors: Weijie Wang, Qihang Cao, Sensen Gao, Donny Y. Chen, Haofei Xu, Wenjing Bian, Songyou Peng, Tat-Jen Cham, +5 more
Organizations: Zhejiang University, China · Nanyang Technological University, Singapore · Monash University, Australia · ETH Zurich, Switzerland · University of Tübingen, Tübingen AI Center, Germany
Reconstructing 3D representations from 2D inputs is a fundamental task in computer vision and graphics, serving as a cornerstone for understanding and interacting with the physical world. While traditional methods achieve high fidelity, they are limited by slow per-scene optimization or category-specific training, which hinders their practical deployment and scalability. Hence, generalizable feed-forward 3D reconstruction has witnessed rapid development in recent years. By learning a model that maps images directly to 3D representations in a single forward pass, these methods enable efficient reconstruction and robust cross-scene generalization. Our survey is motivated by a critical observation: despite the diverse geometric output representations, ranging from implicit fields to explicit primitives, existing feed-forward approaches share similar high-level architectural patterns, such as image feature extraction backbones, multi-view information fusion mechanisms, and geometry-aware design principles. Consequently, we abstract away from these representation differences and instead focus on model design, proposing a novel taxonomy centered on model design strategies that are agnostic to the output format. Our proposed taxonomy organizes the research directions into five key problems that drive recent research development: feature enhancement, geometry awareness, model efficiency, augmentation strategies and temporal-aware models. To support this taxonomy with empirical grounding and standardized evaluation, we further comprehensively review related benchmarks and datasets, and extensively discuss and categorize real-world applications based on feed-forward 3D models. Finally, we outline future directions to address open challenges such as scalability, evaluation standards, and world modeling.
Figures & tables
Figure 1: Outline of the survey. The paper begins with an overview of core 3D representations (NeRF [ 1 ] , 3D Gaussian Splatting [ 2 ] , pointmaps [ 3 ] , and others [ 4 , 5 , 6 , 7 , 8 ] ) and how feed-forward networks generate them. It then analyzes methods along five challenge-driven axes: feature enhancement, geometry awareness, model efficiency, data or visual augmentation, and temporal modeling, followed by reviews of datasets and benchmarks, and practical applications ( e.g. , autonomous driving and robotics). Finally, based on a statistical summary of the surveyed works, we outline recent trends and discuss future directions for feed-forward 3D scene modeling.
Figure 3: Encoder taxonomy in recent feed-forward 3D reconstruction models. Common backbones include ViT [ 87 ] , ResNet [ 88 ] , U-Net [ 89 ] , and Mamba [ 90 ] /Mamba2 [ 91 ] , often initialized with or augmented by large-scale pre-trained priors ( e.g. , CroCo [ 92 ] , DINO [ 93 ] /DINOv2 [ 94 ] , CLIP [ 95 ] , UniMatch [ 96 ] , diffusion models [ 97 ] , or VAEs [ 98 ] ) to inject visual/geometric knowledge learned from 2D data. Representative model instances are discussed in § 4.1.3 .
Figure 4: Overview of research directions for geometry-aware improvement.
Figure 5: Efficiency comparison of different novel view synthesis (NVS) methods across varying numbers of input views (12, 24, and 36). Subplots (a)–(c) illustrate memory consumption, the number of Gaussians, and inference time, respectively. Color encodes the reconstruction method, while hatch patterns indicate the number of input views. All values were measured by the survey authors on RealEstate10K under a unified profiling setting on a single NVIDIA A100 GPU, with input images resized to 256×256 , using the publicly released code and checkpoints for every method.
Figure 6: Comparison between data and visual augmentation. (a) Data Augmentation: Combines synthetic and real data to train the feature extractor and 3D predictor. (b) Visual Augmentation: Refines rendered scenes using image or video diffusion to improve visual realism. Notably, modifying 3D parameters is optional.
Figure 7: Comparison of temporal-aware models. (a) Online Streaming models consume streaming observations and maintain a persistent global state for causal, step-by-step reasoning over time. (b) Offline Processing models jointly use a clip or window, allowing information from later frames to inform earlier predictions. (c) Interactive Modeling models predict not only geometry but also material and physics properties to support interactions. (d) Specialized Tasks models ( e.g. , dynamic removal or joint reconstruct-and-track) build on feed-forward backbones but optimize for specific objectives rather than generic full-scene 4D reconstruction.
Datasets
#Scenes
Type
Source
Train
Test
Geometry-Oriented
DTU [ 21 ]
124
R
MVSNeRF, GeoNeRF
DUSt3R, MASt3R, VGGT
GSO [ 337 ]
1,030
R
IBRNet, GS - LRM, NeuRay
LRM, GRM, Gamma
ABO [ 338 ]
50K
R
–
GS - LRM, LVSM
OmniObject3D [ 339 ]
6,000
S
AGG
MeshFormer
Objaverse [ 340 ]
818K
S
VGGT, LGM, LRM
LRM
Table 1: Representative datasets for feed-forward 3D reconstruction. These datasets are categorized by their primary purpose as Visual-Oriented, Geometry-Oriented, or Mixed. Each entry includes statistics such as dataset scale, source type, and scene category, with representative methods listed for training or evaluation. Within each section, datasets are ordered by category and release year. Symbols: = Objects, = Indoor Scenes, = Outdoor Scenes, + = Mixed Scenes. Source: R = Real, S = Synthetic.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
pixelNeRF [ 99 ]
19.31
0.789
0.382
SRF [ 166 ]
22.12
0.845
0.292
IBRNet [ 100 ]
26.04
0.917
0.190
MVSNeRF [ 162 ]
26.63
0.931
0.168
NeuRay [ 101 ]
26.47
0.875
0.158
GeoNeRF [ 163 ]
26.76
0.893
0.150
Table 2: Results on the DTU 3-view NVS Benchmark. Feed-forward 3D reconstruction methods evaluated on the benchmark are listed along with their reported results under the small-baseline setting.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
IBRNet [ 100 ]
15.99
0.484
0.532
pixelNeRF [ 99 ]
20.43
0.589
0.550
GPNR [ 83 ]
24.11
0.793
0.255
DBARF [ 249 ]
14.79
0.570
0.490
FlowCAM [ 386 ]
18.24
0.455
0.597
AttnRend [ 121 ]
24.78
0.820
0.213
Table 3: Results on the RealEstate10K 2-view NVS Benchmark. Feed-forward 3D reconstruction methods evaluated on the benchmark are listed along with their reported results under the small-baseline setting.
Method
7-Scenes
NRGBD
ETH3D
Acc. ↓
Comp. ↓
NC ↑
Acc. ↓
Comp. ↓
NC ↑
Acc. ↓
Comp. ↓
NC ↑
Overall ↓
DUSt3R [ 3 ]
0.786
1.748
0.597
0.453
2.520
0.515
0.496
9.567
0.563
5.032
MASt3R [ 26 ]
0.330
0.754
0.557
0.749
1.906
0.511
0.739
8.552
0.507
4.645
Spann3R [ 139 ]
0.672
1.648
0.591
0.496
2.555
0.503
0.874
9.339
0.532
5.107
Fast3R [ 390 ]
0.864
1.819
0.564
0.478
2.535
0.490
0.872
9.392
0.535
5.132
π 3 [ 190 ]
0.927
1.392
0.520
0.355
1.514
0.522
0.757
8.436
0.548
4.596
Table 4: Pointmap Estimation on 7-Scenes [ 344 ] , NRGBD [ 389 ] , and ETH3D [ 359 ] . Metrics: Acc. ↓ (pred → GT nearest-point distance, smaller is better), Comp. ↓ (GT → pred nearest-point distance, smaller is better), NC ↑ (normal consistency, larger is better), Overall ↓ = (Acc.+Comp.)/2. The top part reports results under the sparse-view setting using 3 input views, while the bottom part reports results under the dense-view setting using 10 input views.
Figure 8: Illustration of dataset types. The upper part presents geometry-oriented datasets, covering objects, indoor scenes, and outdoor scenes. The lower part presents visual-oriented datasets, including objects, synthetic indoor scenes, and mixed indoor and outdoor datasets.
Figure 9: Visualization of application scenarios for feed-forward 3D reconstruction. Adapted from prior work [ 408 , 409 , 410 , 411 , 412 , 413 , 403 , 414 , 415 , 416 ] .
Recent advances in vision foundation models have revolutionized geometry reconstruction and semantic understanding. Yet, most of the existing approaches treat these capabilities in isolation, leading to redundant pipelines and compounded errors. This paper introduces FF3R, a fully annotation-free feed-forward framework that unifies geometric and semantic reasoning from unconstrained multi-view image sequences. Unlike previous methods, FF3R does not require camera poses, depth maps, or semantic labels, relying solely on rendering supervision for RGB and feature maps, establishing a scalable paradigm for unified 3D reasoning. In addition, we address two critical challenges in feedforward feature reconstruction pipelines, namely global semantic inconsistency and local structural inconsistency, through two key innovations: (i) a Token-wise Fusion Module that enriches geometry tokens with semantic context via cross-attention, and (ii) a Semantic-Geometry Mutual Boosting mechanism combining geometry-guided feature warping for global consistency with semantic-aware voxelization for local coherence. Extensive experiments on ScanNet and DL3DV-10K demonstrate FF3R's superior performance in novel-view synthesis, open-vocabulary semantic segmentation, and depth estimation, with strong generalization to in-the-wild scenarios, paving the way for embodied intelligence systems that demand both spatial and semantic understanding.
Chaoyi Zhou, Run Wang, Feng Luo +4
Microsoft · Clemson University · Texas A&M University
3D vision has rapidly evolved, driven by increasingly diverse data representations, learning paradigms, and modeling strategies. Yet the field remains fragmented across representations and benchmarks, making it difficult to develop unified perspectives on efficiency, fidelity, and scalability. This work provides a data-centric taxonomy of 3D vision that connects geometric representations, datasets, learning frameworks, and applications within a single conceptual map. We begin by analysing the principal structural representations of 3D data--point clouds, meshes, voxels, and 3D Gaussians--along with their acquisition pipelines. We then examine how dataset design, benchmark construction, and supervision regimes shape recent advances, spanning 2D-supervised 3D learning, implicit neural representations, and 4D world modeling. Through this integrative lens, we clarify the relationships among representations, learning paradigms, and downstream tasks in reconstruction, generation, and video modeling, offering a consolidated view of emerging trends toward balancing efficiency and fidelity and toward multimodal geometric grounding.
Hongyang Du, Zongxia Li, Dawei Liu +8
Brown University · University of Maryland, College Park · University of Pennsylvania +4
We present UniQueR, a unified query-based feedforward framework for efficient and accurate 3D reconstruction from unposed images. Existing feedforward models such as DUSt3R, VGGT, and AnySplat typically predict per-pixel point maps or pixel-aligned Gaussians, which remain fundamentally 2.5D and limited to visible surfaces. In contrast, UniQueR formulates reconstruction as a sparse 3D query inference problem. Our model learns a compact set of 3D anchor points that act as explicit geometric queries, enabling the network to infer scene structure, including geometry in occluded regions--in a single forward pass. Each query encodes spatial and appearance priors directly in global 3D space (instead of per-frame camera space) and spawns a set of 3D Gaussians for differentiable rendering. By leveraging unified query interactions across multi-view features and a decoupled cross-attention design, UniQueR achieves strong geometric expressiveness while substantially reducing memory and computational cost. Experiments on Mip-NeRF 360 and VR-NeRF demonstrate that UniQueR surpasses state-of-the-art feedforward methods in both rendering quality and geometric accuracy, using an order of magnitude fewer primitives than dense alternatives.
Chensheng Peng, Quentin Herau, Jiezhi Yang +6
Applied Intuition · UC Berkeley · Stanford University