Training-Free Instance-Aware 3D Scene Reconstruction and Diffusion-Based View Synthesis from Sparse Images

Jiatong Xia; Lingqiao Liu

cs.CVMar 22, 2026

Training-Free Instance-Aware 3D Scene Reconstruction and Diffusion-Based View Synthesis from Sparse Images

Authors: Jiatong Xia, Lingqiao Liu

Organizations: The University of Adelaide, Australia

Abstract

We introduce a novel, training-free system for reconstructing, understanding, and rendering 3D indoor scenes from a sparse set of unposed RGB images. Unlike traditional radiance field approaches that require dense views and per-scene optimization, our pipeline achieves high-fidelity results without any training or pose preprocessing. The system integrates three key innovations: (1) A robust point cloud reconstruction module that filters unreliable geometry using a warping-based anomaly removal strategy; (2) A warping-guided 2D-to-3D instance lifting mechanism that propagates 2D segmentation masks into a consistent, instance-aware 3D representation; and (3) A novel rendering approach that projects the point cloud into new views and refines the renderings with a 3D-aware diffusion model. Our method leverages the generative power of diffusion to compensate for missing geometry and enhances realism, especially under sparse input conditions. We further demonstrate that object-level scene editing such as instance removal can be naturally supported in our pipeline by modifying only the point cloud, enabling the synthesis of consistent, edited views without retraining. Our results establish a new direction for efficient, editable 3D content generation without relying on scene-specific optimization. Project page: https://jiatongxia.github.io/TID3R/

Figures & tables

Figure 1. We present a novel, unified pipeline that transforms sparse image inputs into a clean, instance-aware point cloud without requiring any pose pre-processing or scene-specific learning, and can directly synthesizes photorealistic novel views through a tailored diffusion-based rendering approach. Beyond reconstruction and rendering, our method also showing good potential supports essential downstream tasks such as scene-level editing.

Figure 2. Framework . We apply a feed-forward a feed-forward model (e.g., MV-DUSt3R) to predict the initial point cloud from the input unposed sparse images, followed by a novel warping-based anomaly points removal strategy to eliminate unreliable points in the initial point cloud, forming a clean and accurate scene point cloud. For 2D-to-3D instance segmentation, we first utilize a foundational segmentation model (e.g., SAM) to generate initial segmentation masks. We then introduce a warping-based instance unification strategy to align instances across frames. By associating the instance masks with the corresponding pointmaps, we obtain an instance-segmented 3D point cloud. The instance-segmented 3D point cloud facilitates downstream tasks such as object-level editing. Finally, the point cloud can be directly rendered into high-quality 2D images through our designed diffusion-based rendering pipeline.

Figure 3. Anomaly points . The red box highlights anomaly points, which exhibit abnormal depth values.

Figure 4. Main results . We present the main experimental results, including instance segmentation, point cloud reconstruction, novel view synthesis, and scene edits through object removal. Each row corresponds to a test scene. The left column shows the instance segmentation results. The center section labeled “Original” displays the reconstructed point cloud (top) and novel view synthesis results (bottom, with ground truth images on the top and rendered results on the bottom) of original input images. The two columns on the right illustrate two different object removal results, with the removed objects are highlighted with black dashed boxes in the point clouds

Method	AP ↑	AP50 ↑	AP75 ↑
DEVA ( Cheng et al., 2023 )	13.9	25.9	11.7
PE3R ( Hu et al., 2025 )	20.2	32.6	22.3
w/o Warping-based unifi.	17.0	26.8	18.1
Ours	25.0	41.8	25.4

Table 1. Instance segmentation evaluation . We show overall instance segmentation performance using AP, AP50, and AP75 metrics on the selected scenes.

Method	RMSE ↓	$\delta<1.25$ ↑
DepthAnythingV2 ( Yang et al., 2024b )	0.577	0.689
PE3R ( Hu et al., 2025 )	0.378	0.905
MV-DUSt3R+ ( Tang et al., 2025 )	0.219	0.977
Ours	0.209	0.981

Table 2. Depth evaluation . We show the overall video depth evaluation results on selected reconstruction scenes.

Figure 5. Novel views rendering . We show the visualization comparisons on novel view synthesis.

	Poses	Training	Novel View Synthesis
Method	Free	Free	PSNR ↑	SSIM ↑	LPIPS ↓
3DGS ( Kerbl et al., 2023 )	✗	✗	15.61	0.602	0.541
2DGS ( Huang et al., 2024 )	✗	✗	15.55	0.591	0.544
GaussianGroup. ( Ye et al., 2024 )	✗	✗	15.72	0.610	0.532
DNGaussian ( Li et al., 2024 )	✗	✗	16.82	0.627	0.549
CoR-GS ( Zhang et al., 2024 )	✗	✗	17.12	0.631	0.496

Table 3. Rendering evaluation on novel views . We show the overall novel views synthesis performance of methods on selected reconstruction scenes. We indicate whether each method requires camera parameter input or per-scene training. We apply our rendering pipeline to the reconstructed point clouds produced by PE3R and MV-DUSt3R for a comparison.

Figure 6. Object removal comparison . We present the visualization results of removing the same object under two different views, with the original ground-truth image on the left as reference. For each method, we display the rendered results before and after object removal. The removed object is highlighted with a red box in our results.

Figure 7. Instance segmentation . We present a visual comparison between our 2D-to-3D instance segmentation results and those of PE3R, with the reference raw 3D point cloud on the left. While PE3R’s reconstructed point cloud appears to be of lower quality than ours, it is worth noting that their segmentation results are produced independently and are not affected by the reconstruction quality.

Method	Poses	Reconstruct	Inference	Total
NeRF ( Mildenhall et al., 2020 )	2 mins	146 mins	1.5 mins	149.5 mins
3DGS ( Kerbl et al., 2023 )	2 mins	7 mins	-	9 mins
Ours	-	1 min	2.5 mins	3.5 mins

Table 4. Runtime costs . We show a breakdown of processing cost comparison including camera poses processing, scene reconstruction and views inference between our overall pipeline and conventional approaches.

Figure 8. Anomaly points elimination . We show the reconstruction, warping, and rendering visualization results before and after anomaly point elimination. On the left, the baseline method without anomaly point removal shows a large number of outlier points in the reconstructed point cloud, which significantly degrades the warping and final rendering results. In contrast, our proposed method effectively eliminates these anomaly points, resulting in cleaner reconstructions and better rendering quality.

Figure 9. Rendering masks .We show the object removal results with and without applying rendering masks to the source images. Without rendering masks, the rendered outputs tend to preserve the full geometry from the source images, even after the object has been removed from the point cloud. This leads to a failure in reflecting the intended object removal in the final rendered image.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 10. Editing tasks example . We show the original image of the view on the left. We apply a object point-cloud translation (red arrow) in the first row, followed by warping and rendering. We modify the object (red box) color of the input images before reconstruction, and the final rendered results reflect the applied color changes.

Indoor Scene Reconstruction

Indoor scene reconstruction builds 3D models of rooms and homes from images, video, depth, or sensor scans for mapping, simulation, and robotics.

See all 39 papers

AnyRecon: Arbitrary-View 3D Reconstruction with Video Diffusion Model
Apr 21, 2026Yutian Chen, Shi Guo, Renbiao Jin +7Video Diffusion Models 3D Reconstruction
Shanghai AI Lab · Shanghai Jiao Tong University · The University of Hong Kong +2
Generalizable Sparse-View 3D Reconstruction from Unconstrained Images
Apr 30, 2026Vinayak Gupta, Chih-Hao Lin, Shenlong Wang +23D Gaussian Splatting 3D Reconstruction
University of Maryland, College Park · University of Illinois Urbana-Champaign · Johns Hopkins University
Sparse auto-regressive modeling for scene generation from multi-view images
Sep 3, 2026Thomas Lucas, Maxime Pietrantoni, Philippe Weinzaepfel +4Sparse-View 3D Reconstruction 3D Scene Generation
1NAVER LABS Europe, France · 2NAVER LABS, South Korea · 3Carnegie Mellon University, USA

Apr 21, 2026J/K move · Enter open · S save

AnyRecon: Arbitrary-View 3D Reconstruction with Video Diffusion Model

Yutian Chen, Shi Guo, Renbiao Jin, Tianshuo Yang+6

Shanghai AI Lab · Shanghai Jiao Tong University · The University of Hong Kong+2

Video Diffusion Models 3D Reconstruction Novel View Synthesis Sparse-View 3D Reconstruction Large-Scale 3D Reconstruction

Open paper AskPDF arXiv

Sparse-view 3D reconstruction is essential for modeling scenes from casual captures, but remain challenging for non-generative reconstruction. Existing diffusion-based approaches mitigates this issues by synthesizing novel views, but they often condition on only one or two capture frames, which restricts geometric consistency and limits scalability to large or diverse scenes. We propose AnyRecon, a scalable framework for reconstruction from arbitrary and unordered sparse inputs that preserves explicit geometric control while supporting flexible conditioning cardinality. To support long-range conditioning, our method constructs a persistent global scene memory via a prepended capture view cache, and removes temporal compression to maintain frame-level correspondence under large viewpoint changes. Beyond better generative model, we also find that the interplay between generation and reconstruction is crucial for large-scale 3D scenes. Thus, we introduce a geometry-aware conditioning strategy that couples generation and reconstruction through an explicit 3D geometric memory and geometry-driven capture-view retrieval. To ensure efficiency, we combine 4-step diffusion distillation with context-window sparse attention to reduce quadratic complexity. Extensive experiments demonstrate robust and scalable reconstruction across irregular inputs, large viewpoint gaps, and long trajectories.

Abstract

Indoor Scene Reconstruction

Explore similar work

AnyRecon: Arbitrary-View 3D Reconstruction with Video Diffusion Model

Generalizable Sparse-View 3D Reconstruction from Unconstrained Images

Sparse auto-regressive modeling for scene generation from multi-view images