Dynamic-Robust Photometric-Semantic Reconstruction for Open-Vocabulary 3D Scene Understanding
Authors: Boyu Cai, Li Yang, Yan Xu, Wei Liu, Nian Liu, Sikui Zhang, Yan Wang, Chunfeng Yuan, Weiming Hu
Organizations: School of Information Science and Technology, ShanghaiTech University · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information, Institute of Automation, Chinese Academy of Sciences (CASIA) · Electronic Engineering Department, The Chinese University of Hong Kong · Deepeleph Intelligent Technology
The integration of novel view synthesis (NVS) and open-vocabulary segmentation (OVS) has recently yielded powerful feed-forward 3D foundation models. However, their inherent reliance on static-scene assumptions leads to severe misalignment of spatial features in unconstrained dynamic environments. To bridge this critical gap, we propose SPAR, a novel joint semantic-geometric encoding architecture that explicitly isolates transient dynamic noise prior to latent space aggregation. Furthermore, we introduce a dynamic-region-aware end-to-end training paradigm that structurally couples motion estimation with multi-view visual and semantic learning. This unified approach enables the network to inherently resolve motion conflicts and distill multi-view consistent, temporally stable scene representations from dynamic inputs. Extensive experiments on the challenging D-RE10K benchmark demonstrate that SPAR achieves state-of-the-art performance. Our end-to-end approach achieves exceptional novel view synthesis quality, yielding a PSNR of 22.15 dB and 23.33 dB given only 3 and 4 input views respectively. Despite being trained in a self-supervised manner, our model achieves an mIoU of 88.5% for motion mask prediction. Furthermore, our analysis reveals a strong inter-task synergy between photometric scene reconstruction and semantic understanding, where semantic synthesis learning consistently enhances photometric fidelity in novel view rendering. Code will be available at https://github.com/dmucby/SPAR.
Figures & tables
Figure 1: Comparison under dynamic, unposed settings. Baselines such as LSM [ 9 ] struggle with moving objects, producing ghosting artifacts and inconsistent semantics (left top). By masking dynamic regions prior to latent encoding, our SPAR generates clean, ghost-free photometric and semantic renderings in a single forward pass (left bottom).
Figure 2: Overview of SPAR. Our unified encoder-decoder jointly performs novel view synthesis and open-vocabulary scene understanding from unposed dynamic views. The pipeline first estimates camera poses to build pose-conditioned photometric and semantic tokens. A dynamic region predictor masks moving foregrounds to isolate static geometry. A scene encoder aggregates filtered tokens into a compact latent scene representation, which a dual-head decoder uses to render the target image and semantic features.
Method
D-RE10K
Views = 2
Views = 3
Views = 4
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Optimization-Based Methods
NeRF On-the-go [ 31 ]
15.90
0.518
0.582
18.45
0.624
0.446
19.52
0.620
0.443
3DGS [ 16 ]
13.49
0.442
0.605
14.92
0.514
0.531
16.28
0.552
0.490
T-3DGS [ 26 ]
15.90
0.518
0.582
18.45
0.624
0.446
18.70
0.613
0.454
Table 1: Performance comparison on D-RE10K under view settings ( n=2,3,4 ).
Figure 3: Qualitative comparison of novel view synthesis and semantic segmentation on dynamic scenes.
Method
n=2
n=3
n=8
mIoU ↑
Recall ↑
mIoU ↑
Recall ↑
mIoU ↑
Recall ↑
Supervised Methods
MegaSAM [ 22 ]
-
-
12.7
37.1
35.4
60.7
Segment Any Motion [ 12 ]
31.9
47.2
41.2
57.1
50.9
70.8
Ours w/ Refine
87.8
84.7
88.5
86.2
88.3
85.6
Self-supervised Methods
Table 2: Dynamic Region quality . Comparison of supervised and self-supervised motion segmentation methods under view settings ( n=2,3,8 ). w/ Refine denotes the use of SAM2 to refine the original motion masks.
Figure 4: Qualitative results of our Dynamic Region Predictor (DRP) on diverse multi-view sequences.
Figure 5: Qualitative Results of novel view synthesis and dynamic region prediction on unbounded scenes.
Method
Target View
mIoU ↑
Acc. ↑
PSNR ↑
SSIM ↑
LPIPS ↓
LSeg [ 20 ]
0.5281
0.7612
-
-
-
NeRF-DFF [ 18 ]
0.4037
0.6755
19.86
0.6650
0.3629
Feature-3DGS [ 53 ]
0.4223
0.7174
24.49
0.8132
0.2293
pixelSplat [ 2 ]
-
-
24.89
0.8392
0.1641
SRT [ 33 ]
-
-
17.01
0.4130
0.6350
Table 3: Quantitative comparison on the semantically rich ScanNet [ 8 ] dataset.
Optimization Strategy
PSNR ↑
SSIM ↑
LPIPS ↓
Unconstrained Baseline
15.66
0.489
0.532
Baseline + Random Masking
17.10 (+1.44)
0.536 (+0.047)
0.485 (-0.047)
+ Dynamic-Region-Aware Opt. (Ours)
19.97 (+4.31)
0.627 (+0.138)
0.339 (-0.193)
Table 4: Ablation of Optimization Strategies . Quantitative comparison of novel view synthesis performance under severe dynamic interference.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
w/o semantic branch
19.77
0.615
0.356
w/ semantic branch (ours)
19.97 (+0.20)
0.627 (+0.012)
0.339 (-0.017)
Table 5: Effectiveness of Semantic Branch . Quantitative comparison demonstrating the performance gains after incorporating the semantic branch.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Ours w/o pre-training
18.66
0.578
0.381
Ours w/o CPA
19.82
0.625
0.348
Ours w/o SAM2 refine
19.90
0.624
0.341
Ours
19.97
0.627
0.339
Table 6: Ablation of Auxiliary Components . We ablate pre-training, copy-paste augmentation (CPA), and SAM2 refinement.
Semantic understanding of 3D scenes is essential for robots to operate effectively and safely in complex environments. Existing methods for semantic scene reconstruction and semantic-aware novel view synthesis often rely on dense multi-view inputs and require scene-specific optimization, limiting their practicality and scalability in real-world applications. To address these challenges, we propose SemGS, a feed-forward framework for reconstructing generalizable semantic fields from sparse image inputs. SemGS uses a dual-branch architecture to extract color and semantic features, where the two branches share shallow CNN layers, allowing semantic reasoning to leverage textural and structural cues in color appearance. We also incorporate a camera-aware attention mechanism into the feature extractor to explicitly model geometric relationships between camera viewpoints. The extracted features are decoded into dual-Gaussians that share geometric consistency while preserving branch-specific attributes, and further rasterized to synthesize semantic maps under novel viewpoints. Additionally, we introduce a regional smoothness loss to enhance semantic coherence. Experiments show that SemGS achieves state-of-the-art performance on benchmark datasets, while providing rapid inference and strong generalization capabilities across diverse synthetic and real-world scenarios.
Sheng Ye, Zhen-Hui Dong, Ruoyu Fan +2
Department of Computer Science, Tsinghua University, Beijing, China
Recent advances in vision foundation models have revolutionized geometry reconstruction and semantic understanding. Yet, most of the existing approaches treat these capabilities in isolation, leading to redundant pipelines and compounded errors. This paper introduces FF3R, a fully annotation-free feed-forward framework that unifies geometric and semantic reasoning from unconstrained multi-view image sequences. Unlike previous methods, FF3R does not require camera poses, depth maps, or semantic labels, relying solely on rendering supervision for RGB and feature maps, establishing a scalable paradigm for unified 3D reasoning. In addition, we address two critical challenges in feedforward feature reconstruction pipelines, namely global semantic inconsistency and local structural inconsistency, through two key innovations: (i) a Token-wise Fusion Module that enriches geometry tokens with semantic context via cross-attention, and (ii) a Semantic-Geometry Mutual Boosting mechanism combining geometry-guided feature warping for global consistency with semantic-aware voxelization for local coherence. Extensive experiments on ScanNet and DL3DV-10K demonstrate FF3R's superior performance in novel-view synthesis, open-vocabulary semantic segmentation, and depth estimation, with strong generalization to in-the-wild scenarios, paving the way for embodied intelligence systems that demand both spatial and semantic understanding.
Chaoyi Zhou, Run Wang, Feng Luo +4
Microsoft · Clemson University · Texas A&M University
We present BetterScene, an approach to enhance novel view synthesis (NVS) quality for diverse real-world scenes using extremely sparse, unconstrained photos. BetterScene leverages the production-ready Stable Video Diffusion (SVD) model pretrained on billions of frames as a strong backbone, aiming to mitigate artifacts and recover view-consistent details at inference time. Conventional methods have developed similar diffusion-based solutions to address these challenges of novel view synthesis. Despite significant improvements, these methods typically rely on off-the-shelf pretrained diffusion priors and fine-tune only the UNet module while keeping other components frozen, which still leads to inconsistent details and artifacts even when incorporating geometry-aware regularizations like depth or semantic conditions. To address this, we investigate the latent space of the diffusion model and introduce two components: (1) temporal equivariance regularization and (2) vision foundation model-aligned representation, both applied to the variational autoencoder (VAE) module within the SVD pipeline. BetterScene integrates a feed-forward 3D Gaussian Splatting (3DGS) model to render features as inputs for the SVD enhancer and generate continuous, artifact-free, consistent novel views. We evaluate on the challenging DL3DV-10K dataset and demonstrate superior performance compared to state-of-the-art methods.
Yuci Han, Charles Toth, John E. Anderson +2
Dept. of Electrical and Computer Engineering, The Ohio State University · USACE ERDC GRL