Dynamic-Robust Photometric-Semantic Reconstruction for Open-Vocabulary 3D Scene Understanding
Authors: Boyu Cai, Li Yang, Yan Xu, Wei Liu, Nian Liu, Sikui Zhang, Yan Wang, Chunfeng Yuan, Weiming Hu
Organizations: School of Information Science and Technology, ShanghaiTech University · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information, Institute of Automation, Chinese Academy of Sciences (CASIA) · Electronic Engineering Department, The Chinese University of Hong Kong · Deepeleph Intelligent Technology
The integration of novel view synthesis (NVS) and open-vocabulary segmentation (OVS) has recently yielded powerful feed-forward 3D foundation models. However, their inherent reliance on static-scene assumptions leads to severe misalignment of spatial features in unconstrained dynamic environments. To bridge this critical gap, we propose SPAR, a novel joint semantic-geometric encoding architecture that explicitly isolates transient dynamic noise prior to latent space aggregation. Furthermore, we introduce a dynamic-region-aware end-to-end training paradigm that structurally couples motion estimation with multi-view visual and semantic learning. This unified approach enables the network to inherently resolve motion conflicts and distill multi-view consistent, temporally stable scene representations from dynamic inputs. Extensive experiments on the challenging D-RE10K benchmark demonstrate that SPAR achieves state-of-the-art performance. Our end-to-end approach achieves exceptional novel view synthesis quality, yielding a PSNR of 22.15 dB and 23.33 dB given only 3 and 4 input views respectively. Despite being trained in a self-supervised manner, our model achieves an mIoU of 88.5% for motion mask prediction. Furthermore, our analysis reveals a strong inter-task synergy between photometric scene reconstruction and semantic understanding, where semantic synthesis learning consistently enhances photometric fidelity in novel view rendering. Code will be available at https://github.com/dmucby/SPAR.
Figures & tables
Figure 1: Comparison under dynamic, unposed settings. Baselines such as LSM [ 9 ] struggle with moving objects, producing ghosting artifacts and inconsistent semantics (left top). By masking dynamic regions prior to latent encoding, our SPAR generates clean, ghost-free photometric and semantic renderings in a single forward pass (left bottom).
Figure 2: Overview of SPAR. Our unified encoder-decoder jointly performs novel view synthesis and open-vocabulary scene understanding from unposed dynamic views. The pipeline first estimates camera poses to build pose-conditioned photometric and semantic tokens. A dynamic region predictor masks moving foregrounds to isolate static geometry. A scene encoder aggregates filtered tokens into a compact latent scene representation, which a dual-head decoder uses to render the target image and semantic features.
Method
D-RE10K
Views = 2
Views = 3
Views = 4
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Optimization-Based Methods
NeRF On-the-go [ 31 ]
15.90
0.518
0.582
18.45
0.624
0.446
19.52
0.620
0.443
3DGS [ 16 ]
13.49
0.442
0.605
14.92
0.514
0.531
16.28
0.552
0.490
T-3DGS [ 26 ]
15.90
0.518
0.582
18.45
0.624
0.446
18.70
0.613
0.454
Table 1: Performance comparison on D-RE10K under view settings ( n=2,3,4 ).
Figure 3: Qualitative comparison of novel view synthesis and semantic segmentation on dynamic scenes.
Method
n=2
n=3
n=8
mIoU ↑
Recall ↑
mIoU ↑
Recall ↑
mIoU ↑
Recall ↑
Supervised Methods
MegaSAM [ 22 ]
-
-
12.7
37.1
35.4
60.7
Segment Any Motion [ 12 ]
31.9
47.2
41.2
57.1
50.9
70.8
Ours w/ Refine
87.8
84.7
88.5
86.2
88.3
85.6
Self-supervised Methods
Table 2: Dynamic Region quality . Comparison of supervised and self-supervised motion segmentation methods under view settings ( n=2,3,8 ). w/ Refine denotes the use of SAM2 to refine the original motion masks.
Figure 4: Qualitative results of our Dynamic Region Predictor (DRP) on diverse multi-view sequences.
Figure 5: Qualitative Results of novel view synthesis and dynamic region prediction on unbounded scenes.
Method
Target View
mIoU ↑
Acc. ↑
PSNR ↑
SSIM ↑
LPIPS ↓
LSeg [ 20 ]
0.5281
0.7612
-
-
-
NeRF-DFF [ 18 ]
0.4037
0.6755
19.86
0.6650
0.3629
Feature-3DGS [ 53 ]
0.4223
0.7174
24.49
0.8132
0.2293
pixelSplat [ 2 ]
-
-
24.89
0.8392
0.1641
SRT [ 33 ]
-
-
17.01
0.4130
0.6350
Table 3: Quantitative comparison on the semantically rich ScanNet [ 8 ] dataset.
Optimization Strategy
PSNR ↑
SSIM ↑
LPIPS ↓
Unconstrained Baseline
15.66
0.489
0.532
Baseline + Random Masking
17.10 (+1.44)
0.536 (+0.047)
0.485 (-0.047)
+ Dynamic-Region-Aware Opt. (Ours)
19.97 (+4.31)
0.627 (+0.138)
0.339 (-0.193)
Table 4: Ablation of Optimization Strategies . Quantitative comparison of novel view synthesis performance under severe dynamic interference.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
w/o semantic branch
19.77
0.615
0.356
w/ semantic branch (ours)
19.97 (+0.20)
0.627 (+0.012)
0.339 (-0.017)
Table 5: Effectiveness of Semantic Branch . Quantitative comparison demonstrating the performance gains after incorporating the semantic branch.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Ours w/o pre-training
18.66
0.578
0.381
Ours w/o CPA
19.82
0.625
0.348
Ours w/o SAM2 refine
19.90
0.624
0.341
Ours
19.97
0.627
0.339
Table 6: Ablation of Auxiliary Components . We ablate pre-training, copy-paste augmentation (CPA), and SAM2 refinement.