Interactive VR/MR applications demand photorealistic 3D representations that support real-time rendering and object-level manipulation. While 3D Gaussian Splatting (3DGS) achieves high rendering quality, it represents scenes as unstructured, monolithic primitives. Existing object-aware 3DGS methods jointly optimize geometry and semantic states but suffer from two key failure modes: noisy foundation-model masks on complex geometry, and label-geometry coupling, in which co-optimized identity gradients mislabel boundary Gaussians. We introduce RigidGS, an end-to-end pipeline that decouples geometric learning from object identity. We propose a depth- and temporally-aware mask refinement module that addresses two complementary SAM2 failures: over-inclusion on thin structures and under-inclusion on multimodal-depth objects. We then determine object identities entirely upstream via COLMAP projection voting and freeze them during training, directing mask supervision solely toward geometric optimization. Refined masks alone substantially improve competing methods, yet a residual gap in label-geometry coupling persists that our fixed-identity design eliminates. RigidGS outperforms prior object-aware 3DGS methods on LLFF/NVOS, Mip-NeRF360, and iNeRF360, improving segmentation and boundary IoU by 7.7 and 14.5 points on LLFF/NVOS. Finally, RigidGS enables real-time rigid-body manipulation on Meta Quest2 without mesh extraction. Our code and project page are available at https://gyreyirisansah.github.io/rigidgs.
Figures & tables
Figure 1: Object extraction comparison on Bear scene from IN2N dataset [ 2 ] . RigidGS produces clean object separation with minimal to zero remnant debris or artifacts, enabling direct, mesh-free rigid-body manipulation of Gaussian primitives.
Table 1: Object segmentation quality averaged over scenes per dataset. Per-scene results appear in the supplementary material.
Figure 4: Object segmentation comparison across LLFF, Mip-NeRF 360, iNeRF360, and Tanks & Temples scenes.
Ours
GG
Scene
#Obj
IoU
bIoU
IoU
bIoU
Figurines
10
0.96
0.82
0.37
0.20
Teatime
5
0.97
0.97
0.22
0.13
Table 2: Multi-object segmentation on LERF scenes.
Scene
Configuration
IoU ↑
bIoU ↑
Acc ↑
Fern
COB-GS (Grounded-SAM2)
0.747
0.688
0.916
COB-GS + refined
0.733
0.694
0.912
RigidGS (SAM2)
0.753
0.323
0.915
RigidGS + refined
0.726
0.714
0.907
Flower
COB-GS (Grounded-SAM2)
0.952
0.833
0.989
COB-GS + refined
0.954
0.831
0.989
Table 3: Effect of mask refinement on 3DGS segmentation across all LLFF/NVOS scenes.
Variant
Fern
Fortress
Horns
Trex
Avg
Init only (COLMAP+KNN)
0.485
0.676
0.820
0.324
0.576
+ BCE (no boundary)
0.444
0.962
0.978
0.287
0.668
Full RigidGS (ours)
0.726
0.985
0.978
0.590
0.820
Full + learnable logits
0.730
0.962
0.975
0.466
0.783
Table 4: Component analysis on LLFF scenes (IoU). Boundary stabilization and frozen logits are critical for complex, multi-depth geometry ( trex , fern ).
Corruption
Fern
Trex
Garden
Kitchen
None (baseline)
0.726
0.590
0.963
0.939
bg → fg 10%
0.727
0.591
0.928
0.942
bg → fg 20%
0.726
0.592
0.928
0.941
bg → fg 30%
0.727
0.591
0.928
0.941
fg → bg 10%
0.727
0.593
0.928
0.942
fg → bg 20%
0.729
0.593
0.928
0.942
Table 5: Robustness to initial label corruption.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Gaussian Grouping
14.04
0.217
0.583
RigidGS
16.44
0.452
0.602
Table 6: SPIn-NeRF object removal [ 35 ] . In-mask metrics vs. object-free ground truth; RigidGS leads on PSNR and SSIM.
Figure 5: Object removal and background reconstruction. RigidGS produces clean removal with view-consistent background recovery while Gaussian Grouping retains ghost residue at the vacated region.
Metric
Garden
Bear
Gaussian count
4.03M
2.93M
Static FPS (mean)
32.7
32.8
Interaction FPS (mean)
25.6
26.6
Transform latency (avg)
0.19 ms
0.19 ms
Transform latency (p95)
0.23 ms
0.25 ms
Table 7: Meta Quest 2 deployment. FPS is a 90-frame rolling average; transform latency is the per-subset affine update overhead.