Open-vocabulary panoptic reconstruction is crucial for advanced robotics and simulation. However, existing 3D reconstruction methods, such as NeRF or Gaussian Splatting variants, often struggle to achieve the real-time inference frequency required by robotic control loops. Existing methods incur prohibitive latency when processing the high-dimensional features required for robust open-vocabulary segmentation. We propose Fast-SegSim, a novel, simple, and end-to-end framework built upon 2D Gaussian Splatting, designed to realize real-time, high-fidelity, and 3D-consistent open-vocabulary segmentation reconstruction. Our core contribution is a highly optimized rendering pipeline that specifically addresses the computational bottleneck of high-channel segmentation feature accumulation. We introduce two key optimizations: Precise Tile Intersection to reduce rasterization redundancy, and a novel Top-K Hard Selection strategy. This strategy leverages the geometric sparsity inherent in the 2D Gaussian representation to greatly simplify feature accumulation and alleviate bandwidth limitations, achieving render rates exceeding 40 FPS. Fast-SegSim provides critical value in robotic applications: it serves both as a high-frequency sensor input for simulation platforms like Gazebo, and its 3D-consistent outputs provide essential multi-view 'ground truth' labels for fine-tuning downstream perception tasks. We demonstrate this utility by using the generated labels to fine-tune the perception module in object goal navigation, successfully doubling the navigation success rate. Our superior rendering speed and practical utility underscore Fast-SegSim's potential to bridge the sim-to-real gap.
Figures & tables
Figure 1: Our real-time rendering speed enables robots to acquire realistic sensor observations and segmentation GT with low latency. The 3D-Consistent GT labels are critical for fine-tuning the Perception Module, effectively bridging the simulation and real-world gaps in embodied AI.
Figure 2: The Input uses multi-view RGB-D frames alongside 2D supervision generated by VLM. During Training, the Gaussian Feature Field is learned, where the semantic feature ( fsem ) is processed by a Semantic Decoder, and the instance feature ( fins ) is refined via Query-guided Cross Attention. For Inference, our highly optimized rendering pipeline is employed. The key to real-time performance is the Top-K Hard Selection strategy, which leverages the geometric sparsity inherent in Gaussian Splatting to significantly reduce the computational cost of high-dimensional feature accumulation within the SnugBox and AccuTile rasterization process, enabling real-time multi-modal rendering.
Method
ScanNet
ScanNet++
PQ ↑
SQ ↑
RQ ↑
mIoU ↑
mAcc ↑
mCov ↑
mW-Cov ↑
PQ ↑
SQ ↑
RQ ↑
mIoU ↑
mAcc ↑
mCov ↑
mW-Cov ↑
Panoptic Lifting
57.86
61.96
85.31
67.91
78.59
45.88
59.93
71.14
77.48
88.14
81.34
89.67
56.17
68.51
Contrastive Lift
37.35
41.91
57.60
64.77
75.80
13.21
23.26
47.58
57.23
65.81
81.09
89.30
27.39
36.51
PVLFF
30.11
51.71
44.43
55.41
63.96
45.75
48.41
52.24
66.86
65.56
62.53
70.31
67.95
75.47
Gaussian Grouping
43.75
50.63
72.68
58.05
68.68
52.70
58.10
33.10
40.60
67.27
59.53
68.13
29.83
36.83
OpenGaussian
48.73
51.48
88.10
54.05
68.43
44.43
49.60
51.03
56.93
85.73
61.80
73.97
50.00
51.02
Table 1: Panoptic Segmentation quality using different methods
Figure 3: Comparison of Fast-SegSim before and after acceleration: while significantly reducing the number of rendered Gaussians and substantially increasing rendering speed, segmentation quality remains highly consistent.
Exp
Time ↓
FPS ↑
RN-Total ↓
RN / Tile ↓
PQ ↑
(A)
46
22
6.1M
1230
74.75
(B)
40
25
3.9M
800
74.73
(C)
27
37
1.2M
610
74.17
(D)
22
45
1.2M
580
74.12
Table 2: Impact of Precise Tile Intersection and Top- K Hard Selection on Rendering Inference Speed.
Scene
SR -Before
SR -After
Laboratory Room
20%
100%
House
25%
50%
Table 3: Impact of finetuning with 3D-consistent label ground-truth from Fast-SegSim on navigation task.
Figure 4: Real-time Sensor Simulation with Fast-SegSim.
Open-vocabulary panoptic reconstruction is essential for advanced robotics perception and simulation. However, existing methods based on 3D Gaussian Splatting (3DGS) often struggle to simultaneously achieve geometric accuracy, coherent panoptic understanding, and real-time inference frequency in large-scale scenes. In this paper, we propose a comprehensive framework that integrates geometric reinforcement, end-to-end panoptic learning, and efficient rendering. First, to ensure physical realism in large-scale environments, we leverage LiDAR data to construct plane-constrained multimodal Gaussian Mixture Models (GMMs) and employ 2D Gaussian surfels as the map representation, enabling high-precision surface alignment and continuous geometric supervision. Building upon this, to overcome the error accumulation and cumbersome cross-frame association inherent in traditional multi-stage panoptic segmentation pipelines, we design a query-guided end-to-end learning architecture. By utilizing a local cross-attention mechanism within the view frustum, the system lifts 2D mask features directly into 3D space, achieving globally consistent panoptic understanding. Finally, addressing the computational bottlenecks caused by high-dimensional semantic features, we introduce Precise Tile Intersection and a Top-K Hard Selection strategy to optimize the rendering pipeline. Experimental results demonstrate that our system achieves superior geometric and panoptic reconstruction quality in large-scale scenes while maintaining an inference rate exceeding 40 FPS, meeting the real-time requirements of robotic control loops.
Xuan Yu, Yuxuan Xie, Changjian Jiang +4
Department of Control Science and Engineering, Zhejiang University, Hangzhou, Zhejiang, P. R. China
We propose Semantic-Fast-SAM (SFS), a semantic segmentation framework that combines the Fast Segment Anything model with a semantic labeling pipeline to achieve real-time performance without sacrificing accuracy. FastSAM is an efficient CNN-based re-implementation of the Segment Anything Model (SAM) that runs much faster than the original transformer-based SAM. Building upon FastSAM's rapid mask generation, we integrate a Semantic-Segment-Anything (SSA) labeling strategy to assign meaningful categories to each mask. The resulting SFS model produces high-quality semantic segmentation maps at a fraction of the computational cost and memory footprint of the original SAM-based approach. Experiments on Cityscapes and ADE20K benchmarks demonstrate that SFS matches the accuracy of prior SAM-based methods (mIoU ~ 70.33 on Cityscapes and 48.01 on ADE20K) while achieving approximately 20x faster inference than SSA in the closed-set setting. We also show that SFS effectively handles open-vocabulary segmentation by leveraging CLIP-based semantic heads, outperforming recent open-vocabulary models on broad class labeling. This work enables practical real-time semantic segmentation with the "segment-anything" capability, broadening the applicability of foundation segmentation models in robotics scenarios. The implementation is available at https://github.com/KBH00/Semantic-Fast-SAM.
We present Genie Sim PanoRecon, a feed-forward Gaussian-splatting pipeline that delivers high-fidelity, low-cost 3D scenes for robotic manipulation simulation. The panorama input is decomposed into six non-overlapping cube-map faces, processed in parallel, and seamlessly reassembled. To guarantee geometric consistency across views, we devise a depth-aware fusion strategy coupled with a training-free depth-injection module that steers the monocular feed-forward network to generate coherent 3D Gaussians. The whole system reconstructs photo-realistic scenes in seconds and has been integrated into Genie Sim - a LLM-driven simulation platform for embodied synthetic data generation and evaluation - to provide scalable backgrounds for manipulation tasks. For code details, please refer to: https://github.com/AgibotTech/genie_sim/tree/main/source/geniesim_world.