Habitat-GS: A High-Fidelity Navigation Simulator with Dynamic Gaussian Splatting

Ziyuan Xia; Jingyi Xu; Chong Cui; Yuanhong Yu; Jiazhao Zhang; Qingsong Yan; Tao Ni; Junbo Chen

cs.ROApr 14, 2026

Habitat-GS: A High-Fidelity Navigation Simulator with Dynamic Gaussian Splatting

Authors: Ziyuan Xia, Jingyi Xu, Chong Cui, Yuanhong Yu, Jiazhao Zhang, Qingsong Yan, Tao Ni, Junbo Chen,

Organizations: Zhejiang University, Hangzhou, China · Peking University, Beijing, China · XGRIDS, Shenzhen, China · UDeer AI, Hangzhou, China ·

Abstract

Training embodied AI agents depends critically on the visual fidelity of simulation environments and the ability to model dynamic humans. Current simulators predominantly rely on mesh-based rasterization, for which photorealistic assets are costly to author at scale, and their support for dynamic human avatars is largely constrained to mesh representations, hindering agent generalization to human-populated real-world scenarios. We present Habitat-GS, a navigation-centric embodied AI simulator extended from Habitat-Sim that integrates 3D Gaussian Splatting scene rendering and drivable gaussian avatars while maintaining full compatibility with the Habitat ecosystem. Our system implements a 3DGS renderer for real-time photorealistic rendering and supports scalable 3DGS asset import from diverse sources. For dynamic human modeling, we introduce a gaussian avatar module that enables each avatar to simultaneously serve as a photorealistic visual entity and an effective navigation obstacle, allowing agents to learn human-aware behaviors in realistic settings. Experiments on point-goal navigation demonstrate that agents trained on 3DGS scenes achieve stronger cross-domain generalization. Evaluations on avatar-aware navigation further confirm that gaussian avatars enable effective human-aware navigation, while performance benchmarks validate the system's scalability. Code is available at https://github.com/zju3dv/habitat-gs.

Figures & tables

Figure 1: Habitat-GS is a navigation-centric embodied simulation platform with 3DGS and dynamic gaussian avatars. Compared to traditional mesh-based simulators (left), our 3DGS-based simulator (right) preserves high-frequency visual details and view-dependent effects, while gaussian avatars provide realistic and dynamic human presence for human-aware navigation scenarios.

Platform	Render Asset	Humanoid Avatar	Open-Source	GPU Req.
Habitat [ 30 , 34 , 27 ]	Mesh	Mesh (SMPL-X)	$\checkmark$	No Special Req.
iGibson [ 31 , 15 ]	Mesh	Mesh (Rigid-body)	$\checkmark$	No Special Req.
AI2-THOR [ 13 ]	Mesh	Mesh (URDF)	$\checkmark$	No Special Req.
ThreeDWorld [ 7 ]	Mesh	Mesh (Replicants)	$\checkmark$	No Special Req.
SAPIEN [ 38 ]	Mesh	Mesh (URDF)	$\checkmark$	No Special Req.
Isaac Sim [ 22 ]	Mesh+3DGS	Mesh (Sim-Ready)	Partial	Need RT Cores

Table 1: Comparison of Embodied AI simulation platforms. “GPU Req.” indicates hardware requirements beyond standard CUDA-capable GPUs. “No Special Req.” means standard GPUs are sufficient. Habitat-GS is the only platform combining 3DGS rendering with drivable gaussian avatars while remaining fully open-source and deployable on standard datacenter accelerators.

Figure 2: System overview of Habitat-GS. From left to right: Asset Preparation , where 3DGS scene assets and gaussian avatar assets are prepared respectively. Habitat-GS Simulation Environment , where the render engine performs 3DGS rasterization for scene gaussians and LBS deformation followed by rasterization for avatar gaussians, producing RGB-D observations. The NavMesh blocking module retrieves pre-computed proxy capsules at runtime and injects them into the NavMesh for obstacle blocking. Embodied Agent Loop , where the agent policy consumes sensor observations, executes actions, and receives rewards shaped by both navigation success and avatar collision.

Figure 3: Visual comparison of scene rendering. Mesh-based rendering (left) vs. our 3DGS rendering (right). Our simulator is based on 3DGS, which preserves high-frequency details and supports diverse sources of rendering assets.

Figure 5: VLM scene quality assessment. Gemini 3.0 Pro evaluates 240 rendered screenshots from each domain on three perceptual dimensions. GS scenes consistently outperform mesh scenes, confirming their superior visual fidelity and diversity.

Panel of Table 3 — Table 3: PointNav results and training curves. (a) Results across different training domain mixtures. Agents are trained on varying ratios of mesh (M) and 3DGS (G) scenes and evaluated on disjoint mesh and GS test sets. SR and SPL are in %; DTG is in meters ( $\downarrow$ ). Best and Second best results per column are highlighted. (b) Training success rate curve for 100 Mesh (Config A). (c) Training success rate curve for 100 GS (Config B).

	Self-recorded (Record3D, iPhone)						InCrowd-VI (Aria glasses)
Scene	yard	lab.	gard.	off.	park.	Avg	Zur.	BIN	UZH	TS	AND	Avg
N	9	9	2	9	9	38	3	4	8	4	6	25
A: 100 Mesh	66.2	28.6	78.5	61.0	61.4	55.6	33.8	11.5	8.4	47.9	17.1	20.4
B: 100 GS	78.1	25.2	78.5	62.6	61.8	58.1	44.4	18.8	16.9	41.1	24.1	26.1
C: 80 M + 20 G	76.1	24.5	74.0	62.1	69.2	58.8	70.3	13.3	9.7	41.4	17.2	24.4
D: 50 M + 50 G	79.3	29.2	77.0	65.7	61.7	59.9	71.5	12.5	15.3	60.3	18.8	29.6

Table 4: Real-video open-loop PointNav Evaluation. At each frame we compare the trained agent’s discrete action to the human’s actual next motion and report APA

\pm 1

(%,

\uparrow

). N is the number of episodes per scene. Best , 2nd , and 3rd in each scene column are highlighted. The mixed

>

>

mesh ordering on real video mirrors the cross-domain trend in Tab. 3 .

Panel of Table 5 — Table 5: Avatar-aware PointNav results. Top: Two training configurations are evaluated: a GS scene baseline without avatars, and a GS scene populated with GS avatars. Agents are evaluated on both mesh and GS dynamic test sets to assess in-domain performance and cross-domain generalization. SR, SPL and CR are in %. CR is the fraction of collision steps ( $\downarrow$ ). PSI is the average personal-space intrusion ( $\downarrow$ ). Bottom: Visual demonstration of our avatar-aware navigation policy. Agent successfully navigates through gaussian avatars without collision or personal space intrusion.

	Mesh	(a) Scene Scale (Gaussian Count)						(b) Avatar Count
	Ref.	300 K	500 K	1 M	3 M	5 M	7 M	0	1	2	5	10
FPS $\uparrow$	163.8	159.2	148.8	120.9	82.60	51.46	44.52	94.16	75.14	57.70	37.74	24.67
Frame time (ms) $\downarrow$	6.11	6.28	6.72	8.27	12.1	19.4	22.5	10.6	13.3	17.3	26.5	40.5
Mem (GB) $\downarrow$	2.930	3.299	3.373	3.567	3.723	4.091	4.425	3.917	4.197	4.457	5.513	7.497

Table 6: Rendering performance and GPU memory usage. Mesh Ref. reports Habitat-Sim’s native mesh rasterization on a representative HM3D scene as a baseline. (a) Varying scene Gaussian count with no avatars. (b) Varying avatar count in a medium-scale scene with approximately 2M Gaussians. Habitat-GS maintains real-time throughput under typical RL training loads with up to 5M Gaussians and

<

5 avatars.

Robotics Simulation

Robotics simulation builds virtual robots and environments to model, train, and evaluate autonomous systems before real-world deployment.

See all 320 papers

NavGSim: High-Fidelity Gaussian Splatting Simulator for Large-Scale Navigation
Mar 16, 2026Jiahang Liu, Yuanxing Duan, Jiazhao Zhang +4Robotics Simulation Robot Navigation
Peking University · Galbot · BAAI
ReaDy-Go: Real-to-Sim Dynamic 3D Gaussian Splatting Simulation for Environment-Specific Visual Navigation with Moving Obstacles
Feb 12, 2026Seungyeon Yoo, Youngseok Jang, Dabin Kim +3Robotics Simulation Robot Navigation
Department of Aerospace Engineering, Seoul National University, Seoul, Republic of Korea · InnoCORE AI-Transformed Aerospace Research Center, KAIST, Daejeon, Republic of Korea
Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator
Jul 7, 2026Zihan Wang, Seungjun Lee, Yinghao Xu +1Embodied Navigation Robotics Simulation
National University of Singapore · HKUST

Mar 16, 2026J/K move · Enter open · S save

NavGSim: High-Fidelity Gaussian Splatting Simulator for Large-Scale Navigation

Jiahang Liu, Yuanxing Duan, Jiazhao Zhang, Minghan Li+3

Peking University · Galbot · BAAI

Robotics Simulation Robot Navigation 3D Gaussian Splatting Robot Learning

Open paper AskPDF arXiv

Simulating realistic environments for robots is widely recognized as a critical challenge in robot learning, particularly in terms of rendering and physical simulation. This challenge becomes even more pronounced in navigation tasks, where trajectories often extend across multiple rooms or entire floors. In this work, we present NavGSim, a Gaussian Splatting-based simulator designed to generate high-fidelity, large-scale navigation environments. Built upon a hierarchical 3D Gaussian Splatting framework, NavGSim enables photorealistic rendering in expansive scenes spanning hundreds of square meters. To simulate navigation collisions, we introduce a Gaussian Splatting-based slice technique that directly extracts navigable areas from reconstructed Gaussians. Additionally, for ease of use, we provide comprehensive NavGSim APIs supporting multi-GPU development, including tools for custom scene reconstruction, robot configuration, policy training, and evaluation. To evaluate NavGSim's effectiveness, we train a Vision-Language-Action (VLA) model using trajectories collected from NavGSim and assess its performance in both simulated and real-world environments. Our results demonstrate that NavGSim significantly enhances the VLA model's scene understanding, enabling the policy to handle diverse navigation queries effectively.

Abstract

Robotics Simulation

Explore similar work

NavGSim: High-Fidelity Gaussian Splatting Simulator for Large-Scale Navigation

ReaDy-Go: Real-to-Sim Dynamic 3D Gaussian Splatting Simulation for Environment-Specific Visual Navigation with Moving Obstacles

Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator