Digital human generation has been studied for decades and supports a wide range of real-world applications. However, most existing systems are passively animated, relying on privileged state or scripted control, which limits scalability to novel environments. We instead ask: how can digital humans actively behave using only visual observations and specified goals in novel scenes? Achieving this would enable populating any 3D environments with digital humans at scale that exhibit spontaneous, natural, goal-directed behaviors. To this end, we introduce Visually-grounded Humanoid Agents, a coupled two-layer (world-agent) paradigm that replicates humans at multiple levels: they look, perceive, reason, and behave like real people in real-world 3D scenes. The World Layer reconstructs semantically rich 3D Gaussian scenes from real-world videos via an occlusion-aware pipeline and accommodates animatable Gaussian-based human avatars. The Agent Layer transforms these avatars into autonomous humanoid agents, equipping them with first-person RGB-D perception and enabling them to perform accurate, embodied planning with spatial awareness and iterative reasoning, which is then executed at the low level as full-body actions to drive their behaviors in the scene. We further introduce a benchmark to evaluate humanoid-scene interaction in diverse reconstructed environments. Experiments show our agents achieve robust autonomous behavior, yielding higher task success rates and fewer collisions than ablations and state-of-the-art planning methods. This work enables active digital human population and advances human-centric embodied AI. Data, code, and models will be open-sourced.
Figures & tables
Figure 2: Framework Overview. Our framework consists of two layers. The World Layer processes real-world data (scene videos, object assets, human videos) to build large-scale, semantically detailed environments via occlusion-aware reconstruction, and populates them with GS-based animatable human avatars ( Sec. 3.1 ). Then the Agent Layer drives these avatars for human-scene interaction via a perception-action loop, where visually-grounded agents plan actions from egocentric observations ( Sec. 3.2 ).
Figure 3: Overview of the Occlusion-Aware Semantic Scene Reconstruction. We first reconstruct 3D Gaussians from scene videos utilizing CityGaussian [ 47 , 48 ] . To augment 3DGS with instance-level semantics, we extract 2D masks B based on SAM [ 36 ] , lift them to 3D via contrastive learning, and then segment 3D instances using coarse-to-fine quantization. We introduce occlusion-aware masks and view selection to boost segmentation accuracy in large-scale, occluded outdoor scenes. Finally, each instance is annotated via context-aware visual prompting with a VLM [ 4 ] , yielding a semantically rich environment with spatially annotated landmarks, ready for human-scene interaction.
Figure 4: Our Visually Grounded Humanoid Agent comprises a two-level framework: (1) A context-aware action planning module (high-level planner) that predicts actions from ego-centric observations. It utilizes spatial-aware visual prompting to generate physically viable, spatially grounded proposals and apply goal highlighting for contextual cues, combined with iterative reasoning for multi-step decision making ( Sec. 3.2 ). (2) A controllable motion generation module (low-level controller) that converts the planner’s command into waypoints, which then condition a motion diffusion model to synthesize full-body motion ( Sec. 3.2 ).
Scene
World Eval.
Agent Eval.
Universality Eval.
Recon. Quality
Inter. Range
Sem. Richness
Coll. Mesh
SmallCity [ 34 ]
✓
✓
✓
High
Massive
High
✓
XGRIDS [ 104 ]
✓
✓
Ultra
Massive
High
✓
SAGE-3D [ 56 ]
✓
✓
High
Room-scale
Medium
✓
ArgoVerse2 [ 101 ] , PandaSet [ 106 ]
✓
Medium
Large
Vehicle-only
✓
Mip-NeRF360 [ 5 ] , DL3DV-10K [ 44 ]
✓
Medium
Limited
Low/Medium
✓
MatrixCity [ 39 ] , Horizon-GS [ 26 ]
✓
Low
Massive
High
✓
Table 1: Settings of the scenes used in experiments. We summarize the roles of experiment scenes in world-layer and agent-layer evaluation together with key scene properties.
Task
SimNav
ObstNav
SocialNav
Method
SR ↑
SPL ↑
CR ↓
SR ↑
SPL ↑
CR ↓
SR ↑
SPL ↑
CR ↓
NaVILA [ 11 ]
22.5%
0.199
70.8%
20.8%
0.176
75.0%
8.3%
0.072
84.1%
NaVid [ 114 ]
37.4%
0.279
19.2%
32.5%
0.233
23.1%
17.5%
0.138
51.7%
Uni-NaVid [ 113 ]
38.8%
0.370
12.8%
25.3%
0.242
29.4%
12.5%
0.112
66.7%
Ours
68.3%
0.640
13.3%
55.8%
0.516
30.8%
39.2%
0.366
48.3%
Table 2: Quantitative comparison on SmallCity dataset using our navigation benchmarks. Our method significantly outperforms three SOTA VLN baselines, improving task success rates by roughly 30% across all tasks. Top two results are highlighted by first and second .
Benchmark
XGRIDS
SAGE-3D
Multi-Goal
Method
SR ↑
SPL ↑
CR ↓
SR ↑
SPL ↑
CR ↓
SR ↑
PR ↑
PPL ↑
CR ↓
NaVILA [ 11 ]
17.8%
0.160
71.1%
31.7%
0.293
67.2%
0.0%
21.6%
0.102
78.9 %
NaVid [ 114 ]
40.0%
0.316
35.6%
33.1%
0.324
58.4%
0.0%
17.5%
0.096
70.2 %
Uni-NaVid [ 113 ]
31.1%
0.298
48.9%
25.1%
0.228
35.2%
12.3 %
8.3 %
0.066
48.9 %
Ours
74.7%
0.725
14.7%
58.2%
0.534
33.3%
38.0 %
73.6 %
0.707
34.8 %
Table 3: Generalization on diverse environments and multi-goal evaluation. We evaluate on two outdoor scenes from XGRIDS [ 104 ] , four indoor scenes from SAGE-3D [ 56 ] , and a multi-goal benchmark with 2–5 consecutive landmarks per episode.
Method
SimNav
ObstNav
SocialNav
VP
IR
SR ↑
SPL ↑
CR ↓
SR ↑
SPL ↑
CR ↓
SR ↑
SPL ↑
CR ↓
(a)
✗
✗
53.3 %
0.519
46.7 %
38.3 %
0.365
60.0 %
30.3 %
0.287
69.1 %
(b)
✓
✗
57.5 %
0.501
32.5 %
46.7 %
0.395
47.5 %
25.8 %
0.247
59.2 %
(c)
✗
✓
49.2 %
0.445
7.6 %
44.2 %
0.387
17.7 %
31.7 %
0.274
51.8 %
(d)
✓
✓
68.3 %
0.640
13.3 %
55.8 %
0.516
30.8 %
39.2 %
0.366
48.3 %
Table 4: Ablation study of the VLM-based planning paradigm. VP = Spatial-Aware Visual Prompting; IR = Iterative Reasoning. VP enhances visual cues for goal localization, while IR prevents short-sighted decision-making. The top two results are highlighted.
Figure 5: Qualitative ablation of the VLM-based planning paradigm. Without visual prompting, the agent loses track of the goal after detouring around obstacles. Without iterative reasoning, it follows myopic straight-line paths, leading to frequent collisions. Our full model combines both to produce robust, goal-directed trajectories.
Method
SimNav
ObstNav
SR ↑
SPL ↑
CR ↓
SR ↑
SPL ↑
CR ↓
Uni-NaVid (Default) [ 113 ]
38.8 %
0.370
12.8 %
25.3 %
0.242
29.4 %
Ours (Default)
68.3 %
0.640
13.3 %
55.8 %
0.516
30.8 %
Ours (Run)
56.7 %
0.518
35.8 %
41.6 %
0.382
54.2 %
Ours (Slow)
67.5 %
0.595
14.5 %
56.4 %
0.507
29.7 %
Table 5: Navigation performance under different locomotion styles. Evaluated on SimNav and ObstNav of the SmallCity [ 34 ] scene with varying motion types.
Figure 6: Qualitative results across diverse environments.
Metric
Baselines
Ablation (Ours)
F-3DGS
GW-3DGS
OpenGS
(a) Base
(b) +Occ
(c) +ViewSel
mIoU ↑
0.256
0.025
0.483
0.518
0.556
0.601
mAcc ↑
0.647
0.109
0.675
0.636
0.699
0.733
Table 6: Quantitative evaluation of semantic scene reconstruction. Comparison with Feature-3DGS [ 121 ] , GW-3DGS [ 30 ] , and OpenGaussian [ 103 ] , along with ablation of our proposed components in the World Layer. The best results are highlighted.
Figure 7: Qualitative Ablation on occlusion-aware semantic scene reconstruction. Our framework achieves precise 3D instance segmentation with well-defined boundaries, demonstrating robustness to severe occlusion while successfully recognizing thin or small objects in large-scale outdoor scenes. Zoom in for details.
Figure A1 : Effect of normal regularization on collision mesh extraction. With normal regularization, the surface becomes smoother, enabling more reliable collision detection.
Figure A2 : Visualizations of Context-Aware Instance Annotation. For the VLM visual prompt, the target object is highlighted by a red contour, while the background is dimmed to suppress visual distraction while retaining spatial context. The overlaid text shows the detailed descriptions generated by the annotation pipeline.
Figure A3 : Word cloud visualization of object semantic annotations across representative benchmark scenes. For SmallCity , annotations are generated by our automated open-vocabulary pipeline; for XGRIDS , semantic goals are manually annotated; for SAGE-3D , 3D bounding box annotations are provided by the original dataset. Larger words indicate more frequently occurring object categories.
Figure A4 : Visual Prompting for Social Navigation. In dynamic environments, we extend our visual prompting strategy to mark pedestrians. Humans are detected and highlighted with red bounding boxes labeled ‘HUMAN’, distinct from the green ‘GOAL’ box. These explicit visual cues allow a pre-trained VLM to reason in a zero-shot manner about the spatial relationship between the agent’s planned path and nearby moving obstacles, thereby supporting safer navigation decisions.
Figure A5 : World-frame alignment for human-scene contact. Left: raw asset with unaligned orientation. Right: aligned coordinates used for ground-height estimation and human placement.
Figure B1 : Bird’s-eye-view of outdoor benchmark scenes. Each row shows a scene with its 3DGS rendering (left) and extracted collision mesh (right). From top to bottom: SmallCity [ 34 ] , Church and Town from XGRIDS [ 104 ] .
Figure B2 : Bird’s-eye-view of indoor benchmark scenes from SAGE-3D [ 56 ] . Four room-scale indoor environments used for agent-layer generalization evaluation, covering diverse functional spaces. Left: 3DGS rendering. Right: collision mesh.
Figure B3 : Visualization of single-goal navigation tasks. Each row shows a test scenario with the initial ego-centric view (left), back view (middle), and BEV occupancy map (right) marking the agent’s start position , landmark goal , and optimal path . SimNav tests fundamental goal-reaching with a clear path. ObstNav introduces a static obstacle on the optimal route, forcing the agent to adjust the plan. SocialNav adds dynamic pedestrians whose trajectories (blue lines) intersect the agent’s path, requiring reactive collision avoidance.
Figure B4 : Visualization of a Multi-Goal navigation episode. The agent must sequentially navigate to 5 consecutive landmarks. The top row displays the ego-centric departure view for each sub-goal, and the bottom row shows the BEV map with all goal locations and the agent’s optimal trajectory across the full episode.
Method
SimNav
ObstNav
SocialNav
SR ↑
SPL ↑
CR ↓
SR ↑
SPL ↑
CR ↓
SR ↑
SPL ↑
CR ↓
w/o 3D-aware
45.0 %
0.416
53.3 %
35.8 %
0.332
61.7 %
26.5 %
0.254
69.2 %
Ours (Full)
68.3 %
0.640
13.3 %
55.8 %
0.516
30.8 %
39.2 %
0.366
48.3 %
Table C1 : Ablation study on 3D-aware action proposals. We compare our full model against a baseline without 3D-aware filtering. The 3D-aware mechanism significantly reduces collision rates across all tasks. The best results are highlighted in green .
Method
SimNav
ObstNav
SocialNav
SR ↑
SPL ↑
CR ↓
SR ↑
SPL ↑
CR ↓
SR ↑
SPL ↑
CR ↓
Continuous
37.5 %
0.345
40.8 %
26.7 %
0.238
60.1 %
17.5 %
0.153
73.3 %
5 actions
43.3 %
0.389
28.3 %
36.7 %
0.315
45.8 %
22.5 %
0.214
66.7 %
13 actions
61.7 %
0.558
22.5 %
44.1 %
0.403
40.8 %
27.4 %
0.255
64.2 %
9 actions (Ours-Final)
68.3 %
0.640
13.3 %
55.8 %
0.516
30.8 %
39.2 %
0.366
48.3 %
Table C2 : Ablation study on the action space design. We analyze the impact of action space discretization (Continuous vs. Discrete) and the density of candidate actions. Our discrete approach with 9 candidates strikes the best balance. The best results are highlighted.
Method
SimNav
ObstNav
SocialNav
SR ↑
SPL ↑
CR ↓
SR ↑
SPL ↑
CR ↓
SR ↑
SPL ↑
CR ↓
NaVILA [ 11 ]
22.5 %
0.199
70.8 %
20.8 %
0.176
75.0 %
8.3 %
0.072
84.1 %
NaVid [ 114 ]
37.4 %
0.279
19.2 %
32.5 %
0.233
23.1 %
17.5 %
0.138
51.7 %
Uni-NaVid [ 113 ]
38.8 %
0.370
12.8 %
25.3 %
0.242
29.4 %
12.5 %
0.112
66.7 %
Ours + GPT-4o [ 1 ]
43.2 %
0.397
44.1 %
33.3 %
0.289
57.5 %
23.6 %
0.223
61.8 %
Ours + Gemini-2.5-Flash [ 14 ]
60.0 %
0.556
27.5 %
44.2 %
0.393
45.8 %
34.2 %
0.319
59.2 %
Table C3 : Quantitative comparison on the SmallCity dataset with different VLM backbones. We compare our framework equipped with different VLM backbones against state-of-the-art training-based VLN methods. The top two results are highlighted by first and second .
Figure C1 : Qualitative analysis of visual prompting variations. We visualize different ablation settings compared to our proposed method (d): (a) lacks traversable mask segmentation and depth-aware filtering. Regarding action granularity: (b) shows sparse candidates which may limit control, while (c) illustrates how dense candidates complexs the reasoning process. Our approach (d) achieves an optimal balance between clarity and control.
Figure C2 : Additional qualitative Ablation on occlusion-aware semantic scene reconstruction. Without occlusion-aware masks, features from foreground occluders contaminate the target cluster, producing noisy segmentation. Adding view selection refines boundary quality by prioritizing high-visibility frames. Combining both modules yields the cleanest and most complete 3D instance segmentation. Zoom in for details.
Figure C3 : Visualizations of diverse human motion generation in various reconstructed scenes. Better zoom in for details.
Figure E1 : System Prompt Part 1: Directives and Protocols.
Figure E2 : System Prompt Part 2: JSON Specification.
Figure E3 : User Query Prompt Template.
Figure E4 : Social Navigation Prompt Extensions. For environments with dynamic agents, we augment the system prompt with strict safety protocols, introducing a stop_and_wait action and requiring the VLM to explicitly reason about collision risks presented by humans detected in the scene (marked with red bounding boxes).
Training embodied agents to understand 3D scenes as humans do requires large-scale data of people meaningfully interacting with diverse environments, yet such data is scarce. Real-world capture is costly and limited to controlled settings, while existing synthetic datasets rely on simple geometric heuristics, ignoring rich scene context. In contrast, 2D foundation models trained at internet scale have acquired commonsense knowledge of human-environment interactions. To transfer this knowledge to 3D, we introduce InHabit, an automatic and scalable data generator for populating 3D scenes with interacting humans. InHabit follows a render-generate-lift principle: given a rendered 3D scene, a vision-language model proposes contextually meaningful actions, an image-editing model inserts a human, and an optimization procedure lifts the edited result into physically plausible SMPL-X bodies aligned with the scene geometry. Applied to Habitat-Matterport3D, InHabit produces InHabitants, the first large-scale photorealistic 3D human-scene interaction dataset, with 78K samples across ∼800 building-scale scenes with complete 3D geometry, SMPL-X bodies, and images. Augmenting standard training data with InHabitants improves RGB-based 3D human-scene reconstruction and contact estimation, and in a perceptual user study our data is preferred in 78% of cases over prior art.
Nikita Kister, Pradyumna YM, István Sárándi +3
University of Tübingen, Germany · Bosch Center of Artificial Intelligence, Germany · Zalando SE, Germany +1
Visual captures provide detailed scene appearance, but converting them into interactive simulation environments requires object reconstruction, physical modeling, and correction of intermediate errors. We present SceneAgent, an agentic pipeline that coordinates these steps through simulator-based visual inspection and repair. The pipeline processes 3D Gaussian Splatting (3DGS), photogrammetry, LiDAR captures, AI-generated 3DGS scenes, and 4D videos of dynamic scenes, and combines object segmentation, physical parameter estimation, part decomposition, articulation, and deformable simulation. To support policy learning, SceneAgent generates diverse object variants ("digital sisters") and scripted demonstrations. On a cluttered bin scene from GraspClutter6D, SceneAgent raises F1 at 1 cm for every coding agent with scorable outputs both with and without its skills, and its reconstructions are better than SimFoundry: all agent configurations exceed SimFoundry's F1 score, with a best of 60.75% versus 6.9%. On three dynamic scenes from 4DCodeBench, SceneAgent raises the mean dynamic IoU of Gemini 4 Argon from 0.86% to 99.09%. We further demonstrate diverse simulated interactions and preliminary policy transfer to a real robot.
Luke Hollis, Tianxing Fan, Heng Yang
Computational Robotics Group, School of Engineering and Applied Sciences, Harvard University
Training generalizable embodied agents requires realistic, varied, and simulation-ready indoor environments. Language-guided methods support semantic planning and object-level reasoning, but precise spatial grounding remains difficult. Visual generative models offer rich priors over plausible arrangements, yet their outputs require geometric reasoning to become physically valid 3D scenes. We introduce SceneBloom, an agentic framework that combines visual layout sampling with contract-guided 3D compilation. A hierarchical sampler uses diffusion priors to generate furniture arrangements within functional zones. A compiler then uses relations extracted from each proposal to guide geometric fitting and local pose correction. Across five room types, SceneBloom attains the highest visual quality and physical stability among the compared methods and is the only method with no detected mesh collisions and out-of-bounds objects. Its layouts are also closer to the distribution of real rooms than those of all compared baselines. Object navigation policies trained in these realistic scenes achieve a higher success rate in held-out real scans than those trained on an equal number of real training scans. Code is available at https://github.com/Tinaliu0123/scenebloom.
Yuhan Liu, Lianhui Qin, Shenji Wan
New York University · University of California, San Diego