cs.CVApr 9, 2026
SaveVisually-grounded Humanoid Agents
Organizations: Peking University · Carnegie Mellon University · Tongji University · University of California, Los Angeles ·
Abstract
Digital human generation has been studied for decades and supports a wide range of real-world applications. However, most existing systems are passively animated, relying on privileged state or scripted control, which limits scalability to novel environments. We instead ask: how can digital humans actively behave using only visual observations and specified goals in novel scenes? Achieving this would enable populating any 3D environments with digital humans at scale that exhibit spontaneous, natural, goal-directed behaviors. To this end, we introduce Visually-grounded Humanoid Agents, a coupled two-layer (world-agent) paradigm that replicates humans at multiple levels: they look, perceive, reason, and behave like real people in real-world 3D scenes. The World Layer reconstructs semantically rich 3D Gaussian scenes from real-world videos via an occlusion-aware pipeline and accommodates animatable Gaussian-based human avatars. The Agent Layer transforms these avatars into autonomous humanoid agents, equipping them with first-person RGB-D perception and enabling them to perform accurate, embodied planning with spatial awareness and iterative reasoning, which is then executed at the low level as full-body actions to drive their behaviors in the scene. We further introduce a benchmark to evaluate humanoid-scene interaction in diverse reconstructed environments. Experiments show our agents achieve robust autonomous behavior, yielding higher task success rates and fewer collisions than ablations and state-of-the-art planning methods. This work enables active digital human population and advances human-centric embodied AI. Data, code, and models will be open-sourced.
Figures & tables
Figure 2: Framework Overview. Our framework consists of two layers. The World Layer processes real-world data (scene videos, object assets, human videos) to build large-scale, semantically detailed environments via occlusion-aware reconstruction, and populates them with GS-based animatable human avatars ( Sec. 3.1 ). Then the Agent Layer drives these avatars for human-scene interaction via a perception-action loop, where visually-grounded agents plan actions from egocentric observations ( Sec. 3.2 ).
Figure 3: Overview of the Occlusion-Aware Semantic Scene Reconstruction. We first reconstruct 3D Gaussians from scene videos utilizing CityGaussian [ 47 , 48 ] . To augment 3DGS with instance-level semantics, we extract 2D masks based on SAM [ 36 ] , lift them to 3D via contrastive learning, and then segment 3D instances using coarse-to-fine quantization. We introduce occlusion-aware masks and view selection to boost segmentation accuracy in large-scale, occluded outdoor scenes. Finally, each instance is annotated via context-aware visual prompting with a VLM [ 4 ] , yielding a semantically rich environment with spatially annotated landmarks, ready for human-scene interaction.
Figure 4: Our Visually Grounded Humanoid Agent comprises a two-level framework: (1) A context-aware action planning module (high-level planner) that predicts actions from ego-centric observations. It utilizes spatial-aware visual prompting to generate physically viable, spatially grounded proposals and apply goal highlighting for contextual cues, combined with iterative reasoning for multi-step decision making ( Sec. 3.2 ). (2) A controllable motion generation module (low-level controller) that converts the planner’s command into waypoints, which then condition a motion diffusion model to synthesize full-body motion ( Sec. 3.2 ).
| Scene | World Eval. | Agent Eval. | Universality Eval. | Recon. Quality | Inter. Range | Sem. Richness | Coll. Mesh |
| SmallCity [ 34 ] | ✓ | ✓ | ✓ | High | Massive | High | ✓ |
| XGRIDS [ 104 ] | ✓ | ✓ | Ultra | Massive | High | ✓ | |
| SAGE-3D [ 56 ] | ✓ | ✓ | High | Room-scale | Medium | ✓ | |
| ArgoVerse2 [ 101 ] , PandaSet [ 106 ] | ✓ | Medium | Large | Vehicle-only | ✓ | ||
| Mip-NeRF360 [ 5 ] , DL3DV-10K [ 44 ] | ✓ | Medium | Limited | Low/Medium | ✓ | ||
| MatrixCity [ 39 ] , Horizon-GS [ 26 ] | ✓ | Low | Massive | High | ✓ |
Table 1: Settings of the scenes used in experiments. We summarize the roles of experiment scenes in world-layer and agent-layer evaluation together with key scene properties.
| Task | SimNav | ObstNav | SocialNav | ||||||
| Method | SR | SPL | CR | SR | SPL | CR | SR | SPL | CR |
| NaVILA [ 11 ] | 22.5% | 0.199 | 70.8% | 20.8% | 0.176 | 75.0% | 8.3% | 0.072 | 84.1% |
| NaVid [ 114 ] | 37.4% | 0.279 | 19.2% | 32.5% | 0.233 | 23.1% | 17.5% | 0.138 | 51.7% |
| Uni-NaVid [ 113 ] | 38.8% | 0.370 | 12.8% | 25.3% | 0.242 | 29.4% | 12.5% | 0.112 | 66.7% |
| Ours | 68.3% | 0.640 | 13.3% | 55.8% | 0.516 | 30.8% | 39.2% | 0.366 | 48.3% |
Table 2: Quantitative comparison on SmallCity dataset using our navigation benchmarks. Our method significantly outperforms three SOTA VLN baselines, improving task success rates by roughly across all tasks. Top two results are highlighted by first and second .
| Benchmark | XGRIDS | SAGE-3D | Multi-Goal | |||||||
| Method | SR | SPL | CR | SR | SPL | CR | SR | PR | PPL | CR |
| NaVILA [ 11 ] | 17.8% | 0.160 | 71.1% | 31.7% | 0.293 | 67.2% | 0.0% | 21.6% | 0.102 | 78.9 |
| NaVid [ 114 ] | 40.0% | 0.316 | 35.6% | 33.1% | 0.324 | 58.4% | 0.0% | 17.5% | 0.096 | 70.2 |
| Uni-NaVid [ 113 ] | 31.1% | 0.298 | 48.9% | 25.1% | 0.228 | 35.2% | 12.3 | 8.3 | 0.066 | 48.9 |
| Ours | 74.7% | 0.725 | 14.7% | 58.2% | 0.534 | 33.3% | 38.0 | 73.6 | 0.707 | 34.8 |
Table 3: Generalization on diverse environments and multi-goal evaluation. We evaluate on two outdoor scenes from XGRIDS [ 104 ] , four indoor scenes from SAGE-3D [ 56 ] , and a multi-goal benchmark with 2–5 consecutive landmarks per episode.
| Method | SimNav | ObstNav | SocialNav | ||||||||
| VP | IR | SR | SPL | CR | SR | SPL | CR | SR | SPL | CR | |
| (a) | ✗ | ✗ | 53.3 | 0.519 | 46.7 | 38.3 | 0.365 | 60.0 | 30.3 | 0.287 | 69.1 |
| (b) | ✓ | ✗ | 57.5 | 0.501 | 32.5 | 46.7 | 0.395 | 47.5 | 25.8 | 0.247 | 59.2 |
| (c) | ✗ | ✓ | 49.2 | 0.445 | 7.6 | 44.2 | 0.387 | 17.7 | 31.7 | 0.274 | 51.8 |
| (d) | ✓ | ✓ | 68.3 | 0.640 | 13.3 | 55.8 | 0.516 | 30.8 | 39.2 | 0.366 | 48.3 |
Table 4: Ablation study of the VLM-based planning paradigm. VP = Spatial-Aware Visual Prompting; IR = Iterative Reasoning. VP enhances visual cues for goal localization, while IR prevents short-sighted decision-making. The top two results are highlighted.
Figure 5: Qualitative ablation of the VLM-based planning paradigm. Without visual prompting, the agent loses track of the goal after detouring around obstacles. Without iterative reasoning, it follows myopic straight-line paths, leading to frequent collisions. Our full model combines both to produce robust, goal-directed trajectories.
| Method | SimNav | ObstNav | ||||
| SR | SPL | CR | SR | SPL | CR | |
| Uni-NaVid (Default) [ 113 ] | 38.8 | 0.370 | 12.8 | 25.3 | 0.242 | 29.4 |
| Ours (Default) | 68.3 | 0.640 | 13.3 | 55.8 | 0.516 | 30.8 |
| Ours (Run) | 56.7 | 0.518 | 35.8 | 41.6 | 0.382 | 54.2 |
| Ours (Slow) | 67.5 | 0.595 | 14.5 | 56.4 | 0.507 | 29.7 |
Table 5: Navigation performance under different locomotion styles. Evaluated on SimNav and ObstNav of the SmallCity [ 34 ] scene with varying motion types.
Figure 6: Qualitative results across diverse environments.
| Metric | Baselines | Ablation (Ours) | ||||
| F-3DGS | GW-3DGS | OpenGS | (a) Base | (b) +Occ | (c) +ViewSel | |
| mIoU | 0.256 | 0.025 | 0.483 | 0.518 | 0.556 | 0.601 |
| mAcc | 0.647 | 0.109 | 0.675 | 0.636 | 0.699 | 0.733 |
Table 6: Quantitative evaluation of semantic scene reconstruction. Comparison with Feature-3DGS [ 121 ] , GW-3DGS [ 30 ] , and OpenGaussian [ 103 ] , along with ablation of our proposed components in the World Layer. The best results are highlighted.
Figure 7: Qualitative Ablation on occlusion-aware semantic scene reconstruction. Our framework achieves precise 3D instance segmentation with well-defined boundaries, demonstrating robustness to severe occlusion while successfully recognizing thin or small objects in large-scale outdoor scenes. Zoom in for details.
Figure A1 : Effect of normal regularization on collision mesh extraction. With normal regularization, the surface becomes smoother, enabling more reliable collision detection.
Figure A2 : Visualizations of Context-Aware Instance Annotation. For the VLM visual prompt, the target object is highlighted by a red contour, while the background is dimmed to suppress visual distraction while retaining spatial context. The overlaid text shows the detailed descriptions generated by the annotation pipeline.
Figure A3 : Word cloud visualization of object semantic annotations across representative benchmark scenes. For SmallCity , annotations are generated by our automated open-vocabulary pipeline; for XGRIDS , semantic goals are manually annotated; for SAGE-3D , 3D bounding box annotations are provided by the original dataset. Larger words indicate more frequently occurring object categories.
Figure A4 : Visual Prompting for Social Navigation. In dynamic environments, we extend our visual prompting strategy to mark pedestrians. Humans are detected and highlighted with red bounding boxes labeled ‘HUMAN’, distinct from the green ‘GOAL’ box. These explicit visual cues allow a pre-trained VLM to reason in a zero-shot manner about the spatial relationship between the agent’s planned path and nearby moving obstacles, thereby supporting safer navigation decisions.
Figure A5 : World-frame alignment for human-scene contact. Left: raw asset with unaligned orientation. Right: aligned coordinates used for ground-height estimation and human placement.
Figure B1 : Bird’s-eye-view of outdoor benchmark scenes. Each row shows a scene with its 3DGS rendering (left) and extracted collision mesh (right). From top to bottom: SmallCity [ 34 ] , Church and Town from XGRIDS [ 104 ] .
Figure B2 : Bird’s-eye-view of indoor benchmark scenes from SAGE-3D [ 56 ] . Four room-scale indoor environments used for agent-layer generalization evaluation, covering diverse functional spaces. Left: 3DGS rendering. Right: collision mesh.
Figure B3 : Visualization of single-goal navigation tasks. Each row shows a test scenario with the initial ego-centric view (left), back view (middle), and BEV occupancy map (right) marking the agent’s start position , landmark goal , and optimal path . SimNav tests fundamental goal-reaching with a clear path. ObstNav introduces a static obstacle on the optimal route, forcing the agent to adjust the plan. SocialNav adds dynamic pedestrians whose trajectories (blue lines) intersect the agent’s path, requiring reactive collision avoidance.
Figure B4 : Visualization of a Multi-Goal navigation episode. The agent must sequentially navigate to 5 consecutive landmarks. The top row displays the ego-centric departure view for each sub-goal, and the bottom row shows the BEV map with all goal locations and the agent’s optimal trajectory across the full episode.
| Method | SimNav | ObstNav | SocialNav | ||||||
| SR | SPL | CR | SR | SPL | CR | SR | SPL | CR | |
| w/o 3D-aware | 45.0 | 0.416 | 53.3 | 35.8 | 0.332 | 61.7 | 26.5 | 0.254 | 69.2 |
| Ours (Full) | 68.3 | 0.640 | 13.3 | 55.8 | 0.516 | 30.8 | 39.2 | 0.366 | 48.3 |
Table C1 : Ablation study on 3D-aware action proposals. We compare our full model against a baseline without 3D-aware filtering. The 3D-aware mechanism significantly reduces collision rates across all tasks. The best results are highlighted in green .
| Method | SimNav | ObstNav | SocialNav | ||||||
| SR | SPL | CR | SR | SPL | CR | SR | SPL | CR | |
| Continuous | 37.5 | 0.345 | 40.8 | 26.7 | 0.238 | 60.1 | 17.5 | 0.153 | 73.3 |
| 5 actions | 43.3 | 0.389 | 28.3 | 36.7 | 0.315 | 45.8 | 22.5 | 0.214 | 66.7 |
| 13 actions | 61.7 | 0.558 | 22.5 | 44.1 | 0.403 | 40.8 | 27.4 | 0.255 | 64.2 |
| 9 actions (Ours-Final) | 68.3 | 0.640 | 13.3 | 55.8 | 0.516 | 30.8 | 39.2 | 0.366 | 48.3 |
Table C2 : Ablation study on the action space design. We analyze the impact of action space discretization (Continuous vs. Discrete) and the density of candidate actions. Our discrete approach with 9 candidates strikes the best balance. The best results are highlighted.
| Method | SimNav | ObstNav | SocialNav | ||||||
| SR | SPL | CR | SR | SPL | CR | SR | SPL | CR | |
| NaVILA [ 11 ] | 22.5 | 0.199 | 70.8 | 20.8 | 0.176 | 75.0 | 8.3 | 0.072 | 84.1 |
| NaVid [ 114 ] | 37.4 | 0.279 | 19.2 | 32.5 | 0.233 | 23.1 | 17.5 | 0.138 | 51.7 |
| Uni-NaVid [ 113 ] | 38.8 | 0.370 | 12.8 | 25.3 | 0.242 | 29.4 | 12.5 | 0.112 | 66.7 |
| Ours + GPT-4o [ 1 ] | 43.2 | 0.397 | 44.1 | 33.3 | 0.289 | 57.5 | 23.6 | 0.223 | 61.8 |
| Ours + Gemini-2.5-Flash [ 14 ] | 60.0 | 0.556 | 27.5 | 44.2 | 0.393 | 45.8 | 34.2 | 0.319 | 59.2 |
Table C3 : Quantitative comparison on the SmallCity dataset with different VLM backbones. We compare our framework equipped with different VLM backbones against state-of-the-art training-based VLN methods. The top two results are highlighted by first and second .
Figure C1 : Qualitative analysis of visual prompting variations. We visualize different ablation settings compared to our proposed method (d): (a) lacks traversable mask segmentation and depth-aware filtering. Regarding action granularity: (b) shows sparse candidates which may limit control, while (c) illustrates how dense candidates complexs the reasoning process. Our approach (d) achieves an optimal balance between clarity and control.
Figure C2 : Additional qualitative Ablation on occlusion-aware semantic scene reconstruction. Without occlusion-aware masks, features from foreground occluders contaminate the target cluster, producing noisy segmentation. Adding view selection refines boundary quality by prioritizing high-visibility frames. Combining both modules yields the cleanest and most complete 3D instance segmentation. Zoom in for details.
Figure C3 : Visualizations of diverse human motion generation in various reconstructed scenes. Better zoom in for details.
Figure E1 : System Prompt Part 1: Directives and Protocols.
Figure E2 : System Prompt Part 2: JSON Specification.
Figure E3 : User Query Prompt Template.
Figure E4 : Social Navigation Prompt Extensions. For environments with dynamic agents, we augment the system prompt with strict safety protocols, introducing a stop_and_wait action and requiring the VLM to explicitly reason about collision risks presented by humans detected in the scene (marked with red bounding boxes).
More in this topic
Embodied AI
Embodied AI studies agents that perceive, reason, and act through bodies in physical or simulated environments.
See all 255 papers