cs.AIMar 26, 2026

Modernising Reinforcement Learning-Based Navigation for Embodied Semantic Scene Graph Generation

Authors: Roman Küble, Marco Hüller, Mrunmai Phatak, Rainer Lienhart, Jörg Hähner

Organizations: Organic Computing Group · University of Augsburg, Am Technologiezentrum 8, Augsburg, Germany · Machine Learning and Computer Vision Group

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Semantic world models enable embodied agents to reason about objects, relations, and spatial context beyond purely geometric representations. In Organic Computing, such models are a key enabler for objective-driven self-adaptation under uncertainty and resource constraints. The core challenge is to acquire observations maximising model quality and downstream usefulness within a limited action budget. Semantic scene graphs (SSGs) provide a structured and compact representation for this purpose. However, constructing them within a finite action horizon requires exploration strategies that trade off information gain against navigation cost and decide when additional actions yield diminishing returns. This work presents a modular navigation component for Embodied Semantic Scene Graph Generation and modernises its decision-making by replacing the policy-optimisation method and revisiting the discrete action formulation. We study compact and finer-grained, larger discrete motion sets and compare a single-head policy over atomic actions with a factorised multi-head policy over action components. We evaluate curriculum learning and optional depth-based collision supervision, and assess SSG completeness, execution safety, and navigation behaviour. Results show that replacing the optimisation algorithm alone improves SSG completeness by 21% relative to the baseline under identical reward shaping. Depth mainly affects execution safety (collision-free motion), while completeness remains largely unchanged. Combining modern optimisation with a finer-grained, factorised action representation yields the strongest overall completeness--efficiency trade-off.

Figures & tables

Figure 1: Overview of the navigation model architecture used in our learning setup: multimodal encoders →\rightarrow LSTM policy core →\rightarrow action heads (single-head or multi-head) and optional collision auxiliary head.
Figure 2: Learning curves over training blocks. Node Recall and Episodic Return are evaluated on the held-out FloorPlans 28–30 every 50 blocks, while Move Success Rate is computed on the training scenes after each block; shaded areas denote the 95% confidence interval across seeds. SH/MH: single-head/multi-head policy; D: depth input; CL: curriculum learning; IL: imitation learning.
Figure 3: Evaluation scatter plot relating traversed path length to Node Recall and episode length; each point represents one trained run (one seed) at the final evaluation checkpoint.
Figure 4: Representative trajectories on evaluation scenes (FloorPlans 28–30). Legends report episode length and Node Recall for the shown episodes. All plotted scenarios except the Baseline use depth input.
Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Representative trajectories on held-out scenes (FloorPlans 28–30). Numbers in parentheses in the legends report (steps | score) . All plotted scenarios except the Baseline are with depth input.
ScenarioNode Recall ↑\uparrowEpisodic Return ↑\uparrowEpisode LengthMove Success Rate ↑\uparrowPath Length ↑\uparrow
Baseline (R+SH16+IL)0.48 ±\pm 0.220.6914.00.140.1
S0 (R+SH16+IL+D)0.50 ±\pm 0.240.6915.40.140.2
S1 (R+SH16+D)0.50 ±\pm 0.230.6916.20.150.2
S2 (PPO+SH16)0.58 ±\pm 0.290.8421.00.711.6
S3 (PPO+SH16+D)0.58 ±\pm 0.290.8620.80.772.2
S4 (PPO+SH504+D)0.92 ±\pm 0.140.8932.70.427.9
Appendix
Table 1: Final held-out performance on FloorPlans 28–30 using the final training checkpoint of each run (means across seeds; Node Recall reported as mean ±\pm std). Node Recall : fraction of scene graph nodes recalled at episode end. Episodic Return : cumulative reward under the composite reward function. Episode Length : number of steps taken until episode termination. Move Success Rate : fraction of executed translation actions that succeed without collision. Path Length : total path length traversed during the episode. ↑\uparrow indicates that higher values are better.
ScenarioRL-AlgorithmILAction VariantDepth w/ Coll.-AuxCL
BaselineREINFORCE✓Single-Head (16)––
S0REINFORCE✓Single-Head (16)✓–
S1REINFORCE–Single-Head (16)✓–
S2PPO–Single-Head (16)––
S3PPO–Single-Head (16)✓–
S4PPO–Single-Head (504)✓–
Appendix
Table 2: Overview of ablation scenarios. IL : imitation learning pretraining. Actions : single-head parameterisation (either compact 16 actions or enlarged 504 actions) versus factorised multi-head parameterisation (Rotation angle, Length, Stop) at 504 resolution. Depth w/ Coll.-Aux : depth input with an auxiliary head for collision prediction. CL : curriculum learning with staged action-space expansion.
ParameterValue
EMA smoothing α\alpha0.7
Embedding change threshold τdisc\tau_{\text{disc}}0.1
Scene-graph dimension dsgd_{\text{sg}}512
Divisor kk10
Stagnation embedding dimension dstagd_{\text{stag}}32
Appendix
Table 3: Hyperparameters for stagnation detection.
ParameterValue
Min. stage blocks50
Plateau evaluation window WW (blocks)50
Recent window RR (blocks)10
Plateau threshold0.02
Forced-promotion backstop (blocks)250
Entropy boost factor2.0
Appendix
Table 4: Curriculum learning hyperparameters for large action spaces.
Stage#AnglesAngle step#LengthsTotalLengths
1845∘45^{\circ}216{0.0,0.3}\{0.0,0.3\}
2845∘45^{\circ}648{0.0,0.3,0.7,1.2,1.6,2.0}\{0.0,0.3,0.7,1.2,1.6,2.0\}
3845∘45^{\circ}20160{0.0,0.1,…,2.0}∖{1.9}\{0.0,0.1,\dots,2.0\}\setminus\{1.9\}
42415∘15^{\circ}21504{0.0,0.1,…,2.0}\{0.0,0.1,\dots,2.0\}
Appendix
Table 5: Curriculum stages for large action spaces ( Stop available throughout). Lengths in metres.
ScenarioBlocks / trialWarmup (blocks)PrunerStartup trials
REINFORCE+IL400200Median2
PPO SH16400200Median2
PPO SH504800600Percentile (25%)3
PPO MH500300Median2
Appendix
Table 6: Per-trial training budget and pruning configuration used during hyperparameter optimisation. PPO entries listed as “SH16 / SH504 / MH” correspond to the compact single-head action space (16 actions), the full single-head action space (504 actions), and the multi-head factorised action space (24 rotation angles, 21 lengths, Stop ), respectively.
ParameterSearch Space (categorical)Best
REINFORCE (all setups)
α\alpha{1e-5, 3e-5, 5e-5, 1e-4, 5e-4}\{1\text{e-}5,\,3\text{e-}5,\,5\text{e-}5,\,1\text{e-}4,\,5\text{e-}4\}5e-4
γ\gamma{0.90, 0.95, 0.97, 0.99}\{0.90,\,0.95,\,0.97,\,0.99\}0.97
cauxc_{\text{aux}} (collision){0.2, 0.3, 0.4}\{0.2,\,0.3,\,0.4\}0.3
centc_{\text{ent}} (entropy){0.01, 0.05, 0.10, 0.15}\{0.01,\,0.05,\,0.10,\,0.15\}0.05
PPO (Single-Head, 16 actions)
Appendix
Table 7: Optuna hyperparameter search spaces. All parameters were optimised over categorical candidate sets. The optimisation target is the composite objective in Eq. 16 .
ComponentParameterValue
Input dimensions
 RGB Image224 ×\times 224 ×\times 3
 Depth Map (optional)224 ×\times 224 ×\times 1
Feature Extractors
 RGB encoder (ResNet18)512-dim
 Depth CNN layers3
Appendix
Table 8: Feature dimensions and architecture specifications.
CategoryParameterValue
PPO hyperparameters (SH16 / SH504 / MH)
 Learning rate α\alpha1e-41\text{e-}4 / 1e-41\text{e-}4 / 1e-41\text{e-}4
 Discount factor γ\gamma0.99
 GAE lambda λ\lambda0.90 / 0.97 / 0.97
 Clipping parameter ϵ\epsilon0.25 / 0.15 / 0.15
 Value loss coefficient cvc_{v}0.5
Appendix
Table 9: Hyperparameters for training and reward shaping. PPO entries listed as “SH16 / SH504 / MH” correspond to the compact single-head action space (16 actions), the full single-head action space (504 actions), and the multi-head factorised action space (24 rotation angles, 21 lengths, Stop ), respectively.
More in this topic

Robotic RL

Robotic reinforcement learning uses trial-and-error learning to help robots control, navigate, and manipulate objects.

See all 299 papers

Explore similar work

CardsList
Jun 30, 2026

Hierarchical 3D Scene Graph Construction and Belief-based Planning for Semantic Navigation

Semantic navigation is a fundamental task for embodied agents operating in unseen environments, requiring both semantic understanding and long-term decision-making. Recent foundation models have empowered agents with rich semantic priors for this task. However, without structured global representations, decision-making often falls back on local observations and greedy strategies, resulting in inefficient exploration and myopic behaviors, especially in long-distance navigation. To address these challenges, we propose a zero-shot semantic navigation framework. Our method incrementally maintains an online Hierarchical 3D Scene Graph (HSG) to form a multi-granular semantic topology over objects, zones, and regions, serving as a compact state abstraction for global planning. Building on this memory, we introduce a hierarchical belief-based planning framework that fuses semantic priors with exploration evidence on the HSG, and performs finite-horizon rollouts on an HSG-based simulator to explicitly estimate the long-term expected returns of candidate macro-actions. This enables globally consistent decisions and reduces redundant backtracking. Extensive experiments in high-fidelity simulation environments across multiple tasks and datasets demonstrate that our method outperforms existing state-of-the-art methods, particularly in long-distance scenarios, where our approach improves SR and SPL by an average of 9.4% and 5.0%, respectively.

Bing Wu, Zuyao Chen, Changwen Chen

The Hong Kong Polytechnic University

Belief-Space PlanningEmbodied Navigation3D SGGLong-Horizon Agent Tasks
AskPDF
Jun 24, 2026

SAGE-Nav: Leveraging LLM Planning and Alignment Fusion for Hierarchical Scene Graph-Guided Navigation

Object-Goal Navigation (ObjNav) requires embodied agents to autonomously locate specified targets using only egocentric visual observations. Existing monolithic methods struggle with long-horizon reasoning and generalize poorly to novel environments. To address these limitations, we propose SAGE-Nav, a novel hierarchical framework that integrates the reasoning capabilities of Large Language Models (LLMs) with dynamic scene graphs. Crucially, it decouples asynchronous global semantic planning from the high-frequency reactive control loop. The LLM serves as a global planner, decomposing abstract instructions into a sequence of semantically grounded waypoints. To translate these plans into dense multi-modal guidance, we design a Hierarchical Scene Graph Encoder (HSGE) that leverages relational graph convolutions to produce structure-aware embeddings preserving both semantic and spatial topology. Furthermore, we develop the Goal-aware Alignment-Fusion Network (GAFN) to dynamically fuse real-time perception with these structural priors. Using an adaptive gating mechanism with an explicit inductive bias, GAFN ensures robust visual-topological alignment for the low-level policy. Extensive evaluations in the i-THOR and RoboTHOR environments demonstrate that SAGE-Nav achieves state-of-the-art performance, delivering substantial gains in navigation efficiency and zero-shot generalization while maintaining the low control latency required for physical robotic deployment.

Hao Su, Yuehao Huang, Yukai Ma +2

Zhejiang University, Hangzhou, China

Robot Policy LearningGraph Neural NetworksLarge Language Model-Based Robot PlanningRobot NavigationObject-Goal Navigation
AskPDF
Aug 9, 2025

Imaginative World Modeling with Scene Graphs for Embodied Agent Navigation

Semantic navigation requires an agent to navigate toward a specified target in an unseen environment. Employing an imaginative navigation strategy that predicts future scenes before taking action, can empower the agent to find target faster. Inspired by this idea, we propose SGImagineNav, a novel imaginative navigation framework that leverages symbolic world modeling to proactively build a global environmental representation. SGImagineNav maintains an evolving hierarchical scene graph and uses large vision language models to predict and explore unseen parts of the environment. While existing methods solely relying on past observations, this imaginative scene graph provides richer semantic context, enabling the agent to proactively estimate target locations. Building upon this, SGImagineNav adopts an adaptive navigation strategy that exploits semantic shortcuts when promising and explores unknown areas otherwise to gather additional context. This strategy continuously expands the known environment and accumulates valuable semantic contexts, ultimately guiding the agent toward the target. SGImagineNav is evaluated in both real-world scenarios and simulation benchmarks. SGImagineNav consistently outperforms previous methods, improving the success rate to 65.4% and 66.8% on HM3D and HSSD, and demonstrating cross-floor and cross-room navigation in real-world environments. All source code is open-sourced at https://github.com/UMich-CURLY/SGImagineNav.

Yue Hu, Junzhe Wu, Ruihan Xu +5

University of Michigan, Ann Arbor, US

World Model LearningEmbodied NavigationEmbodied World Models
AskPDF