Modernising Reinforcement Learning-Based Navigation for Embodied Semantic Scene Graph Generation

Roman Küble; Marco Hüller; Mrunmai Phatak; Rainer Lienhart; Jörg Hähner

cs.AIMar 26, 2026

Modernising Reinforcement Learning-Based Navigation for Embodied Semantic Scene Graph Generation

Authors: Roman Küble, Marco Hüller, Mrunmai Phatak, Rainer Lienhart, Jörg Hähner

Organizations: Organic Computing Group · University of Augsburg, Am Technologiezentrum 8, Augsburg, Germany · Machine Learning and Computer Vision Group

Abstract

Semantic world models enable embodied agents to reason about objects, relations, and spatial context beyond purely geometric representations. In Organic Computing, such models are a key enabler for objective-driven self-adaptation under uncertainty and resource constraints. The core challenge is to acquire observations maximising model quality and downstream usefulness within a limited action budget. Semantic scene graphs (SSGs) provide a structured and compact representation for this purpose. However, constructing them within a finite action horizon requires exploration strategies that trade off information gain against navigation cost and decide when additional actions yield diminishing returns. This work presents a modular navigation component for Embodied Semantic Scene Graph Generation and modernises its decision-making by replacing the policy-optimisation method and revisiting the discrete action formulation. We study compact and finer-grained, larger discrete motion sets and compare a single-head policy over atomic actions with a factorised multi-head policy over action components. We evaluate curriculum learning and optional depth-based collision supervision, and assess SSG completeness, execution safety, and navigation behaviour. Results show that replacing the optimisation algorithm alone improves SSG completeness by 21% relative to the baseline under identical reward shaping. Depth mainly affects execution safety (collision-free motion), while completeness remains largely unchanged. Combining modern optimisation with a finer-grained, factorised action representation yields the strongest overall completeness--efficiency trade-off.

Figures & tables

Figure 1: Overview of the navigation model architecture used in our learning setup: multimodal encoders

\rightarrow

LSTM policy core

\rightarrow

action heads (single-head or multi-head) and optional collision auxiliary head.

Figure 2: Learning curves over training blocks. Node Recall and Episodic Return are evaluated on the held-out FloorPlans 28–30 every 50 blocks, while Move Success Rate is computed on the training scenes after each block; shaded areas denote the 95% confidence interval across seeds. SH/MH: single-head/multi-head policy; D: depth input; CL: curriculum learning; IL: imitation learning.

Figure 3: Evaluation scatter plot relating traversed path length to Node Recall and episode length; each point represents one trained run (one seed) at the final evaluation checkpoint.

Panel of Figure 4 — Figure 4: Representative trajectories on evaluation scenes (FloorPlans 28–30). Legends report episode length and Node Recall for the shown episodes. All plotted scenarios except the Baseline use depth input.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Representative trajectories on held-out scenes (FloorPlans 28–30). Numbers in parentheses in the legends report (steps | score) . All plotted scenarios except the Baseline are with depth input.

Scenario	Node Recall $\uparrow$	Episodic Return $\uparrow$	Episode Length	Move Success Rate $\uparrow$	Path Length $\uparrow$
Baseline (R+SH16+IL)	0.48 $\pm$ 0.22	0.69	14.0	0.14	0.1
S0 (R+SH16+IL+D)	0.50 $\pm$ 0.24	0.69	15.4	0.14	0.2
S1 (R+SH16+D)	0.50 $\pm$ 0.23	0.69	16.2	0.15	0.2
S2 (PPO+SH16)	0.58 $\pm$ 0.29	0.84	21.0	0.71	1.6
S3 (PPO+SH16+D)	0.58 $\pm$ 0.29	0.86	20.8	0.77	2.2
S4 (PPO+SH504+D)	0.92 $\pm$ 0.14	0.89	32.7	0.42	7.9

Appendix

Table 1: Final held-out performance on FloorPlans 28–30 using the final training checkpoint of each run (means across seeds; Node Recall reported as mean

\pm

std). Node Recall : fraction of scene graph nodes recalled at episode end. Episodic Return : cumulative reward under the composite reward function. Episode Length : number of steps taken until episode termination. Move Success Rate : fraction of executed translation actions that succeed without collision. Path Length : total path length traversed during the episode.

\uparrow

indicates that higher values are better.

Scenario	RL-Algorithm	IL	Action Variant	Depth w/ Coll.-Aux	CL
Baseline	REINFORCE	✓	Single-Head (16)	–	–
S0	REINFORCE	✓	Single-Head (16)	✓	–
S1	REINFORCE	–	Single-Head (16)	✓	–
S2	PPO	–	Single-Head (16)	–	–
S3	PPO	–	Single-Head (16)	✓	–
S4	PPO	–	Single-Head (504)	✓	–

Appendix

Table 2: Overview of ablation scenarios. IL : imitation learning pretraining. Actions : single-head parameterisation (either compact 16 actions or enlarged 504 actions) versus factorised multi-head parameterisation (Rotation angle, Length, Stop) at 504 resolution. Depth w/ Coll.-Aux : depth input with an auxiliary head for collision prediction. CL : curriculum learning with staged action-space expansion.

Parameter	Value
EMA smoothing $\alpha$	0.7
Embedding change threshold $\tau_{\text{disc}}$	0.1
Scene-graph dimension $d_{\text{sg}}$	512
Divisor $k$	10
Stagnation embedding dimension $d_{\text{stag}}$	32

Appendix

Table 3: Hyperparameters for stagnation detection.

Parameter	Value
Min. stage blocks	50
Plateau evaluation window $W$ (blocks)	50
Recent window $R$ (blocks)	10
Plateau threshold	0.02
Forced-promotion backstop (blocks)	250
Entropy boost factor	2.0

Appendix

Table 4: Curriculum learning hyperparameters for large action spaces.

Stage	#Angles	Angle step	#Lengths	Total	Lengths
1	8	$45^{\circ}$	2	16	$\{0.0,0.3\}$
2	8	$45^{\circ}$	6	48	$\{0.0,0.3,0.7,1.2,1.6,2.0\}$
3	8	$45^{\circ}$	20	160	$\{0.0,0.1,\dots,2.0\}\setminus\{1.9\}$
4	24	$15^{\circ}$	21	504	$\{0.0,0.1,\dots,2.0\}$

Appendix

Table 5: Curriculum stages for large action spaces ( Stop available throughout). Lengths in metres.

Scenario	Blocks / trial	Warmup (blocks)	Pruner	Startup trials
REINFORCE+IL	400	200	Median	2
PPO SH16	400	200	Median	2
PPO SH504	800	600	Percentile (25%)	3
PPO MH	500	300	Median	2

Appendix

Table 6: Per-trial training budget and pruning configuration used during hyperparameter optimisation. PPO entries listed as “SH16 / SH504 / MH” correspond to the compact single-head action space (16 actions), the full single-head action space (504 actions), and the multi-head factorised action space (24 rotation angles, 21 lengths, Stop ), respectively.

Parameter	Search Space (categorical)	Best
REINFORCE (all setups)
$\alpha$	$\{1\text{e-}5,\,3\text{e-}5,\,5\text{e-}5,\,1\text{e-}4,\,5\text{e-}4\}$	5e-4
$\gamma$	$\{0.90,\,0.95,\,0.97,\,0.99\}$	0.97
$c_{\text{aux}}$ (collision)	$\{0.2,\,0.3,\,0.4\}$	0.3
$c_{\text{ent}}$ (entropy)	$\{0.01,\,0.05,\,0.10,\,0.15\}$	0.05
PPO (Single-Head, 16 actions)

Appendix

Table 7: Optuna hyperparameter search spaces. All parameters were optimised over categorical candidate sets. The optimisation target is the composite objective in Eq. 16 .

Component	Parameter	Value
Input dimensions
	RGB Image	224 $\times$ 224 $\times$ 3
	Depth Map (optional)	224 $\times$ 224 $\times$ 1
Feature Extractors
	RGB encoder (ResNet18)	512-dim
	Depth CNN layers	3

Appendix

Table 8: Feature dimensions and architecture specifications.

Category	Parameter	Value
PPO hyperparameters (SH16 / SH504 / MH)
	Learning rate $\alpha$	$1\text{e-}4$ / $1\text{e-}4$ / $1\text{e-}4$
	Discount factor $\gamma$	0.99
	GAE lambda $\lambda$	0.90 / 0.97 / 0.97
	Clipping parameter $\epsilon$	0.25 / 0.15 / 0.15
	Value loss coefficient $c_{v}$	0.5

Appendix

Table 9: Hyperparameters for training and reward shaping. PPO entries listed as “SH16 / SH504 / MH” correspond to the compact single-head action space (16 actions), the full single-head action space (504 actions), and the multi-head factorised action space (24 rotation angles, 21 lengths, Stop ), respectively.

Robotic RL

Robotic reinforcement learning uses trial-and-error learning to help robots control, navigate, and manipulate objects.

See all 299 papers

Hierarchical 3D Scene Graph Construction and Belief-based Planning for Semantic Navigation
Jun 30, 2026Bing Wu, Zuyao Chen, Changwen ChenBelief-Space Planning Embodied Navigation
The Hong Kong Polytechnic University
SAGE-Nav: Leveraging LLM Planning and Alignment Fusion for Hierarchical Scene Graph-Guided Navigation
Jun 24, 2026Hao Su, Yuehao Huang, Yukai Ma +2Robot Policy Learning Graph Neural Networks
Zhejiang University, Hangzhou, China
Imaginative World Modeling with Scene Graphs for Embodied Agent Navigation
Aug 9, 2025Yue Hu, Junzhe Wu, Ruihan Xu +5World Model Learning Embodied Navigation
University of Michigan, Ann Arbor, US

Jun 30, 2026J/K move · Enter open · S save

Hierarchical 3D Scene Graph Construction and Belief-based Planning for Semantic Navigation

Bing Wu, Zuyao Chen, Changwen Chen

The Hong Kong Polytechnic University

Belief-Space Planning Embodied Navigation 3D SGG Long-Horizon Agent Tasks

Open paper AskPDF arXiv

Semantic navigation is a fundamental task for embodied agents operating in unseen environments, requiring both semantic understanding and long-term decision-making. Recent foundation models have empowered agents with rich semantic priors for this task. However, without structured global representations, decision-making often falls back on local observations and greedy strategies, resulting in inefficient exploration and myopic behaviors, especially in long-distance navigation. To address these challenges, we propose a zero-shot semantic navigation framework. Our method incrementally maintains an online Hierarchical 3D Scene Graph (HSG) to form a multi-granular semantic topology over objects, zones, and regions, serving as a compact state abstraction for global planning. Building on this memory, we introduce a hierarchical belief-based planning framework that fuses semantic priors with exploration evidence on the HSG, and performs finite-horizon rollouts on an HSG-based simulator to explicitly estimate the long-term expected returns of candidate macro-actions. This enables globally consistent decisions and reduces redundant backtracking. Extensive experiments in high-fidelity simulation environments across multiple tasks and datasets demonstrate that our method outperforms existing state-of-the-art methods, particularly in long-distance scenarios, where our approach improves SR and SPL by an average of 9.4% and 5.0%, respectively.

Abstract

Robotic RL

Explore similar work

Hierarchical 3D Scene Graph Construction and Belief-based Planning for Semantic Navigation

SAGE-Nav: Leveraging LLM Planning and Alignment Fusion for Hierarchical Scene Graph-Guided Navigation

Imaginative World Modeling with Scene Graphs for Embodied Agent Navigation