Multi-Scale Semantic Mapping in Urban Environments via Observation Calibration and Policy Dependence Regularization

Runling Long; Junhao Feng; Jia Wan

cs.CVSep 28, 2026

Multi-Scale Semantic Mapping in Urban Environments via Observation Calibration and Policy Dependence Regularization

Authors: Runling Long, Junhao Feng, Jia Wan

Organizations: Harbin Institute of Technology, Shenzhen

Abstract

Semantic mapping is fundamental to embodied navigation, yet existing methods are developed for indoor environments, where objects exhibit relatively limited scale variation and are observed from a restricted range of viewpoints. Urban environments pose substantially greater challenges: agents must map objects ranging from pedestrians to buildings while navigating large spaces with highly diverse viewing distances. These conditions introduce two key difficulties that existing datasets and methods fail to cover. First, object scale and observation distance can be severely mismatched. For example, small objects may be viewed from far away, whereas large objects may be observed at extremely close range, resulting in unreliable observation likelihoods. Second, objects with substantially different sizes and geometries require distinct mapping behaviors, which are difficult to capture with a single shared value estimator. To investigate these challenges, we introduce a large-scale urban semantic mapping dataset featuring realistic city layouts, high-fidelity rendering, and instance-level annotations spanning multiple object scales. We then propose a category-aware likelihood calibration policy that identifies and alleviates unreliable observations according to object category and viewing distance. Because the calibration and motion policies are optimized toward the same mapping objective, they may learn redundant shortcuts and become excessively coupled. We therefore introduce a mutual-information (MI) regularizer that penalizes their estimated representation dependence and encourages complementary behaviors. To better model heterogeneous mapping strategies across object scales, we further employ category-wise value estimators. We formulate their joint optimization as a Pareto optimization problem to mitigate conflicting gradients across categories.

Figures & tables

Figure 1: Existing indoor semantic mapping (a) vs. our multi-scale urban semantic mapping scenes (b). Ours explicitly considers the multi-scale objects and large navigable areas challenge, which is essential for outdoor navigation.

Dataset	Type	Platform	Scenes	Scale	Object volume ( $\text{m}^{3}$ )	Avg. objects/scene	Obj.-level ann.
OpenFly [ 19 ]	VLN	UE4	21	-	$10-100\mathrm{k}$	-	$\times$
EmbodiedCity [ 18 ]	VLN	UE5	1	City	$2-100\mathrm{k}$	-	$\times$
UrbanScene 3D [ 30 ]	Map	UE4	16	City	$10-100\mathrm{k}$	865.0	$\times$
GLEAM [ 12 ]	Map	Habitat	1152	House	$0.2-5.0$	-	$\times$
MP3D [ 5 ]	Sem-Map	Habitat	90	House	$0.2-5.0$	564.6	$\checkmark$
EmbodiedScan [ 41 ]	Sem-Map	Habitat	5185	House	$0.2-5.0$	30.9	$\checkmark$

Table 1: Dataset comparison. Ours contains objects spanning a wide scale range and explicit object-level annotations.

Figure 2: Dataset visualization. (a) High-fidelity visual conditions of the simulator. (b) Object name word cloud, showing the semantic diversity of generated urban objects. (c) Merged object category distribution.

Figure 3: (a) The agent workflow. The agent additionally predicts a likelihood calibration. (b) The agent architecture, with a CLUB module for motion-perception dependence regularization and a multi-class value head. (c) The CLUB module uses the motion latent $Z_{\alpha}$ to estimate positive and negative perception latents $Z_{\beta}^{+}$ and $Z_{\beta}^{-}$ . The resulting joint-marginal likelihood gap penalizes estimated MI during policy training.

Method	CCR (%) $\uparrow$
	$c_{\text{small}}$		$c_{\text{medium}}$		$c_{\text{large}}$
	CLIP	DINOv3	CLIP	DINOv3	CLIP	DINOv3
Uncertainty	61.7 $\pm$ 0.3	62.9 $\pm$ 1.9	74.7 $\pm$ 3.4	78.5 $\pm$ 2.1	95.5 $\pm$ 2.1	96.2 $\pm$ 1.3
Zhang et al.	52.5 $\pm$ 1.5	54.7 $\pm$ 4.8	75.3 $\pm$ 4.2	79.7 $\pm$ 3.3	96.5 $\pm$ 0.7	97.9 $\pm$ 0.7
RayFronts	28.5 $\pm$ 5.1	27.8 $\pm$ 4.4	59.8 $\pm$ 8.6	62.6 $\pm$ 6.9	91.9 $\pm$ 2.2	92.2 $\pm$ 3.5
ActiveSGM	57.6 $\pm$ 4.8	60.2 $\pm$ 6.6	93.8 $\pm$ 2.4	95.6 $\pm$ 2.2	95.7 $\pm$ 2.4	97.4 $\pm$ 1.5

Table 2: Comparison with state-of-the-art rule- and learning-based methods using two vision-language feature extractors. Ours achieves the highest OCR and consistent performance across the three scale categories.

Components				CCR (%) $\uparrow$			OCR (%) $\uparrow$	Var $\downarrow$
LC	MV	PO	MI	$c_{\text{small}}$	$c_{\text{medium}}$	$c_{\text{large}}$	OCR (%) $\uparrow$	Var $\downarrow$
–	–	–	–	82.0	97.4	99.1	92.8	59.5
$\checkmark$	–	–	–	84.3	96.8	99.2	93.4	42.9
$\checkmark$	$\checkmark$	–	–	50.2	86.5	99.4	78.7	433.9
$\checkmark$	$\checkmark$	$\checkmark$	–	92.1	97.5	99.8	96.5	10.4
$\checkmark$	–	–	$\checkmark$	88.3	96.2	99.7	94.7	22.7

Table 3: Ablation studies on the main components. LC denotes likelihood calibration, MV denotes multi-value prediction, PO denotes Pareto optimization, and MI denotes mutual-information dependence regularization. The row with all components enabled is the full model.

$k$	CCR (%) $\uparrow$			OCR (%) $\uparrow$	Var $\downarrow$
$k$	$c_{\text{small}}$	$c_{\text{medium}}$	$c_{\text{large}}$	OCR (%) $\uparrow$	Var $\downarrow$
0.01	91.0	98.5	99.8	96.4	14.9
0.05	92.7	98.5	99.8	97.0	9.6
0.10	93.5	99.6	99.7	97.6	8.4
0.15	90.4	98.5	99.8	96.2	17.3
0.20	87.8	96.1	99.9	94.6	25.4

Table 4: Sensitivity to the MI regularization weight

k

Figure 4: (a) Performance on $c_{\text{small}}$ and $c_{\text{large}}$ under different reward allocations; ours is more balanced and closer to the Pareto frontier. (b) Training-set CCR on $c_{\text{small}}$ ; MI stabilizes training and improves final performance. (c) Step-level test performance. The rows show global-map and affected-grid accuracy; ours achieves faster coverage and more accurate updates than GLEAM.

Figure 5: Qualitative semantic maps from ours and GLEAM. Colors denote correct classes; white, gray, and black denote unexplored regions, errors, and background. GLEAM misses small objects frequently, whereas ours maintains consistent multi-scale coverage.

Method	CCR (%) $\uparrow$			OCR (%) $\uparrow$	Var $\downarrow$
Method	$c_{\text{small}}$	$c_{\text{medium}}$	$c_{\text{large}}$	OCR (%) $\uparrow$	Var $\downarrow$
GLEAM-0.2	78.9	92.4	98.4	89.9	66.2
GLEAM-0.6	80.6	93.2	99.2	91.0	60.2
GLEAM-1.0	83.7	93.3	98.8	91.9	38.8
GLEAM-1.4	80.3	94.5	99.4	91.4	65.5
GLEAM-1.8	76.3	95.8	98.5	90.2	97.4

Table 5: Fixed likelihood scaling versus learned adaptive calibration trained with LC and MI. ‘GLEAM-

x

’ uses a fixed likelihood weight of

x

. MI introduces no additional inference-time modules or parameters.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

You are an intelligent agent to plan a city layput using CityEngine. You will receive a list of scene types and block types, and also a list of available object asset names. You need to design a CGA library for CityEngine. Each CGA should contain these informations: 1. Scene type. 2. Block type. 3. Available object asset names. 4. Distribution functions for each object. 5. Parameters for each distribution function. Do not use scenes, blocks and assets out of the provided lists. Think step by step of the distribution design and output your thought. Output a json file that contains all the information of the CGA library. Scene type: { scene_types } Block type: { block_types } Object assets: { object_assets }

Appendix

Table 6: Prompt for LLM asset design.

You are an intelligent agent to plan a city layput using CityEngine. You will receive a json file of current scene that contains its type with block information, and a list of CGA rules. You need to assign the CGA rules to each block, and assign the parameters of the distribution functions in the rules. Think step by step the type, consider its block type, and the surrounding blocks. Think about how real world objects ditribute and make sure that the parameters are aligned with real world. Scene info path: { scene_info_path } CGA library path: { cga_library_path }

Appendix

Table 7: Prompt for LLM scene planning.

Method	mAUC (%) $\uparrow$		mIoU (%) $\uparrow$		F-1 (%) $\uparrow$
Method	CLIP	DINOv3	CLIP	DINOv3	CLIP	DINOv3
Uncertainty	94.3 $\pm$ 1.1	96.8 $\pm$ 0.2	70.2 $\pm$ 1.8	72.6 $\pm$ 3.2	80.1 $\pm$ 4.3	82.2 $\pm$ 2.8
Zhang et al.	94.9 $\pm$ 2.0	96.4 $\pm$ 0.9	68.0 $\pm$ 1.4	70.1 $\pm$ 0.5	78.9 $\pm$ 1.8	79.8 $\pm$ 0.9
RayFronts	87.8 $\pm$ 2.4	88.9 $\pm$ 2.7	55.3 $\pm$ 3.4	59.1 $\pm$ 3.2	67.3 $\pm$ 2.8	69.0 $\pm$ 3.5
ActiveSGM	94.5 $\pm$ 0.6	96.8 $\pm$ 0.2	70.6 $\pm$ 2.0	72.2 $\pm$ 2.0	79.5 $\pm$ 3.1	81.4 $\pm$ 1.7
GLEAM	95.8 $\pm$ 1.1	98.6 $\pm$ 0.1	88.6 $\pm$ 2.1	90.9 $\pm$ 1.0	93.7 $\pm$ 0.7	95.1 $\pm$ 0.5

Appendix

Table 8: Additional metrics corresponding to Tab. 2 .

Components				mAUC (%) $\uparrow$	mIoU (%) $\uparrow$	F-1 (%) $\uparrow$
LC	MV	PO	MI	mAUC (%) $\uparrow$	mIoU (%) $\uparrow$	F-1 (%) $\uparrow$
–	–	–	–	98.5	89.8	94.4
$\checkmark$	–	–	–	99.2	92.4	95.9
$\checkmark$	$\checkmark$	–	–	98.9	77.4	92.8
$\checkmark$	$\checkmark$	$\checkmark$	–	99.4	95.1	97.5
$\checkmark$	–	–	$\checkmark$	99.3	94.3	97.0

Appendix

Table 9: Additional metrics corresponding to Tab. 3 .

$k$	mAUC (%) $\uparrow$	mIoU (%) $\uparrow$	F-1 (%) $\uparrow$
0.01	99.1	93.4	96.5
0.05	99.3	93.4	96.5
0.10	99.1	95.7	97.8
0.15	99.3	94.6	97.2
0.20	99.0	92.4	95.9

Appendix

Table 10: Additional metrics corresponding to Tab. 4 .

Method	mAUC (%) $\uparrow$	mIoU (%) $\uparrow$	F-1 (%) $\uparrow$
GLEAM-0.2	98.8	90.0	94.5
GLEAM-0.6	98.9	90.1	94.6
GLEAM-1.0	98.5	90.9	95.1
GLEAM-1.4	99.0	90.5	94.8
GLEAM-1.8	98.9	88.4	93.5
Ours	99.3	94.3	97.0

Appendix

Table 11: Additional metrics corresponding to Tab. 5 .

Method	Inference time (ms) $\downarrow$
Uncertainty	366.30
Zhang et al.	401.61
RayFronts	934.58
ActiveSGM	578.03
GLEAM	31.56
Ours	27.39

Appendix

Table 12: Per-step inference time (RTX 4090).

Method	Inference time (ms) $\downarrow$
Uncertainty	366.30
Zhang et al.	401.61
RayFronts	934.58
ActiveSGM	578.03
GLEAM	31.56
Ours	27.39

Appendix

Table 12: Per-step inference time (RTX 4090).

Method	CCR (%) $\uparrow$		OCR (%) $\uparrow$	Var $\downarrow$	mAUC (%) $\uparrow$	mIoU (%) $\uparrow$	F-1 (%) $\uparrow$
Method	car	building	OCR (%) $\uparrow$	Var $\downarrow$	mAUC (%) $\uparrow$	mIoU (%) $\uparrow$	F-1 (%) $\uparrow$
Uncertainty	55.9	59.5	57.7	3.3	72.0	37.8	52.8
Zhang et al.	50.3	79.4	64.8	212.8	71.8	34.7	49.8
RayFronts	29.5	68.5	49.0	380.6	79.6	31.3	41.2
ActiveSGM	46.1	69.5	57.8	136.7	70.6	32.3	47.4
GLEAM	66.8	87.5	77.1	107.8	87.2	44.6	52.4

Appendix

Table 13: UrbanBIS results. Metrics are in % except Var.

Configuration	CCR (%) $\uparrow$			OCR (%) $\uparrow$	Var $\downarrow$	mAUC (%) $\uparrow$	mIoU (%) $\uparrow$	F-1 (%) $\uparrow$
Configuration	$c_{\text{small}}$	$c_{\text{medium}}$	$c_{\text{large}}$	OCR (%) $\uparrow$	Var $\downarrow$	mAUC (%) $\uparrow$	mIoU (%) $\uparrow$	F-1 (%) $\uparrow$
LC/LC	84.3	96.8	99.2	93.4	42.9	99.2	92.4	95.9
LC/LC+MI	86.5	96.3	99.8	94.2	31.7	99.1	92.4	95.9
LC+MI/LC	87.3	96.5	99.6	94.5	27.6	99.0	93.3	96.4
LC+MI/LC+MI	88.3	96.2	99.7	94.7	22.7	99.3	94.3	97.0

Appendix

Table 14: Post-training module replacement without retraining; X/Y denotes motion/likelihood source checkpoints.

Metric	LC	LC+MI	$\boldsymbol{\Delta}$	Rel. $\boldsymbol{\Delta}$
Max CCA	0.9910	0.9938	$+0.0027$	$+0.28\%$
Mean CCA	0.0625	0.0451	$-0.0174$	$-27.85\%$
Top-10 CCA	0.6372	0.4922	$-0.1450$	$-22.76\%$

Appendix

Table 15: Projected-feature CCA with unrounded differences.

Figure 6: Qualitative results on the real-world UrbanBIS dataset. (a) Visualization in Isaac Sim of the 3D reconstructed mesh of the test scene provided by UrbanBIS. (b) Semantic reconstruction maps produced by different methods.

Explore similar work

Sep 23, 2026cs.RO

NaviScale: Generating Large-Scale Semantic Map Datasets for Object Navigation

Embodied navigation requires spatial representations that generalize across unseen environments, yet collecting large amounts of annotated data from real 3D environments is difficult. We propose NaviScale for semantic-map-based object navigation (ObjectNav), whose predictor can be trained on pairs of partial and complete semantic maps without reconstructing a complete 3D environment for every training sample. The framework generates large-scale semantic map training data by composing floorplans of real homes with room-level semantic and obstacle maps extracted from MP3D and HM3DSem. NaviScale increases data diversity in two ways: inter-room scaling increases floorplan-level structural diversity, while intra-room scaling fills each fixed floorplan with different combinations of room maps matched by room category. Visibility through Ray Casting (VisRC) converts the composed maps into partial observations that account for field of view, sensing range, and occlusion. The resulting dataset contains 192,000 semantic maps generated from 24,000 floorplans associated with 12,794 properties. With 300k training iterations and the training and inference settings described in this paper, the system reaches 64.3% SR and 34.8% SPL on HM3D, together with 43.1% SR and 16.8% SPL on MP3D, without changing the prediction architecture. Additional experiments evaluate the quality of the composed maps, the effects of semantic-segmentation errors, and deployment on a physical robot.

Chuanlin Lan, Yanwei Zheng, Yuxi Jing +6

School of Computer Science and Technology, Shandong University, Qingdao, China

Object Goal Navigation Navigation Sequential Scaling

Ask PDF

Jul 28, 2026cs.RO

Leveraging Semantic Maps for City-Scale Cross-View Localization

We want robots to localize in previously untraversed environments against commonly available prior data. Rich semantic data available from OpenStreetMap can be useful in this task. However, existing methods either ignore this semantic information, directly matching panoramas and overhead imagery, or dramatically compress the semantic information, working with a small set of fixed classes. To leverage this rich semantic information, two challenges need to be overcome. First, useful semantic information needs to be extracted from the robot's egocentric observations. Second, the observed information must be quickly associated with the large prior semantic map (e.g., up to 628 km^2). We show that VLMs are effective at both extracting relevant landmarks from panoramas, and identifying feasible correspondences between these landmarks and prior overhead landmarks. However, using VLMs to propose all correspondences scales poorly as the number of mapped landmarks increases. Instead, we propose distilling a lightweight matcher from a VLM which computes correspondences for all entities in a map. We use this output to form an observation likelihood which is fused over time with a Bayes filter to create a time series of pose estimates. To support further investigation into generalizable cross-view methods that leverage semantic information, we release a dataset of extracted semantics and evaluation trajectories spanning eleven environments, including panoramas we collected in a snowstorm and at night in Boston. We demonstrate our method, trained on a single city's fair-weather data, generalizes across location, lighting, weather, and other challenges. Code and datasets are available at https://efahnestock.github.io/loci/.

Ethan Fahnestock, Erick Fuentes, Philip R Osteen +1

CSAIL, MIT · DEVCOM Army Research Laboratory (ARL)

Semantic Mapping Robotic Perception Cross-View Robot Systems

Ask PDF

Jul 14, 2026cs.RO

Instance-Enriched Semantic Maps for Visual Language Navigation

Visual Language Navigation (VLN) aims to enable an embodied agent to navigate complex environments by following natural language instructions. Recent approaches build semantic spatial maps and leverage Large Language Models (LLMs) for reasoning and decision making. Despite these advances, existing systems lack instance-level object detail and robustness to diverse user queries, limiting reliable navigation in complex indoor environments. To address these limitations, we propose Instance-Enriched Semantic Maps, a unified framework with three key contributions: (1) Instance-level two-and-a-half-dimensional (2.5D) rich information mapping that constructs maps from color and depth observations via open-vocabulary panoptic segmentation, preserving vertical distinctions and capturing small objects, while storing diverse semantic attributes and natural language captions enriched with room-level context. (2) Robust query processing via LLM-based target selection, which dynamically routes queries across type-specialized experts and integrates their outputs through score-level fusion, enabling consistent goal selection across diverse query formulations. (3) Storage-efficient semantic representation that achieves approximately 96% reduction compared to three-dimensional (3D) scene-graph approaches while preserving sufficient spatial information for navigation. The proposed 2.5D representation outperforms the 3D baseline by over 27% in prediction-normalized Area Under the Curve (AUC). In navigation experiments, our method achieves over 17% improvement in object retrieval and over 23% in navigation success compared to the baseline across diverse query types. The project page is available at https://rcilab.github.io/iesm_vln.

Jiho Hong, Eunae Kang, Sanghyun Kim +1

Department of Mechanical Engineering, Kyung Hee University, Yongin-si, 17104, Republic of Korea · Advanced Institutes of Convergence Technology (AICT), Suwon, 16229, Republic of Korea · School of Mechanical Engineering, Kyungpook National University, Daegu, 41566, Republic of Korea

Vision-Language Navigation Semantic Mapping Object Goal Navigation Navigation Panoptic Segmentation

Ask PDF