Multi-Scale Semantic Mapping in Urban Environments via Observation Calibration and Policy Dependence Regularization

Runling Long; Junhao Feng; Jia Wan

cs.CVSep 28, 2026

Multi-Scale Semantic Mapping in Urban Environments via Observation Calibration and Policy Dependence Regularization

Authors: Runling Long, Junhao Feng, Jia Wan

Organizations: Harbin Institute of Technology, Shenzhen

Abstract

Semantic mapping is fundamental to embodied navigation, yet existing methods are developed for indoor environments, where objects exhibit relatively limited scale variation and are observed from a restricted range of viewpoints. Urban environments pose substantially greater challenges: agents must map objects ranging from pedestrians to buildings while navigating large spaces with highly diverse viewing distances. These conditions introduce two key difficulties that existing datasets and methods fail to cover. First, object scale and observation distance can be severely mismatched. For example, small objects may be viewed from far away, whereas large objects may be observed at extremely close range, resulting in unreliable observation likelihoods. Second, objects with substantially different sizes and geometries require distinct mapping behaviors, which are difficult to capture with a single shared value estimator. To investigate these challenges, we introduce a large-scale urban semantic mapping dataset featuring realistic city layouts, high-fidelity rendering, and instance-level annotations spanning multiple object scales. We then propose a category-aware likelihood calibration policy that identifies and alleviates unreliable observations according to object category and viewing distance. Because the calibration and motion policies are optimized toward the same mapping objective, they may learn redundant shortcuts and become excessively coupled. We therefore introduce a mutual-information (MI) regularizer that penalizes their estimated representation dependence and encourages complementary behaviors. To better model heterogeneous mapping strategies across object scales, we further employ category-wise value estimators. We formulate their joint optimization as a Pareto optimization problem to mitigate conflicting gradients across categories.

Figures & tables

Figure 1: Existing indoor semantic mapping (a) vs. our multi-scale urban semantic mapping scenes (b). Ours explicitly considers the multi-scale objects and large navigable areas challenge, which is essential for outdoor navigation.

Dataset	Type	Platform	Scenes	Scale	Object volume ( $\text{m}^{3}$ )	Avg. objects/scene	Obj.-level ann.
OpenFly [ 19 ]	VLN	UE4	21	-	$10-100\mathrm{k}$	-	$\times$
EmbodiedCity [ 18 ]	VLN	UE5	1	City	$2-100\mathrm{k}$	-	$\times$
UrbanScene 3D [ 30 ]	Map	UE4	16	City	$10-100\mathrm{k}$	865.0	$\times$
GLEAM [ 12 ]	Map	Habitat	1152	House	$0.2-5.0$	-	$\times$
MP3D [ 5 ]	Sem-Map	Habitat	90	House	$0.2-5.0$	564.6	$\checkmark$
EmbodiedScan [ 41 ]	Sem-Map	Habitat	5185	House	$0.2-5.0$	30.9	$\checkmark$

Table 1: Dataset comparison. Ours contains objects spanning a wide scale range and explicit object-level annotations.

Figure 2: Dataset visualization. (a) High-fidelity visual conditions of the simulator. (b) Object name word cloud, showing the semantic diversity of generated urban objects. (c) Merged object category distribution.

Figure 3: (a) The agent workflow. The agent additionally predicts a likelihood calibration. (b) The agent architecture, with a CLUB module for motion-perception dependence regularization and a multi-class value head. (c) The CLUB module uses the motion latent $Z_{\alpha}$ to estimate positive and negative perception latents $Z_{\beta}^{+}$ and $Z_{\beta}^{-}$ . The resulting joint-marginal likelihood gap penalizes estimated MI during policy training.

Method	CCR (%) $\uparrow$
	$c_{\text{small}}$		$c_{\text{medium}}$		$c_{\text{large}}$
	CLIP	DINOv3	CLIP	DINOv3	CLIP	DINOv3
Uncertainty	61.7 $\pm$ 0.3	62.9 $\pm$ 1.9	74.7 $\pm$ 3.4	78.5 $\pm$ 2.1	95.5 $\pm$ 2.1	96.2 $\pm$ 1.3
Zhang et al.	52.5 $\pm$ 1.5	54.7 $\pm$ 4.8	75.3 $\pm$ 4.2	79.7 $\pm$ 3.3	96.5 $\pm$ 0.7	97.9 $\pm$ 0.7
RayFronts	28.5 $\pm$ 5.1	27.8 $\pm$ 4.4	59.8 $\pm$ 8.6	62.6 $\pm$ 6.9	91.9 $\pm$ 2.2	92.2 $\pm$ 3.5
ActiveSGM	57.6 $\pm$ 4.8	60.2 $\pm$ 6.6	93.8 $\pm$ 2.4	95.6 $\pm$ 2.2	95.7 $\pm$ 2.4	97.4 $\pm$ 1.5

Table 2: Comparison with state-of-the-art rule- and learning-based methods using two vision-language feature extractors. Ours achieves the highest OCR and consistent performance across the three scale categories.

Components				CCR (%) $\uparrow$			OCR (%) $\uparrow$	Var $\downarrow$
LC	MV	PO	MI	$c_{\text{small}}$	$c_{\text{medium}}$	$c_{\text{large}}$	OCR (%) $\uparrow$	Var $\downarrow$
–	–	–	–	82.0	97.4	99.1	92.8	59.5
$\checkmark$	–	–	–	84.3	96.8	99.2	93.4	42.9
$\checkmark$	$\checkmark$	–	–	50.2	86.5	99.4	78.7	433.9
$\checkmark$	$\checkmark$	$\checkmark$	–	92.1	97.5	99.8	96.5	10.4
$\checkmark$	–	–	$\checkmark$	88.3	96.2	99.7	94.7	22.7

Table 3: Ablation studies on the main components. LC denotes likelihood calibration, MV denotes multi-value prediction, PO denotes Pareto optimization, and MI denotes mutual-information dependence regularization. The row with all components enabled is the full model.

$k$	CCR (%) $\uparrow$			OCR (%) $\uparrow$	Var $\downarrow$
$k$	$c_{\text{small}}$	$c_{\text{medium}}$	$c_{\text{large}}$	OCR (%) $\uparrow$	Var $\downarrow$
0.01	91.0	98.5	99.8	96.4	14.9
0.05	92.7	98.5	99.8	97.0	9.6
0.10	93.5	99.6	99.7	97.6	8.4
0.15	90.4	98.5	99.8	96.2	17.3
0.20	87.8	96.1	99.9	94.6	25.4

Table 4: Sensitivity to the MI regularization weight

k

Figure 4: (a) Performance on $c_{\text{small}}$ and $c_{\text{large}}$ under different reward allocations; ours is more balanced and closer to the Pareto frontier. (b) Training-set CCR on $c_{\text{small}}$ ; MI stabilizes training and improves final performance. (c) Step-level test performance. The rows show global-map and affected-grid accuracy; ours achieves faster coverage and more accurate updates than GLEAM.

Figure 5: Qualitative semantic maps from ours and GLEAM. Colors denote correct classes; white, gray, and black denote unexplored regions, errors, and background. GLEAM misses small objects frequently, whereas ours maintains consistent multi-scale coverage.

Method	CCR (%) $\uparrow$			OCR (%) $\uparrow$	Var $\downarrow$
Method	$c_{\text{small}}$	$c_{\text{medium}}$	$c_{\text{large}}$	OCR (%) $\uparrow$	Var $\downarrow$
GLEAM-0.2	78.9	92.4	98.4	89.9	66.2
GLEAM-0.6	80.6	93.2	99.2	91.0	60.2
GLEAM-1.0	83.7	93.3	98.8	91.9	38.8
GLEAM-1.4	80.3	94.5	99.4	91.4	65.5
GLEAM-1.8	76.3	95.8	98.5	90.2	97.4

Table 5: Fixed likelihood scaling versus learned adaptive calibration trained with LC and MI. ‘GLEAM-

x

’ uses a fixed likelihood weight of

x

. MI introduces no additional inference-time modules or parameters.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

You are an intelligent agent to plan a city layput using CityEngine. You will receive a list of scene types and block types, and also a list of available object asset names. You need to design a CGA library for CityEngine. Each CGA should contain these informations: 1. Scene type. 2. Block type. 3. Available object asset names. 4. Distribution functions for each object. 5. Parameters for each distribution function. Do not use scenes, blocks and assets out of the provided lists. Think step by step of the distribution design and output your thought. Output a json file that contains all the information of the CGA library. Scene type: { scene_types } Block type: { block_types } Object assets: { object_assets }

Appendix

Table 6: Prompt for LLM asset design.

You are an intelligent agent to plan a city layput using CityEngine. You will receive a json file of current scene that contains its type with block information, and a list of CGA rules. You need to assign the CGA rules to each block, and assign the parameters of the distribution functions in the rules. Think step by step the type, consider its block type, and the surrounding blocks. Think about how real world objects ditribute and make sure that the parameters are aligned with real world. Scene info path: { scene_info_path } CGA library path: { cga_library_path }

Appendix

Table 7: Prompt for LLM scene planning.

Method	mAUC (%) $\uparrow$		mIoU (%) $\uparrow$		F-1 (%) $\uparrow$
Method	CLIP	DINOv3	CLIP	DINOv3	CLIP	DINOv3
Uncertainty	94.3 $\pm$ 1.1	96.8 $\pm$ 0.2	70.2 $\pm$ 1.8	72.6 $\pm$ 3.2	80.1 $\pm$ 4.3	82.2 $\pm$ 2.8
Zhang et al.	94.9 $\pm$ 2.0	96.4 $\pm$ 0.9	68.0 $\pm$ 1.4	70.1 $\pm$ 0.5	78.9 $\pm$ 1.8	79.8 $\pm$ 0.9
RayFronts	87.8 $\pm$ 2.4	88.9 $\pm$ 2.7	55.3 $\pm$ 3.4	59.1 $\pm$ 3.2	67.3 $\pm$ 2.8	69.0 $\pm$ 3.5
ActiveSGM	94.5 $\pm$ 0.6	96.8 $\pm$ 0.2	70.6 $\pm$ 2.0	72.2 $\pm$ 2.0	79.5 $\pm$ 3.1	81.4 $\pm$ 1.7
GLEAM	95.8 $\pm$ 1.1	98.6 $\pm$ 0.1	88.6 $\pm$ 2.1	90.9 $\pm$ 1.0	93.7 $\pm$ 0.7	95.1 $\pm$ 0.5

Appendix

Table 8: Additional metrics corresponding to Tab. 2 .

Components				mAUC (%) $\uparrow$	mIoU (%) $\uparrow$	F-1 (%) $\uparrow$
LC	MV	PO	MI	mAUC (%) $\uparrow$	mIoU (%) $\uparrow$	F-1 (%) $\uparrow$
–	–	–	–	98.5	89.8	94.4
$\checkmark$	–	–	–	99.2	92.4	95.9
$\checkmark$	$\checkmark$	–	–	98.9	77.4	92.8
$\checkmark$	$\checkmark$	$\checkmark$	–	99.4	95.1	97.5
$\checkmark$	–	–	$\checkmark$	99.3	94.3	97.0

Appendix

Table 9: Additional metrics corresponding to Tab. 3 .

$k$	mAUC (%) $\uparrow$	mIoU (%) $\uparrow$	F-1 (%) $\uparrow$
0.01	99.1	93.4	96.5
0.05	99.3	93.4	96.5
0.10	99.1	95.7	97.8
0.15	99.3	94.6	97.2
0.20	99.0	92.4	95.9

Appendix

Table 10: Additional metrics corresponding to Tab. 4 .

Method	mAUC (%) $\uparrow$	mIoU (%) $\uparrow$	F-1 (%) $\uparrow$
GLEAM-0.2	98.8	90.0	94.5
GLEAM-0.6	98.9	90.1	94.6
GLEAM-1.0	98.5	90.9	95.1
GLEAM-1.4	99.0	90.5	94.8
GLEAM-1.8	98.9	88.4	93.5
Ours	99.3	94.3	97.0

Appendix

Table 11: Additional metrics corresponding to Tab. 5 .

Method	Inference time (ms) $\downarrow$
Uncertainty	366.30
Zhang et al.	401.61
RayFronts	934.58
ActiveSGM	578.03
GLEAM	31.56
Ours	27.39

Appendix

Table 12: Per-step inference time (RTX 4090).

Method	Inference time (ms) $\downarrow$
Uncertainty	366.30
Zhang et al.	401.61
RayFronts	934.58
ActiveSGM	578.03
GLEAM	31.56
Ours	27.39

Appendix

Table 12: Per-step inference time (RTX 4090).

Method	CCR (%) $\uparrow$		OCR (%) $\uparrow$	Var $\downarrow$	mAUC (%) $\uparrow$	mIoU (%) $\uparrow$	F-1 (%) $\uparrow$
Method	car	building	OCR (%) $\uparrow$	Var $\downarrow$	mAUC (%) $\uparrow$	mIoU (%) $\uparrow$	F-1 (%) $\uparrow$
Uncertainty	55.9	59.5	57.7	3.3	72.0	37.8	52.8
Zhang et al.	50.3	79.4	64.8	212.8	71.8	34.7	49.8
RayFronts	29.5	68.5	49.0	380.6	79.6	31.3	41.2
ActiveSGM	46.1	69.5	57.8	136.7	70.6	32.3	47.4
GLEAM	66.8	87.5	77.1	107.8	87.2	44.6	52.4

Appendix

Table 13: UrbanBIS results. Metrics are in % except Var.

Configuration	CCR (%) $\uparrow$			OCR (%) $\uparrow$	Var $\downarrow$	mAUC (%) $\uparrow$	mIoU (%) $\uparrow$	F-1 (%) $\uparrow$
Configuration	$c_{\text{small}}$	$c_{\text{medium}}$	$c_{\text{large}}$	OCR (%) $\uparrow$	Var $\downarrow$	mAUC (%) $\uparrow$	mIoU (%) $\uparrow$	F-1 (%) $\uparrow$
LC/LC	84.3	96.8	99.2	93.4	42.9	99.2	92.4	95.9
LC/LC+MI	86.5	96.3	99.8	94.2	31.7	99.1	92.4	95.9
LC+MI/LC	87.3	96.5	99.6	94.5	27.6	99.0	93.3	96.4
LC+MI/LC+MI	88.3	96.2	99.7	94.7	22.7	99.3	94.3	97.0

Appendix

Table 14: Post-training module replacement without retraining; X/Y denotes motion/likelihood source checkpoints.

Metric	LC	LC+MI	$\boldsymbol{\Delta}$	Rel. $\boldsymbol{\Delta}$
Max CCA	0.9910	0.9938	$+0.0027$	$+0.28\%$
Mean CCA	0.0625	0.0451	$-0.0174$	$-27.85\%$
Top-10 CCA	0.6372	0.4922	$-0.1450$	$-22.76\%$

Appendix

Table 15: Projected-feature CCA with unrounded differences.

Figure 6: Qualitative results on the real-world UrbanBIS dataset. (a) Visualization in Isaac Sim of the 3D reconstructed mesh of the test scene provided by UrbanBIS. (b) Semantic reconstruction maps produced by different methods.

Explore similar work

NaviScale: Generating Large-Scale Semantic Map Datasets for Object Navigation
Sep 23, 2026Chuanlin Lan, Yanwei Zheng, Yuxi Jing +6Object Goal Navigation Navigation
School of Computer Science and Technology, Shandong University, Qingdao, China
Leveraging Semantic Maps for City-Scale Cross-View Localization
Jul 28, 2026Ethan Fahnestock, Erick Fuentes, Philip R Osteen +1Semantic Mapping Robotic Perception
CSAIL, MIT · DEVCOM Army Research Laboratory (ARL)
Instance-Enriched Semantic Maps for Visual Language Navigation
Jul 14, 2026Jiho Hong, Eunae Kang, Sanghyun Kim +1Vision-Language Navigation Semantic Mapping
Department of Mechanical Engineering, Kyung Hee University, Yongin-si, 17104, Republic of Korea · Advanced Institutes of Convergence Technology (AICT), Suwon, 16229, Republic of Korea · School of Mechanical Engineering, Kyungpook National University, Daegu, 41566, Republic of Korea

Sep 23, 2026, cs.ROJ/K move · Enter open · S save

NaviScale: Generating Large-Scale Semantic Map Datasets for Object Navigation

Chuanlin Lan, Yanwei Zheng, Yuxi Jing, Weijian Liu+5

School of Computer Science and Technology, Shandong University, Qingdao, China

Object Goal Navigation Navigation Sequential Scaling

Open paper AskPDF arXiv

Embodied navigation requires spatial representations that generalize across unseen environments, yet collecting large amounts of annotated data from real 3D environments is difficult. We propose NaviScale for semantic-map-based object navigation (ObjectNav), whose predictor can be trained on pairs of partial and complete semantic maps without reconstructing a complete 3D environment for every training sample. The framework generates large-scale semantic map training data by composing floorplans of real homes with room-level semantic and obstacle maps extracted from MP3D and HM3DSem. NaviScale increases data diversity in two ways: inter-room scaling increases floorplan-level structural diversity, while intra-room scaling fills each fixed floorplan with different combinations of room maps matched by room category. Visibility through Ray Casting (VisRC) converts the composed maps into partial observations that account for field of view, sensing range, and occlusion. The resulting dataset contains 192,000 semantic maps generated from 24,000 floorplans associated with 12,794 properties. With 300k training iterations and the training and inference settings described in this paper, the system reaches 64.3% SR and 34.8% SPL on HM3D, together with 43.1% SR and 16.8% SPL on MP3D, without changing the prediction architecture. Additional experiments evaluate the quality of the composed maps, the effects of semantic-segmentation errors, and deployment on a physical robot.

Abstract

Explore similar work

NaviScale: Generating Large-Scale Semantic Map Datasets for Object Navigation

Leveraging Semantic Maps for City-Scale Cross-View Localization

Instance-Enriched Semantic Maps for Visual Language Navigation