ExcavaTwin: Training-Free Geometry-Guided Semantic Elevation Mapping for Autonomous Excavation

Yu Deng; Lingshan Zeng; Tong Hu; Rushi Dai

cs.ROSep 28, 2026

ExcavaTwin: Training-Free Geometry-Guided Semantic Elevation Mapping for Autonomous Excavation

Authors: Yu Deng, Lingshan Zeng, Tong Hu, Rushi Dai

Organizations: The Smart Manufacturing Thrust, The Hong Kong University of Science and Technology (Guangzhou) · Department of Electronic and Electrical Engineering, Southern University of Science and Technology. · Capstone Technology Co., Ltd., Shenzhen, China.

Abstract

Autonomous excavation requires a spatial representation that jointly captures terrain geometry and task-relevant semantics. Existing excavation mapping is largely elevation-centric, while generic semantic models remain unstable in unstructured outdoor scenes. We present ExcavaTwin, a pure-vision geometry-guided semantic elevation mapping framework without excavation-specific training. Given multi-view RGB images, the framework: 1) reconstructs scene geometry and semantic observations using frozen vision models; 2) derives terrain and non-terrain geometric support; 3) performs geometry-constrained multi-view semantic fusion to suppress implausible predictions and recover incomplete observations; and 4) projects the fused state into a task-oriented semantic elevation map. Experiments on public datasets and real excavation scenes demonstrate reliable geometric and semantic perception. In real excavation, the system achieved an average update interval of approximately 1.4 s and a mean elevation error of 12.74cm in dynamically modified regions. Larger errors mainly occur during rapid terrain changes and transient visual disturbances caused by machine motion.

Figures & tables

Fig. 2: Main pipeline of ExcavaTwin. The pipeline consists of two stages: vision-based twinning and geometry–semantic coupling. First, frozen visual reconstruction and segmentation models, such as VGGT and YOLOE, recover the scene structure and semantics. Then, geometric features are fused with semantic information to produce an excavation-oriented spatial map with semantic and elevation representations.

Dataset	Source	Key scale	Visual input	Geometric Reference	Evaluation role
RELLIS-3D [ 8 ]	Public	13,556 LiDAR scans; 6,235 RGB images; 20 semantic classes	RGB video	LiDAR-based terrain reference	Terrain elevation
Objectron	Public	14,819 annotated video sequences; 9 categories; 180 videos sampled	RGB video	Object-level metric reference	Object dimension recovery
GOOSE-Ex [ 19 ]	Public	Selected outdoor sequences with semantic annotations	RGB video	–	Semantic mapping
OCES	Self-collected	25 video sequences; 5 object categories; $\sim$ 15k RGB frames; 3,500 LiDAR scans	RGB video	LiDAR-based terrain reference + object dimensions	Object dimension recovery; terrain elevation; semantic mapping

TABLE I: Comparison of the four evaluation datasets and their roles in this study.

Fig. 3: Outdoor Construction Excavation Site (OCES) Dataset

Task	Dataset	MAE (cm) $\downarrow$	RMSE (cm) $\downarrow$	Rel. Error (%) $\downarrow$
Object Dimension	Objectron	2.63	4.55	9.01
Object Dimension	OCES	6.23	8.94	10.43
Terrain Elevation	RELLIS-3D	28.30	32.14	–
Terrain Elevation	OCES	12.61	15.32	–

TABLE II: Geometric accuracy of the proposed scene representation.

		GOOSE-Ex		OCES
Backbone	Method	mIoU (%) $\uparrow$	Coverage (%) $\uparrow$	mIoU (%) $\uparrow$	Coverage (%) $\uparrow$
YOLOE	Backbone	3.34	2.57	6.53	2.70
YOLOE	+ Ours	20.77	90.52	34.57	80.40
Mask2Former	Backbone	21.40	94.26	34.71	91.85
Mask2Former	+ Ours	38.03	96.15	53.19	99.28

TABLE III: Semantic accuracy and spatial coverage.

Fig. 4: The qualitative comparison of GT, YOLOE, Mask2Former and our method.

Fig. 5: Evolution of the proposed spatial representation during real-world excavation

	GOOSE-Ex		OCES
Configuration	mIoU	Cov.	mIoU	Cov.
YOLOE backbone	3.34	2.57	6.53	2.70
w/o Local geometry	9.12	70.46	21.82	64.13
w/o Terrain continuity	17.84	81.28	29.63	79.72
w/o Object support	18.40	89.07	30.28	81.94
w/o Spatial consistency	10.36	63.18	17.92	50.47

TABLE IV: Ablation study.

Explore similar work

Semantic-Aware Guided Drone Exploration for Language-Conditioned 3D Indoor Mapping
May 22, 2026Nitin Vegesna, Avideh ZakhorAerial Robotics UAV Navigation
Department of Electrical Engineering and Computer Sciences University of California, Berkeley Berkeley, CA, USA
RoboAtlas: Contextual Active SLAM
Jun 24, 2026Alexander Schperberg, Shivam K. Panda, Abraham P. Vinod +2Simultaneous Localization and Mapping Robot Navigation
Mitsubishi Electric Research Laboratories, Cambridge, MA 02139, USA · Department of Mechanical and Aerospace Engineering, University of California, Los Angeles, Los Angeles, CA 90095, USA
ActiveLang: Active Open-Vocabulary 3D Mapping with Semantic-Uncertainty-Guided Exploration
Oct 7, 2026Liyan Chen, Hairong Yin, Huangying Zhan +33D Scene Representation Open-Vocabulary 3D Segmentation
Stevens Institute of Technology · Purdue University · Goertek Alpha Labs

May 22, 2026, cs.ROJ/K move · Enter open · S save

Semantic-Aware Guided Drone Exploration for Language-Conditioned 3D Indoor Mapping

Nitin Vegesna, Avideh Zakhor

Department of Electrical Engineering and Computer Sciences University of California, Berkeley Berkeley, CA, USA

Aerial Robotics UAV Navigation Frontier-Based Exploration

Open paper AskPDF arXiv

We present Semantic-Aware Guided Exploration, SAGE, a system for open-vocabulary exploration in unknown 3D indoor environments that preserves coverage-oriented behavior while allowing semantic cues to reprioritize frontier selection. Building on the FALCON volumetric explorer, SAGE integrates Contrastive Language-Image Pre-training (CLIP) via four key components: object-centric embedding storage, a temporal cache that projects recent observations onto the free-unknown boundary, object frontiers for high-similarity detections, and a unified semantic-geometric planning cost. This cost function bounds semantic reweighting influence, ensuring frontiers are prioritized without sacrificing total coverage. In Matterport3D-based simulations, SAGE outperforms FALCON and a semantic-only ablation in object discovery across map-query pairs. Compared to Finding Things in the Unknown (FTU), SAGE completes exploration 9.0 to 25.9 times faster across the nine shared map-query pairs, achieving a mean speedup of 13.7. Furthermore, SAGE achieves substantially higher volumetric throughput than FTU. Finally, we deploy SAGE in five real-world flights in two environments on a Modal AI Starling 2 quadrotor with onboard sensing and planning, and offboard CLIP inference. Comparing SAGE and FALCON, we find that while FALCON results in faster exploration and shorter mapping trajectories, SAGE outperforms FALCON in terms of object discovery.

Abstract

Explore similar work

Semantic-Aware Guided Drone Exploration for Language-Conditioned 3D Indoor Mapping

RoboAtlas: Contextual Active SLAM

ActiveLang: Active Open-Vocabulary 3D Mapping with Semantic-Uncertainty-Guided Exploration