Vision Guided Target Conditioned Control for Autonomous Excavation

Shuai Zhao; Ji-An Pan; Junwei Li; Xun Tang; Fansen Xi; Qing Xu; Keqiang Li; Jianqiang Wang

cs.ROAug 22, 2026

Vision Guided Target Conditioned Control for Autonomous Excavation

Authors: Shuai Zhao, Ji-An Pan, Junwei Li, Xun Tang, Fansen Xi, Qing Xu, Keqiang Li, Jianqiang Wang

Organizations: Liaoning University, Shenyang, China · Northeastern University, Shenyang, China · Tsinghua University, Beijing, China

Abstract

Autonomous excavation requires an intelligent control system that can convert spatial work intent into coordinated bucket motion under contact-rich soil interaction. This paper presents a target-conditioned intelligent control framework for autonomous excavation in a physics-based deformable-soil simulation workflow. An image-aligned target mask serves as a visual spatial command for the desired digging region, while a mask-conditioned Action Chunking Transformer maps multi-view RGB observations, proprioception, and the target mask to temporally extended joystick commands. To reduce target-ignoring behavior, demonstrations are organized with paired-condition supervision, where the same or closely matched scene is demonstrated with different target masks and corresponding action chunks. The framework is evaluated through both a diagnostic manipulation task and an excavation simulation benchmark with single-scoop and sequential pile-clearing protocols. In manipulation, target success is 4% for no-condition ACT, 63% for non-paired mask-conditioned ACT, and 96% for paired-condition mask-conditioned ACT. In sequential pile clearing, paired-condition mask-conditioned ACT removes 76.8% of the pile versus 27.4% and 15.7% for the two baselines, with 91.0% human-normalized efficiency. The results show that visual target conditioning, paired demonstration structure, and action-chunk control form a practical cyber-physical simulation pipeline for excavator automation.

Figures & tables

Fig. 1: Closed-loop target-conditioned intelligent control framework for autonomous excavation. (a) A visual target mask specifies the commanded dig region, and the mask-conditioned ACT controller maps multi-view observations and proprioception to action-chunk work-control commands in a physics-based excavation simulator. (b) Paired-condition supervision creates target-action contrasts by pairing the same or closely matched excavation scene with different target masks and corresponding demonstrated action chunks.

Metric or variable	Role in evaluation
Target success	Loading begins in the commanded dig region
Retained mass	Deposited mass retained after a scoop cycle
Pile removed	Fraction of initial pile removed after matched cycles
Human-normalized efficiency	Performance relative to joystick demonstrations
Residual bucket mass	Checks whether dumping is complete
Hard target collision count	Strict safety and failure indicator

TABLE I: Engineering metrics for target-conditioned excavation evaluation.

Fig. 2: Target-conditioned excavation evaluation. The physics-based simulation provides controlled deformable-soil evaluation with commanded dig regions and material-removal metrics. The scaled physical testbed is included only as an interface-level feasibility check; all quantitative excavation comparisons are reported in simulation.

Setting	Test mask	Demos	Success	Avg return
No-condition ACT	None	200	4%	32.06
Mask-conditioned ACT, non-paired	Correct	200	63%	448
Mask-conditioned ACT, paired	Correct	200	96%	658
Mask-conditioned ACT, paired	Zero	200	2%	0
Mask-conditioned ACT, paired	Random	200	1%	12.5
Mask-conditioned ACT, paired	Mismatched	200	0%	0

TABLE II: Target-conditioned manipulation results. Correct-mask rows compare no-condition, non-paired mask-conditioned, and paired-condition mask-conditioned policies. Mask-corruption rows evaluate the paired model and report original-target success.

Fig. 3: Target-conditioned manipulation evaluation. With correct masks, paired-condition mask-conditioned ACT reaches 96% target success. Zero, random, and mismatched masks reduce original-target success to 0–2%, indicating that the learned behavior depends on the provided target condition.

Protocol	Method	Trials	Target Succ.	Retained Mass	Pile Removed	Human-Norm. Eff.
Single scoop	No-condition ACT	50	n/a	$418\pm 82$ kg	n/a	$86.4\pm 9.1\%$
Single scoop	Mask-conditioned ACT, paired	50	46/50 (92.0%)	$405\pm 88$ kg	n/a	$83.5\pm 8.7\%$
Single scoop	Joystick demonstrations	50	–	$485\pm 65$ kg	n/a	$100\%$
Pile clearing	No-condition ACT	10	n/a	–	$15.7\pm 4.3\%$	$18.6\pm 5.1\%$
Pile clearing	Mask-conditioned ACT, non-paired	10	–	–	$27.4\pm 6.2\%$	$32.5\pm 7.4\%$
Pile clearing	Mask-conditioned ACT, paired	10	–	–	$76.8\pm 7.1\%$	$91.0\pm 8.4\%$

TABLE III: Excavation simulation results. Single-scoop trials evaluate one dig-load-dump primitive. Sequential pile-clearing trials evaluate target-conditioned allocation of repeated scoops over a pile. Human-normalized efficiency is computed relative to joystick demonstrations under the matched protocol.

Robotic Excavation

See all 12 papers

From Target Selection to Digging: A Learning-Based Framework for Continuous Autonomous Excavation
Sep 24, 2026Shuai Zhao, Ji-an Pan, Quantao Yang +4RL for Robotics Robotic Excavation
Liaoning University. · Northeastern University. · KTH Royal Institute of Technology, Stockholm, Sweden. +1
IL-ACT: Imitation Learning with Adaptive Cartesian Tracking Control for a 30-ton Excavator
Sep 15, 2026Mehdi Heydari Shahna, Seihun Kim, Soyi Jung +3Robotic Control Robot Trajectory Tracking
Faculty of Engineering and Natural Sciences, Tampere University, Tampere, Finland · Department of Defense Convergence Technology, Korea University, Seoul, Republic of Korea · Department of Electrical and Computer Engineering, Korea University, Seoul, Republic of Korea +2
Size Doesn't Matter: Material-State Reinforcement Learning for Excavator Transferable Soil Manipulation
Sep 11, 2026Lennart Werner, Pol Eyschen, Sean Costello +3RL for Robotics Deformable Object Manipulation
Robotic Systems Lab, ETH Zürich, Leonhardstrasse 21, 8092 Zürich, Switzerland · Hexagon Innovation Hub GmbH, Heinrich-Wild-Strasse 201, 9435 Heerbrugg, Switzerland

Sep 24, 2026J/K move · Enter open · S save

From Target Selection to Digging: A Learning-Based Framework for Continuous Autonomous Excavation

Shuai Zhao, Ji-an Pan, Quantao Yang, Zheng Wang+3

Liaoning University. · Northeastern University. · KTH Royal Institute of Technology, Stockholm, Sweden.+1

RL for Robotics Robotic Excavation Vision-Based Robot Control

Open paper AskPDF arXiv

Repeated excavation continuously reshapes pile geometry, requiring an autonomous excavator to adapt its digging targets and coordinate motion across successive excavation cycles. We present a learning-based framework for continuous autonomous excavation that integrates terrain-aware target selection with reinforcement- and imitation-learning controllers. The framework separates target-conditioned motion from local digging: a shared task-conditioned RL policy controls waypoint-guided approach and loaded transport, while an IL policy learns vision-based digging and lifting from expert demonstrations. Digging targets are selected from LiDAR elevation maps and converted into bucket-tip waypoints for motion control. The control architecture coordinates the learned policies and deterministic unloading through a shared motion interface. The complete system is deployed on a scaled hydraulic excavator with multimodal sensing and closed-loop actuator control. Offline replay and physical experiments demonstrate more consistent target selection, shorter local motion time, and increased payload compared with the respective baselines. The learned digging policy achieves a mean payload of 6.52 kg per completed cycle, compared with 2.68 kg for Fixed Dig. Three five-scoop runs further demonstrate consecutive autonomous excavation under continuously changing pile geometry.

Abstract

Robotic Excavation

Explore similar work

From Target Selection to Digging: A Learning-Based Framework for Continuous Autonomous Excavation

IL-ACT: Imitation Learning with Adaptive Cartesian Tracking Control for a 30-ton Excavator

Size Doesn't Matter: Material-State Reinforcement Learning for Excavator Transferable Soil Manipulation