Organizations: National School of Elite Engineers, Northeastern University, Shenyang, Liaoning, China. · School of Mechanical Engineering and Automation, Northeastern University, Shenyang, Liaoning, China. · State Key Laboratory of Intelligent Deep Metal Mining and Equipment, Shenyang 110819, China. · School of Civil Engineering, Tsinghua University, Beijing, China. · College of Information, Liaoning University, Shenyang, Liaoning, China. · School of Vehicle and Mobility, Tsinghua University, Beijing, China.
Reliable state feedback of excavator front-end mechanisms is essential for autonomous excavation, yet retrofitting existing machines with dedicated joint sensors, external cameras, or visual markers can increase deployment and maintenance burdens. This paper proposes a closed-loop autonomous excavation framework for continuous inference of excavator joint states using onboard visual observations and control-related temporal information. The framework encodes structural cues from the current cab-view image, models short-term motion trends from historical joystick commands, and introduces a rolling state buffer for the bucket joint to preserve temporal continuity under severe occlusion. Mechanical joint-range constraints are incorporated into the learning objective to suppress infeasible estimates. The current cab-view image and inferred states are supplied to an Action Chunking with Transformers (ACT)-based control policy for online action generation. Experiments in an AGX Dynamics for Unity simulation environment under diverse operating conditions evaluate joint-state inference and end-to-end task execution. Across 20 autonomous excavation trials, the framework achieved a 95% task-completion rate, with an average joint-state-inference-and-control latency of 10.11 ms per step. The bucket-joint inference achieved a mean absolute error (MAE) of 0.965 degrees. These results demonstrate a sensor-light and computationally efficient joint-state-inference approach that can support closed-loop autonomous excavation with reduced dependence on additional sensing infrastructure.
Figures & tables
Figure 1: Representative cab-view scenes in real and simulated excavation settings. (a)–(b) are collected from the publicly available Fuxi excavator-motion dataset ( Fuxi Robot 2025 ) , which provides synchronized RGB observations and joint-state measurements, whereas (c)–(d) are generated from the in-house AGX Dynamics for Unity simulation dataset. The bucket and distal working-mechanism structures can be partially or fully obscured during digging and loading, reducing the reliability of single-frame joint-state inference.
Figure 2: Overview of the proposed closed-loop autonomous excavation framework. Cab-view images, joystick-input history, and a rolling state buffer support joint-state inference; the image, inferred states, and task context are provided to ACT for action generation.
Process
Online operation
Observation
Ot
←
{Itc,Ut−1}
State inference
q^t
←
fθ(Itc,Ut−1,Q^t−1)
Action generation
At
←
πACT(Itc,q^t,ct)
Temporal ensembling
a~t
←
TE(At−H+1,…,At)
Excavator execution
qt+1
←
E(a~t)
State feedback
Q^t
←
Append(Q^t−1,q^t)
Table 1: Online closed-loop execution procedure.
Figure 3: Visual-only joint-state inference errors on the public Fuxi dataset. (a) Joint-wise MAE and RMSE. (b) Absolute error profiles ordered by display position, with the mostly visible and occlusion-dominant ranges indicated; the horizontal axis denotes ranked display position rather than time.
Joint
MAE ( ∘ )
RMSE ( ∘ )
Dmax ( ∘ )
Swing
0.694
0.868
2.787
Boom
0.018
0.048
0.793
Stick
0.128
0.213
2.382
Bucket
0.965
1.550
7.083
Table 2: Joint-level prediction metrics on the simulation dataset
Method
MAE
Bucket Dmax
Swing Dmax
Boom Dmax
Stick Dmax
ResNet
4.374
23.424
7.193
4.042
8.817
ResNet + historical states
31.632
35.483
20.530
26.294
30.983
ResNet + control inputs
9.470
21.560
2.787
0.793
2.382
ResNet + states + controls
0.451
7.083
-
-
-
Table 3: Comparison of input configurations (all values in degrees)
Figure 4: Target and predicted trajectories for the four joints over a representative 8,000-step segment of the simulation test data. The upper panel shows swing and bucket, whereas the lower panel shows boom and stick. Colored curves denote predictions; dark-gray curves with open-circle markers denote target angles.
Figure 5: Appearance perturbation settings: initial condition, two lighting changes, and a representative color-change condition.
Condition
MAE ( ∘ )
RMSE ( ∘ )
Relative MAE Change (%)
Initial condition
0.965
1.550
Baseline
Lighting change 1
1.072
1.704
+11.09%
Lighting change 2
1.013
1.683
+4.97%
Component color 1
0.946
1.519
-1.97%
Component color 2
1.055
1.691
+9.33%
Table 4: Bucket prediction error under appearance perturbations
State input to ACT
Successful trials
Completion rate (%)
Demonstration-normalized excavation yield (%)
Latency (ms)
Ground-truth joint states
20/20
100
98
8.24
Inferred joint states
19/20
95
93
10.11
Table 5: Closed-loop task performance using ground-truth and inferred joint-state inputs.
Autonomous excavation requires an intelligent control system that can convert spatial work intent into coordinated bucket motion under contact-rich soil interaction. This paper presents a target-conditioned intelligent control framework for autonomous excavation in a physics-based deformable-soil simulation workflow. An image-aligned target mask serves as a visual spatial command for the desired digging region, while a mask-conditioned Action Chunking Transformer maps multi-view RGB observations, proprioception, and the target mask to temporally extended joystick commands. To reduce target-ignoring behavior, demonstrations are organized with paired-condition supervision, where the same or closely matched scene is demonstrated with different target masks and corresponding action chunks. The framework is evaluated through both a diagnostic manipulation task and an excavation simulation benchmark with single-scoop and sequential pile-clearing protocols. In manipulation, target success is 4% for no-condition ACT, 63% for non-paired mask-conditioned ACT, and 96% for paired-condition mask-conditioned ACT. In sequential pile clearing, paired-condition mask-conditioned ACT removes 76.8% of the pile versus 27.4% and 15.7% for the two baselines, with 91.0% human-normalized efficiency. The results show that visual target conditioning, paired demonstration structure, and action-chunk control form a practical cyber-physical simulation pipeline for excavator automation.
Shuai Zhao, Ji-An Pan, Junwei Li +5
Liaoning University, Shenyang, China · Northeastern University, Shenyang, China · Tsinghua University, Beijing, China
Autonomous excavator control is challenged by coupled kinematics, actuation lag, and uncertainty. We propose imitation learning and adaptive Cartesian tracking (IL-ACT), a novel motion control framework for a 30-ton-class excavator. An anchored, 14-input imitation policy pretrained on operator demonstrations generates nominal joint rates; adaptive Cartesian feedback and gated gain/bias estimation correct these commands before a stopping-distance governor constrains joint-reference generation. Simscape evaluation covers 100 sequential goals and spiral, figure-eight, and rounded-raster tracking, including 88 additional runs across three training seeds, two initializations, and speeds, under hydraulic response and sensing conditions. Compared with Teacher+ACT, IL-ACT completes all goals with shorter duration and lower terminal errors under both response conditions. Telemetry-initialized IL-ACT lowers RMSE in all 24 figure-eight and rounded-raster seed comparisons and lowers additional-load spiral mean RMSE by approximately 29%. Original spiral RMSE also improves over IL-only and PID. Under a shared sensor-noise realization, telemetry-initialized IL-ACT achieves 27.67% lower mean RMSE than Teacher+ACT; enabling estimation reduces mean RMSE by 22.44% relative to the frozen estimator. Pretrained-weight effects remain mixed, and the original teacher comparison exhibits a spiral RMSE--maximum-error tradeoff. Analysis establishes bounded adaptive states and Cartesian feedback, with reference admissibility conditional on governor feasibility.
Mehdi Heydari Shahna, Seihun Kim, Soyi Jung +3
Faculty of Engineering and Natural Sciences, Tampere University, Tampere, Finland · Department of Defense Convergence Technology, Korea University, Seoul, Republic of Korea · Department of Electrical and Computer Engineering, Korea University, Seoul, Republic of Korea +2
Repeated excavation continuously reshapes pile geometry, requiring an autonomous excavator to adapt its digging targets and coordinate motion across successive excavation cycles. We present a learning-based framework for continuous autonomous excavation that integrates terrain-aware target selection with reinforcement- and imitation-learning controllers. The framework separates target-conditioned motion from local digging: a shared task-conditioned RL policy controls waypoint-guided approach and loaded transport, while an IL policy learns vision-based digging and lifting from expert demonstrations. Digging targets are selected from LiDAR elevation maps and converted into bucket-tip waypoints for motion control. The control architecture coordinates the learned policies and deterministic unloading through a shared motion interface. The complete system is deployed on a scaled hydraulic excavator with multimodal sensing and closed-loop actuator control. Offline replay and physical experiments demonstrate more consistent target selection, shorter local motion time, and increased payload compared with the respective baselines. The learned digging policy achieves a mean payload of 6.52 kg per completed cycle, compared with 2.68 kg for Fixed Dig. Three five-scoop runs further demonstrate consecutive autonomous excavation under continuously changing pile geometry.
Shuai Zhao, Ji-an Pan, Quantao Yang +4
Liaoning University. · Northeastern University. · KTH Royal Institute of Technology, Stockholm, Sweden. +1