Organizations: National School of Elite Engineers, Northeastern University, Shenyang, Liaoning, China. · School of Mechanical Engineering and Automation, Northeastern University, Shenyang, Liaoning, China. · State Key Laboratory of Intelligent Deep Metal Mining and Equipment, Shenyang 110819, China. · School of Civil Engineering, Tsinghua University, Beijing, China. · College of Information, Liaoning University, Shenyang, Liaoning, China. · School of Vehicle and Mobility, Tsinghua University, Beijing, China.
Reliable state feedback of excavator front-end mechanisms is essential for autonomous excavation, yet retrofitting existing machines with dedicated joint sensors, external cameras, or visual markers can increase deployment and maintenance burdens. This paper proposes a closed-loop autonomous excavation framework for continuous inference of excavator joint states using onboard visual observations and control-related temporal information. The framework encodes structural cues from the current cab-view image, models short-term motion trends from historical joystick commands, and introduces a rolling state buffer for the bucket joint to preserve temporal continuity under severe occlusion. Mechanical joint-range constraints are incorporated into the learning objective to suppress infeasible estimates. The current cab-view image and inferred states are supplied to an Action Chunking with Transformers (ACT)-based control policy for online action generation. Experiments in an AGX Dynamics for Unity simulation environment under diverse operating conditions evaluate joint-state inference and end-to-end task execution. Across 20 autonomous excavation trials, the framework achieved a 95% task-completion rate, with an average joint-state-inference-and-control latency of 10.11 ms per step. The bucket-joint inference achieved a mean absolute error (MAE) of 0.965 degrees. These results demonstrate a sensor-light and computationally efficient joint-state-inference approach that can support closed-loop autonomous excavation with reduced dependence on additional sensing infrastructure.
Figures & tables
Figure 1: Representative cab-view scenes in real and simulated excavation settings. (a)–(b) are collected from the publicly available Fuxi excavator-motion dataset ( Fuxi Robot 2025 ) , which provides synchronized RGB observations and joint-state measurements, whereas (c)–(d) are generated from the in-house AGX Dynamics for Unity simulation dataset. The bucket and distal working-mechanism structures can be partially or fully obscured during digging and loading, reducing the reliability of single-frame joint-state inference.
Figure 2: Overview of the proposed closed-loop autonomous excavation framework. Cab-view images, joystick-input history, and a rolling state buffer support joint-state inference; the image, inferred states, and task context are provided to ACT for action generation.
Process
Online operation
Observation
Ot
←
{Itc,Ut−1}
State inference
q^t
←
fθ(Itc,Ut−1,Q^t−1)
Action generation
At
←
πACT(Itc,q^t,ct)
Temporal ensembling
a~t
←
TE(At−H+1,…,At)
Excavator execution
qt+1
←
E(a~t)
State feedback
Q^t
←
Append(Q^t−1,q^t)
Table 1: Online closed-loop execution procedure.
Figure 3: Visual-only joint-state inference errors on the public Fuxi dataset. (a) Joint-wise MAE and RMSE. (b) Absolute error profiles ordered by display position, with the mostly visible and occlusion-dominant ranges indicated; the horizontal axis denotes ranked display position rather than time.
Joint
MAE ( ∘ )
RMSE ( ∘ )
Dmax ( ∘ )
Swing
0.694
0.868
2.787
Boom
0.018
0.048
0.793
Stick
0.128
0.213
2.382
Bucket
0.965
1.550
7.083
Table 2: Joint-level prediction metrics on the simulation dataset
Method
MAE
Bucket Dmax
Swing Dmax
Boom Dmax
Stick Dmax
ResNet
4.374
23.424
7.193
4.042
8.817
ResNet + historical states
31.632
35.483
20.530
26.294
30.983
ResNet + control inputs
9.470
21.560
2.787
0.793
2.382
ResNet + states + controls
0.451
7.083
-
-
-
Table 3: Comparison of input configurations (all values in degrees)
Figure 4: Target and predicted trajectories for the four joints over a representative 8,000-step segment of the simulation test data. The upper panel shows swing and bucket, whereas the lower panel shows boom and stick. Colored curves denote predictions; dark-gray curves with open-circle markers denote target angles.
Figure 5: Appearance perturbation settings: initial condition, two lighting changes, and a representative color-change condition.
Condition
MAE ( ∘ )
RMSE ( ∘ )
Relative MAE Change (%)
Initial condition
0.965
1.550
Baseline
Lighting change 1
1.072
1.704
+11.09%
Lighting change 2
1.013
1.683
+4.97%
Component color 1
0.946
1.519
-1.97%
Component color 2
1.055
1.691
+9.33%
Table 4: Bucket prediction error under appearance perturbations
State input to ACT
Successful trials
Completion rate (%)
Demonstration-normalized excavation yield (%)
Latency (ms)
Ground-truth joint states
20/20
100
98
8.24
Inferred joint states
19/20
95
93
10.11
Table 5: Closed-loop task performance using ground-truth and inferred joint-state inputs.
Faculty of Engineering and Natural Sciences, Tampere University, Tampere, Finland · Department of Defense Convergence Technology, Korea University, Seoul, Republic of Korea · Department of Electrical and Computer Engineering, Korea University, Seoul, Republic of Korea +2