Stability-aware Residual Reinforcement Learning Framework for Robotic Manipulator Disturbance Compensation
Authors: Jihong Kim, Joonhyuk Kwon, Hwa Soo Kim, TaeWon Seo, Hyung-Tae Seo
Organizations: School of Mechanical Engineering, Hanyang University, Seoul 04763, Republic of Korea · Department of Mechanical System Design, Kyonggi University, Suwon 16227, South Korea · School of Electrical Engineering, Kookmin University, Seoul 02707, South Korea
Although conventional controllers and disturbance observers (DOBs) are the standard for precision tracking in manipulators, they suffer from parameter uncertainty, nonlinear friction, and compound disturbances. This study proposes a residual reinforcement learning DOB framework that pairs an analytical observer with an RL policy. The deterministic baseline operates within a reliable region, whereas the RL policy explicitly targets the residuals that the model cannot capture. To make this compensation disturbance-aware, an estimator network aligns the observation history with a privileged disturbance context, organizing the latent space by disturbance regime and enabling rapid adaptation across disturbance transitions. To guarantee stability, we derived and enforced a state-dependent action bound on the RL policy from an input-to-state stability (ISS) analysis such that the closed loop provably confines the tracking error to a certified envelope for arbitrary policy outputs. Experiments on a 6-DOF manipulator demonstrated consistent improvements in disturbance estimation and tracking, including a 27.8% tracking-error reduction on real hardware under zero-shot sim-to-real transfer and a 38.0% reduction under a base-vibration disturbance that was not observed during training.
Figures & tables
Fig. 1: Overview of the proposed framework. The NMPC and model-based DOB form the nominal loop, and the residual RL policy compensates for the disturbance components that cannot be captured by the DOB.
Fig. 2: System identification. (a) DOB output of the real robot compared with simulation before and after identification along the calibration trajectory. (b) Identified parameters per joint with initial values and search bounds.
Fig. 3: Training-time structure of the proposed framework. Solid blocks represent components used at deployment. Dashed blue lines (critic, target encoder, prototypes, and context head) indicate components that exist only during training.
Source
Mechanism
Range
Sinusoidal torque
Asin(2πft+ϕ) on J1–J3
A∼U(0.2,2.0) N ⋅ m f∼U(0.2,2.5) Hz (curriculum) ϕ∼U(0,2π)
Impulse torque
half-sine impact, random joint
U(2,8) N ⋅ m, ≈60 ms
Joint friction
nominal profile × scale
sμ∼N(1,0.22)
Payload
added mass at gripper
m∼U(−0.1,3.0) kg
Sensor noise
Gaussian on q,q˙
σq=0.001 rad σq˙=0.02 rad/s
TABLE I: Disturbance sources and randomization ranges used during training. The ranges of the sinusoid frequency and payload were expanded stage by stage through the curriculum.
Term
Expression
Weight
Residual matching
0.1/(∥drl,t−dres,t∥1+0.1)
3.5
Compensation smoothness
∥Δ(d^filt,t+drl,t)∥2
−0.3
EE smoothness
exp(−∥p¨ee,t∥2/0.5)
0.3
Action magnitude
∥at∥2
−0.01
Joint acceleration
∥q¨t∥2
−2.5×10−5
TABLE II: Reward function components and weights.
Channel
Meaning
Source
A1..3
sinusoid amplitudes (J1–J3)
episode parameters
f1..3
sinusoid frequencies (J1–J3)
episode parameters
m
end-effector payload offset
mass randomization
sμ
joint friction scale
friction randomization
TABLE III: Composition of the privileged disturbance context e∈R8 .
Fig. 4: Estimator network. The source encoder maps the observation history to a disturbance estimate and latent context, and is aligned with a target encoder that embeds the privileged context through prototype assignment and context regression.
Fig. 5: Simulation results. (a) Joint-wise disturbance estimation in IsaacLab. (b) and (c) Zero-shot tracking in Gazebo under compound disturbances on circular and figure-eight trajectories, respectively, with per-cycle RMSE.
Fig. 6: Stability filter and ablation. (a) State error under an adversarial policy with and without the ISS bound. (b) Admissible bound ρk under a time-varying disturbance. (c) and (d) Residual compensation error and episode reward during training, respectively; ablations are cumulative.
Fig. 7: t-SNE of the latent context z colored by privileged context components. Top: proposed method. Bottom: Ours without Lregime .
Fig. 8: Real-robot tracking under sinusoidal disturbance and a 500g countermass. Top: error versus joint speed, mean ± standard deviation of ten trials. Bottom: error time series of a representative trial.
Fig. 9: Base-shaking experiment. Top: slider-crank bench that translates the manipulator base along linear rails. (a) path-geometry and tracking Root Mean Square Error(RMSE), and (b) cumulative Integral of Absolute Error(IAE).
Redundant robotic manipulators operating in constrained and human-interactive environments require accurate task-space tracking together with rigorous safety guarantees under dynamic uncertainties. Classical operational space computed torque controller (OSCTC) relies on accurate dynamic models and degrades in the presence of disturbances. In contrast, the data-driven paradigm of residual learning approximates disturbances as functions learned from full-state measurements, which are often noisy in practice, lack rigorous theoretical guarantees, and introduce additional design complexity. This paper proposes a robust OSCTC framework that integrates an extended state observer (ESO) with conformal prediction to combine model-based robustness and data-driven adaptability. The ESO estimates lumped disturbances directly in operational space without requiring full-state measurements as in residual learning, and a robust control barrier function (CBF) is constructed to enforce safety under uncertainty. However, robust CBFs require a known disturbance-variation bound to guarantee absolute safety, which often leads to conservatism in practice. To address this limitation, we further employ a sliding-window conformal prediction mechanism to estimate the bound online in a distribution-free manner, thereby achieving practical probabilistic safety guarantees. Experiments on a 7-DoF Franka Research 3 manipulator demonstrate millimeter-level tracking accuracy and real-time safe control at 1~kHz under various disturbances.
Wenhua Liu, Fan Zhang, Qin Lin
Department of Engineering Technology and the Department of Electrical and Computer Engineering, University of Houston, Houston, TX 77004, USA.
Accurate trajectory tracking in cable-driven lower-limb rehabilitation robots is challenging because model uncertainty, external disturbances, joint constraints, and pull-only cable actuation can degrade nominal control performance. Conventional model-based controllers provide an interpretable control structure but remain sensitive to model mismatch, whereas fully learning-based control can reduce transparency and complicate constraint-aware operation. This study proposes a residual deep reinforcement learning-enhanced computed torque control framework in which computed torque control generates the nominal command and a bounded Deep Deterministic Policy Gradient policy supplies only an additional compensating torque. The approach is evaluated in simulation under nominal, uncertain, disturbed, combined, and generalization conditions, together with trajectory-tracking, joint-limit, cable-demand, workspace-feasibility, and cable-Jacobian diagnostics. Across the evaluated conditions, the residual controller improves tracking and disturbance rejection relative to computed torque control while preserving the interpretable model-based command structure and satisfying the reported feasibility checks in the representative evaluation. Broader tests indicate that tracking improvements can persist beyond the representative case while also exposing trajectory-dependent constraint limitations. These results support bounded residual learning as a practical robustness-enhancement strategy for simulation-based rehabilitation robot control and motivate further constraint-aware and experimental validation.
Mohammad-Hossein Fakouri, Ali Keymasi-Khalaji
Department of Mechanical Engineering, Faculty of Engineering, Kharazmi University, Tehran, Iran
Humanoids deployed in human-centered environments must handle force-interactive tasks, where external contacts introduce unexpected disturbances that disrupt locomotion accuracy and stability. Existing learning-based approaches rely on broad domain randomization, task-specific force objectives, or learning-based force estimators from motion history, each of which compromises accuracy, task transferability, or out-of-distribution (OOD) robustness. We present Analytical Disturbance-Aware Policy Training (ADAPT), a framework that equips humanoid policies with a physically grounded disturbance observer. The core of ADAPT is an analytical whole-body disturbance observer that estimates residual force/torque online with the accessible robot dynamics, without requiring force/torque sensors. Fed directly into the policy, the estimated disturbances give the humanoid an explicit, physics-derived sense of external force/torque that can generalize across diverse unseen scenes. Experiments on a Unitree G1 humanoid show that ADAPT achieves accurate disturbance prediction and stronger robustness than a proprioception-only baseline under torso perturbations, standing pushes, and asymmetric hand payloads, with improved velocity tracking even on OOD disturbances. Moreover, ADAPT enables penalizing inferred disturbances at lower-body joints to encourage lighter locomotion.