Stability-aware Residual Reinforcement Learning Framework for Robotic Manipulator Disturbance Compensation

Jihong Kim; Joonhyuk Kwon; Hwa Soo Kim; TaeWon Seo; Hyung-Tae Seo

cs.ROSep 18, 2026

Stability-aware Residual Reinforcement Learning Framework for Robotic Manipulator Disturbance Compensation

Authors: Jihong Kim, Joonhyuk Kwon, Hwa Soo Kim, TaeWon Seo, Hyung-Tae Seo

Organizations: School of Mechanical Engineering, Hanyang University, Seoul 04763, Republic of Korea · Department of Mechanical System Design, Kyonggi University, Suwon 16227, South Korea · School of Electrical Engineering, Kookmin University, Seoul 02707, South Korea

Abstract

Although conventional controllers and disturbance observers (DOBs) are the standard for precision tracking in manipulators, they suffer from parameter uncertainty, nonlinear friction, and compound disturbances. This study proposes a residual reinforcement learning DOB framework that pairs an analytical observer with an RL policy. The deterministic baseline operates within a reliable region, whereas the RL policy explicitly targets the residuals that the model cannot capture. To make this compensation disturbance-aware, an estimator network aligns the observation history with a privileged disturbance context, organizing the latent space by disturbance regime and enabling rapid adaptation across disturbance transitions. To guarantee stability, we derived and enforced a state-dependent action bound on the RL policy from an input-to-state stability (ISS) analysis such that the closed loop provably confines the tracking error to a certified envelope for arbitrary policy outputs. Experiments on a 6-DOF manipulator demonstrated consistent improvements in disturbance estimation and tracking, including a 27.8% tracking-error reduction on real hardware under zero-shot sim-to-real transfer and a 38.0% reduction under a base-vibration disturbance that was not observed during training.

Figures & tables

Fig. 1: Overview of the proposed framework. The NMPC and model-based DOB form the nominal loop, and the residual RL policy compensates for the disturbance components that cannot be captured by the DOB.

Fig. 2: System identification. (a) DOB output of the real robot compared with simulation before and after identification along the calibration trajectory. (b) Identified parameters per joint with initial values and search bounds.

Fig. 3: Training-time structure of the proposed framework. Solid blocks represent components used at deployment. Dashed blue lines (critic, target encoder, prototypes, and context head) indicate components that exist only during training.

Source	Mechanism	Range
Sinusoidal torque	$A\sin(2\pi ft+\phi)$ on J1–J3	$A\!\sim\!\mathcal{U}(0.2,\,2.0)$ N $\cdot$ m $f\!\sim\!\mathcal{U}(0.2,\,2.5)$ Hz (curriculum) $\phi\!\sim\!\mathcal{U}(0,\,2\pi)$
Impulse torque	half-sine impact, random joint	$\mathcal{U}(2,\,8)$ N $\cdot$ m, $\approx\!60$ ms
Joint friction	nominal profile $\times$ scale	$s_{\mu}\sim\mathcal{N}(1,\,0.2^{2})$
Payload	added mass at gripper	$m\!\sim\!\mathcal{U}(-0.1,\,3.0)$ kg
Sensor noise	Gaussian on $q,\dot{q}$	$\sigma_{q}\!=\!0.001$ rad $\sigma_{\dot{q}}\!=\!0.02$ rad/s

TABLE I: Disturbance sources and randomization ranges used during training. The ranges of the sinusoid frequency and payload were expanded stage by stage through the curriculum.

Term	Expression	Weight
Residual matching	$0.1\,/\,\big(\lVert d_{\mathrm{rl},t}-d_{\mathrm{res},t}\rVert_{1}+0.1\big)$	$3.5$
Compensation smoothness	$\lVert\Delta(\hat{d}_{\mathrm{filt},t}+d_{\mathrm{rl},t})\rVert^{2}$	$-0.3$
EE smoothness	$\exp\!\big(-\lVert\ddot{p}_{ee,t}\rVert^{2}/0.5\big)$	$0.3$
Action magnitude	$\lVert a_{t}\rVert^{2}$	$-0.01$
Joint acceleration	$\lVert\ddot{q}_{t}\rVert^{2}$	$-2.5\!\times\!10^{-5}$

TABLE II: Reward function components and weights.

Channel	Meaning	Source
$A_{1..3}$	sinusoid amplitudes (J1–J3)	episode parameters
$f_{1..3}$	sinusoid frequencies (J1–J3)	episode parameters
$m$	end-effector payload offset	mass randomization
$s_{\mu}$	joint friction scale	friction randomization

TABLE III: Composition of the privileged disturbance context

e\in\mathbb{R}^{8}

Fig. 4: Estimator network. The source encoder maps the observation history to a disturbance estimate and latent context, and is aligned with a target encoder that embeds the privileged context through prototype assignment and context regression.

Fig. 5: Simulation results. (a) Joint-wise disturbance estimation in IsaacLab. (b) and (c) Zero-shot tracking in Gazebo under compound disturbances on circular and figure-eight trajectories, respectively, with per-cycle RMSE.

Fig. 6: Stability filter and ablation. (a) State error under an adversarial policy with and without the ISS bound. (b) Admissible bound $\rho_{k}$ under a time-varying disturbance. (c) and (d) Residual compensation error and episode reward during training, respectively; ablations are cumulative.

Fig. 7: t-SNE of the latent context $z$ colored by privileged context components. Top: proposed method. Bottom: Ours without $L_{regime}$ .

Fig. 8: Real-robot tracking under sinusoidal disturbance and a 500g countermass. Top: error versus joint speed, mean ± standard deviation of ten trials. Bottom: error time series of a representative trial.

Fig. 9: Base-shaking experiment. Top: slider-crank bench that translates the manipulator base along linear rails. (a) path-geometry and tracking Root Mean Square Error(RMSE), and (b) cumulative Integral of Absolute Error(IAE).

Explore similar work

Robust Operational Space Control with Conformal Disturbance Bounds for Safe Redundant Manipulation
Jul 1, 2026Wenhua Liu, Fan Zhang, Qin LinRobotic Control Control Barrier Functions
Department of Engineering Technology and the Department of Electrical and Computer Engineering, University of Houston, Houston, TX 77004, USA.
Residual Deep Reinforcement Learning-Based Computed Torque Control for a Cable-Driven Lower-Limb Rehabilitation Robot under Disturbances and Parametric Uncertainties
Aug 27, 2026Mohammad-Hossein Fakouri, Ali Keymasi-KhalajiRobotic Control Robot Trajectory Tracking
Department of Mechanical Engineering, Faculty of Engineering, Kharazmi University, Tehran, Iran
ADAPT: Analytical Disturbance-Aware Policy Training for Humanoid Locomotion
Jun 15, 2026Bofan Lyu, Jindou Jia, Kuangji Zuo +7Robotic Control Humanoid Robot Locomotion
1MARS Lab, Nanyang Technological University

Jul 1, 2026, cs.ROJ/K move · Enter open · S save

Robust Operational Space Control with Conformal Disturbance Bounds for Safe Redundant Manipulation

Wenhua Liu, Fan Zhang, Qin Lin

Department of Engineering Technology and the Department of Electrical and Computer Engineering, University of Houston, Houston, TX 77004, USA.

Robotic Control Control Barrier Functions Robust Control Robotic Manipulation

Open paper AskPDF arXiv

Redundant robotic manipulators operating in constrained and human-interactive environments require accurate task-space tracking together with rigorous safety guarantees under dynamic uncertainties. Classical operational space computed torque controller (OSCTC) relies on accurate dynamic models and degrades in the presence of disturbances. In contrast, the data-driven paradigm of residual learning approximates disturbances as functions learned from full-state measurements, which are often noisy in practice, lack rigorous theoretical guarantees, and introduce additional design complexity. This paper proposes a robust OSCTC framework that integrates an extended state observer (ESO) with conformal prediction to combine model-based robustness and data-driven adaptability. The ESO estimates lumped disturbances directly in operational space without requiring full-state measurements as in residual learning, and a robust control barrier function (CBF) is constructed to enforce safety under uncertainty. However, robust CBFs require a known disturbance-variation bound to guarantee absolute safety, which often leads to conservatism in practice. To address this limitation, we further employ a sliding-window conformal prediction mechanism to estimate the bound online in a distribution-free manner, thereby achieving practical probabilistic safety guarantees. Experiments on a 7-DoF Franka Research 3 manipulator demonstrate millimeter-level tracking accuracy and real-time safe control at 1~kHz under various disturbances.

Abstract

Explore similar work

Robust Operational Space Control with Conformal Disturbance Bounds for Safe Redundant Manipulation

Residual Deep Reinforcement Learning-Based Computed Torque Control for a Cable-Driven Lower-Limb Rehabilitation Robot under Disturbances and Parametric Uncertainties

ADAPT: Analytical Disturbance-Aware Policy Training for Humanoid Locomotion