Residual Deep Reinforcement Learning-Based Computed Torque Control for a Cable-Driven Lower-Limb Rehabilitation Robot under Disturbances and Parametric Uncertainties

Mohammad-Hossein Fakouri; Ali Keymasi-Khalaji

cs.ROAug 27, 2026

Residual Deep Reinforcement Learning-Based Computed Torque Control for a Cable-Driven Lower-Limb Rehabilitation Robot under Disturbances and Parametric Uncertainties

Authors: Mohammad-Hossein Fakouri, Ali Keymasi-Khalaji

Organizations: Department of Mechanical Engineering, Faculty of Engineering, Kharazmi University, Tehran, Iran

Abstract

Accurate trajectory tracking in cable-driven lower-limb rehabilitation robots is challenging because model uncertainty, external disturbances, joint constraints, and pull-only cable actuation can degrade nominal control performance. Conventional model-based controllers provide an interpretable control structure but remain sensitive to model mismatch, whereas fully learning-based control can reduce transparency and complicate constraint-aware operation. This study proposes a residual deep reinforcement learning-enhanced computed torque control framework in which computed torque control generates the nominal command and a bounded Deep Deterministic Policy Gradient policy supplies only an additional compensating torque. The approach is evaluated in simulation under nominal, uncertain, disturbed, combined, and generalization conditions, together with trajectory-tracking, joint-limit, cable-demand, workspace-feasibility, and cable-Jacobian diagnostics. Across the evaluated conditions, the residual controller improves tracking and disturbance rejection relative to computed torque control while preserving the interpretable model-based command structure and satisfying the reported feasibility checks in the representative evaluation. Broader tests indicate that tracking improvements can persist beyond the representative case while also exposing trajectory-dependent constraint limitations. These results support bounded residual learning as a practical robustness-enhancement strategy for simulation-based rehabilitation robot control and motivate further constraint-aware and experimental validation.

Figures & tables

Literature group	Main emphasis	Role in the present study
Cable-driven rehabilitation systems	Mechanism design, low moving inertia, task-specific assistance, and human-oriented control	Defines the application domain and motivates reproducible lower-limb motion generation
Cable feasibility and CDPR control	Positive-tension feasibility, wrench distribution, force allocation, robustness, and Jacobian properties	Motivates nonnegative-tension feasibility, Jacobian-rank, minimum-singular-value, and conditioning analyses
Nonlinear rehabilitation control	Computed-torque, robust/adaptive, impedance, admittance, and assist-as-needed control	Establishes the interpretable model-based control layer and comparative baseline
DDPG and residual RL	Continuous-action learning and learned correction around an existing controller	Provides the bounded compensation mechanism used here
Present study	Computed-torque control with bounded residual DDPG for a planar 3-DOF, three-cable lower-limb model	Integrates trajectory-tracking evaluation with joint-limit, cable-tension, workspace-feasibility, and Jacobian-conditioning analyses

Table 1: Technical positioning of the present study relative to the principal literature groups.

Symbol	Meaning
$\bm{q},\dot{\bm{q}},\ddot{\bm{q}}$	Joint position, velocity, and acceleration vectors
$\bm{x},\bm{x}_{d}$	Actual and desired Cartesian end-effector positions
$\bm{p}_{i},\bm{p}_{ci}$	Joint and link-center-of-mass positions
$\bm{a}_{i},\bm{r}_{i}$	Fixed anchor and moving attachment of cable $i$
$l_{i},\bm{u}_{i}$	Cable length and anchor-to-attachment unit vector
$\bm{J}_{c}$	Implemented cable actuation Jacobian, $-\partial\bm{l}/\partial\bm{q}$

Table 2: Main symbols in the kinematic, dynamic, and control formulation.

Panel of Figure 1 — Figure 1: Planar lower-limb CDRR geometry used in the simulation model. Panel (a) defines the three-link serial mechanism and joint coordinates; panel (b) shows the fixed cable anchors, link-side attachment points, and pull directions that determine the three cable lengths.

Quantity	Value	Description
$b_{1},b_{2},b_{3}$	$0.45,0.35,0.21$ m	Calibrated nominal link lengths
$m_{1},m_{2},m_{3}$	$11.125,5.05,1.38$ kg	Link masses
$I_{cg,1},I_{cg,2},I_{cg,3}$	$0.149187,0.0522254,0.004784$ kg m 2	COM inertias
$g$	$9.81$ m s -2	Gravitational acceleration
$\bm{q}_{\min}$	$[80,2,250]^{T}$ deg	Lower joint limits
$\bm{q}_{\max}$	$[250,160,330]^{T}$ deg	Upper joint limits

Table 3: Nominal robot parameters, cable anchors, and active joint limits.

Parameter	Value
Observation/action dimensions	12 / 3
Residual torque bounds	$\pm[5,3,2]^{T}$ N m
Actor hidden layers	256 and 256 ReLU neurons
Critic feature width	256 neurons per path/stage
Agent sample time	$T_{s}=0.01$ s
Discount factor	$\gamma=0.99$

Table 4: Archived residual DDPG implementation and training settings.

Figure 2: Hybrid CTC–residual DDPG control architecture and parallel cable-demand audit. Constrained reference generation supplies the desired joint states and derivatives; CTC produces the nominal model-based torque

\bm{\tau}_{\mathrm{ctc}}

, and the trained actor supplies the bounded correction

\bm{\tau}_{\mathrm{RL}}

. Their sum

\bm{\tau}_{\mathrm{cmd}}=\bm{\tau}_{\mathrm{ctc}}+\bm{\tau}_{\mathrm{RL}}

is applied directly to the torque-driven simulated plant. In parallel,

\bm{\tau}_{\mathrm{cmd}}

and the measured joint coordinates are supplied to an algebraic cable-demand audit, whose inferred output is logged but does not drive the plant. The external disturbance

\bm{\tau}_{d}

enters the plant separately. Plant-dynamics parameter mismatch is active from 1 to 5 s, while the calibrated kinematic geometry remains fixed; disturbance is active from 5 to 9 s.

Figure 3: Residual DDPG actor–critic structure and off-policy training. The unnormalized 12-dimensional observation is mapped by the actor to three bounded residual torques. The critic estimates

Q(\bm{o},\bm{a})

from separate observation and action feature paths. Replay-buffer sampling, gradient updates, and soft target-network updates are used during training; deterministic evaluation uses the saved actor without exploration noise or online learning.

Step	Operation
1	Read $\bm{q},\dot{\bm{q}}$ and form $\bm{e},\dot{\bm{e}}$ using the constrained reference.
2	Evaluate $\bm{D}_{n},\bm{C}_{n},\bm{G}_{n}$ and compute $\bm{\tau}_{\mathrm{ctc}}$ from Eqs. ( 39 )–( 40 ).
3	Form $\bm{o}_{t}$ and evaluate the actor; add exploration only during training.
4	Scale the actor output to the bounds in Eq. ( 44 ) and form $\bm{\tau}_{\mathrm{cmd}}=\bm{\tau}_{\mathrm{ctc}}+\bm{\tau}_{\mathrm{RL}}$ .
5	Integrate the uncertain/disturbed generalized-coordinate plant and compute the reward.
6	During training, store $(\bm{o}_{t},\bm{a}_{t},r_{t},\bm{o}_{t+1})$ , sample a minibatch, update critic and actor, and softly update target networks.

Table 5: Controller execution and DDPG training/evaluation sequence.

Figure 4: Cartesian model-consistency comparison for the analytical equation-of-motion, Simscape Multibody, and MSC ADAMS implementations under the same nominal reference. Panels (a) and (b) compare the end-effector

x

- and

y

-coordinate histories, respectively.

Figure 5: Joint-angle model-consistency comparison among the analytical equation-of-motion, Simscape Multibody, and MSC ADAMS implementations. Panels (a)–(c) report $q_{1}$ , $q_{2}$ , and $q_{3}$ , respectively, under the same nominal reference.

Figure 6: Simscape-minus-equation-of-motion joint-angle differences for

q_{1}

q_{2}

, and

q_{3}

in panels (a)–(c), respectively. The difference histories localize the remaining implementation discrepancy, which is largest in the third joint channel.

Signal	Unit	Simscape–EOM	ADAMS–EOM
$q_{1}$	deg	$0.102\,/\,0.233$	$0.091\,/\,0.224$
$q_{2}$	deg	$0.409\,/\,0.984$	$0.362\,/\,0.832$
$q_{3}$	deg	$1.715\,/\,3.537$	$1.684\,/\,3.023$
$x$	mm	$3.050\,/\,7.877$	$2.959\,/\,7.812$
$y$	mm	$3.742\,/\,8.628$	$3.692\,/\,8.609$

Table 6: Archived model-consistency discrepancies relative to the analytical equation-of-motion implementation. Each entry reports RMS

/

maximum absolute discrepancy.

Check	Purpose	Interpretation
Forward/inverse kinematics	Verify Cartesian-reference and joint-coordinate consistency	Kinematic consistency of the implemented reference-generation chain
Simscape Multibody	Compare the analytical dynamics with an independent multibody formulation	Nominal simulation-to-simulation consistency under matched parameters
MSC ADAMS	Provide an additional independent multibody implementation check	Cross-platform nominal dynamic consistency
Tension post-processing	Audit the pull-only cable-demand mapping	Generalized-torque demand mapped to inferred cable tension
Workspace and $\bm{J}_{c}$ analysis	Evaluate sampled reachability, static nonnegative-tension feasibility, rank, and conditioning	Sampled, trajectory-specific cable-feasibility evidence

Table 7: Purpose and interpretive scope of the simulation checks.

Setting	Value
Simulation duration / fixed step	10 s / 0.01 s
Archived solver setting	Fixed-step ode3 under FixedStepAuto
Nominal Cartesian initial state	$[-0.90,-0.05]^{T}$ m
Primary trajectory	Eq. ( 14 )
CTC gains	$K_{p}=60I_{3}$ , $K_{d}=20I_{3}$
Uncertainty	Simultaneous $+10\%$ perturbation of masses, COM inertias, and length-dependent plant-dynamics coefficients; calibrated kinematic geometry remains fixed

Table 8: Simulation and comparison settings used for the reported final results.

Case	Parametric uncertainty	External disturbance
C1: nominal	inactive	inactive
C2: uncertainty only	10%, 1–5 s	inactive
C3: disturbance only	inactive	5–9 s
C4: combined	10%, 1–5 s	5–9 s

Table 9: Four primary evaluation scenarios.

Figure 7: Sampled joint-limit workspace and desired Cartesian reference path for the planar 3-DOF mechanism. The light background denotes the ideal geometric reach bound, the sampled points denote configurations satisfying the active joint limits, and the high-contrast curve denotes the desired path; the path is closed for display by repeating its first plotted Cartesian point without modifying the evaluated trajectory data. The plotting orientation is horizontal

Y

and vertical

X

, with 45 samples per joint.

Figure 8: Sampled cable-feasibility map for the planar 3-DOF, three-cable mechanism. Light points denote joint-limited workspace samples, the emphasized subset satisfies the nominal static gravity-torque mapping with full-rank $\bm{J}_{c}$ and nonnegative inferred cable tension, and the high-contrast curve denotes the desired Cartesian path. The path is closed for display by repeating its first plotted point without modifying saved trajectory data or performance calculations; axes are horizontal $Y$ and vertical $X$ .

Figure 9: Cable-actuation Jacobian diagnostics along the desired trajectory: minimum singular value

\sigma_{\min}(\bm{J}_{c})

, two-norm condition number

\kappa_{2}(\bm{J}_{c})

, and matrix rank. Rank remains three over the logged path, while

\sigma_{\min}

and

\kappa_{2}

identify the least favorably conditioned intervals. Light-blue and light-peach shading denote the 1–5 s uncertainty and 5–9 s disturbance intervals, respectively.

Quantity	Value
Samples per joint / total samples	45 / 91,125
Statically nonnegative-tension samples	33,921
Nonnegative-tension sample ratio	37.22%
Desired path inside geometric sample	100%
Desired path inside tested feasible sample	100%
Minimum $\sigma_{\min}(\bm{J}_{c})$ along path	0.01481

Table 10: Sampled workspace and cable-Jacobian summary for the desired trajectory.

Panel of Figure 10 — Figure 10: Cartesian tracking under combined parametric uncertainty and external disturbance. Panel (a) compares the desired path with the realized CTC and CTC + bounded residual DDPG trajectories; panel (b) reports the Euclidean Cartesian tracking-error norm. Light-blue shading marks the 1–5 s uncertainty interval and light-peach shading marks the 5–9 s disturbance interval.

Metric	CTC	CTC + residual DDPG	Change
RMS Cartesian error (m)	0.0221419	0.0128462	42.0% lower
Peak Cartesian error (m)	0.0970994	0.0782787	19.4% lower
IAE (m s)	0.156619	0.0572133	63.5% lower
ISE (m 2 s)	0.00487701	0.00162135	66.8% lower
Recovery time after disturbance onset (s)	0.08	0	–
Minimum logged cable-demand output (N)	0	3.41445	–

Table 11: Representative combined uncertainty/disturbance results from the detailed post-processing dataset.

Figure 11: Algebraically inferred cable-tension demands

F_{1}

F_{2}

, and

F_{3}

in the representative combined case. Dashed curves correspond to CTC and solid curves to CTC + bounded residual DDPG. Light-blue and light-peach shading identify the uncertainty and disturbance intervals, respectively.

Figure 12: Minimum logged algebraic cable-demand audit output,

\min_{i}F_{i}(t)

, in the representative combined case. The audit logic returns zero when its nonnegative-tension test rejects a solution, so the logged signal is nonnegative by construction. CTC reaches this zero fallback boundary, whereas the CTC + bounded residual DDPG branch retains a minimum of 3.41445 N. This diagnostic output is not applied to the simulated plant. Light-blue and light-peach shading denote the uncertainty and disturbance intervals.

Figure 13: Joint-coordinate responses

q_{1}

q_{2}

, and

q_{3}

with their active lower and upper limits in the representative combined case. The CTC trajectory contains 61 out-of-limit channel-samples, whereas the CTC + bounded residual DDPG trajectory contains none under the same sampling and tolerance convention.

Case	Controller	RMS (m)	Peak (m)	Min. tension (N)	RMS reduction
Nominal	CTC	0.016479	0.097099	0	–
Nominal	CTC + residual DDPG	0.013001	0.078279	3.7196	21.1%
Uncertainty only	CTC	0.021239	0.097099	0	–
Uncertainty only	CTC + residual DDPG	0.012875	0.078279	3.7197	39.4%
Disturbance only	CTC	0.018034	0.097099	0	–
Disturbance only	CTC + residual DDPG	0.013034	0.078279	3.9467	27.7%

Table 12: Four-case comparison from the independent case-matrix dataset.

Panel of Figure 14 — Figure 14: Cartesian tracking metrics for the nominal, uncertainty-only, disturbance-only, and combined scenarios: panel (a) reports RMS error and panel (b) reports peak error for CTC and CTC + bounded residual DDPG.

Figure 15: Integral Cartesian tracking metrics for the four scenarios. The grouped values compare IAE and ISE for CTC and CTC + bounded residual DDPG using the independently generated case-matrix dataset.

Panel of Figure 16 — Figure 16: Cartesian path and Euclidean tracking-error histories for C1 (nominal) and C2 (10% parametric uncertainty active from 1 to 5 s). Each case compares CTC with CTC + bounded residual DDPG under the same desired path and initial condition.

Panel of Figure 17 — Figure 17: Cartesian path and Euclidean tracking-error histories for C3 (external disturbance active from 5 to 9 s) and C4 (10% uncertainty from 1 to 5 s followed by disturbance from 5 to 9 s). Each case compares CTC with CTC + bounded residual DDPG.

Panel of Figure 18 — Figure 18: Ten-seed Cartesian-error summary for the four scenarios. Bars report the mean and error bars report one standard deviation for panel (a) RMS error and panel (b) peak error. Seed-dependent variability appears in the disturbance-enabled scenarios; the nominal and uncertainty-only scenarios are deterministic in this evaluation.

Figure 19: Interval-wise Cartesian RMS error averaged over ten seed runs for 0–1 s (initial transient), 1–5 s (parametric uncertainty), 5–9 s (external disturbance), and 9–10 s (post-disturbance recovery).

Case	$x_{0}$ (m)	$y_{0}$ (m)	Role
IC1	$-0.85$	$0.05$	Upper-side offset
IC2	$-0.85$	$0.00$	Mild offset
IC3	$-0.90$	$-0.05$	Nominal main-run initial state
IC4	$-0.90$	$-0.10$	Lower-side offset

Table 13: Cartesian initial conditions used in the local sensitivity test.

Figure 20: Cartesian trajectory-transfer evaluation from four nearby initial configurations under the combined uncertainty/disturbance scenario. The dotted curve is the desired trajectory, solid magenta curves are the CTC + bounded residual DDPG responses, dashed black curves are the CTC responses, and filled markers denote the four initial Cartesian positions. The plotting orientation is horizontal

Y

and vertical

X

Figure 21: Residual-action usage relative to the joint-wise limits. Panel (a) reports

\rho_{j}^{\max}=\max_{t}|\tau_{\mathrm{RL},j}|/\tau_{\mathrm{RL},j}^{\max}

. Panel (b) reports the percentage of samples satisfying

|\tau_{\mathrm{RL},j}|/\tau_{\mathrm{RL},j}^{\max}\geq 0.95

. Joint 1 reaches 0.972 and is near the bound for 0.200% of samples; joints 2 and 3 have no near-limit samples.

Figure 22: Circular-trajectory transfer evaluation under the combined uncertainty/disturbance scenario. Panel (a) compares the desired Cartesian path with the realized CTC and CTC + bounded residual DDPG trajectories; panel (b) reports the Euclidean Cartesian tracking-error norm. Light-blue and light-peach shading denote the 1–5 s uncertainty and 5–9 s disturbance intervals. RMS error decreases from 0.021738 to 0.012052 m; the associated source dataset contains 44 and 101 joint-limit channel-samples, respectively, identifying trajectory-dependent joint-coordinate coverage for subsequent constraint-aware training.

Panel of Figure 23 — Figure 23: Joint-torque components in the representative combined case. Panel (a) shows the nominal computed-torque command $\tau_{\mathrm{CTC},j}$ and panel (b) shows the bounded residual action $\tau_{\mathrm{RL},j}$ for joints $j=1,2,3$ .

Panel of Figure 24 — Figure 24: Controller command and plant disturbance in the representative combined case. Panel (a) reports the combined commanded joint torque $\bm{\tau}_{\mathrm{cmd}}=\bm{\tau}_{\mathrm{ctc}}+\bm{\tau}_{\mathrm{RL}}$ , and panel (b) reports the externally applied disturbance torque $\bm{\tau}_{d}$ . The disturbance enters the simulated plant separately from the cable-allocation command.

Explore similar work

Stability-aware Residual Reinforcement Learning Framework for Robotic Manipulator Disturbance Compensation
Sep 18, 2026Jihong Kim, Joonhyuk Kwon, Hwa Soo Kim +2Robotic Control RL Control
School of Mechanical Engineering, Hanyang University, Seoul 04763, Republic of Korea · Department of Mechanical System Design, Kyonggi University, Suwon 16227, South Korea · School of Electrical Engineering, Kookmin University, Seoul 02707, South Korea
Learning-Based Dynamics Modeling and Robust Control for Tendon-Driven Continuum Robots
Apr 28, 2026Ziqing Zou, Ke Qiu, Fei Wang +3Robot Trajectory Tracking Robot Dynamics Learning
Department of Control Science and Engineering, Zhejiang University, Hangzhou, China
Robust Operational Space Control with Conformal Disturbance Bounds for Safe Redundant Manipulation
Jul 1, 2026Wenhua Liu, Fan Zhang, Qin LinRobotic Control Control Barrier Functions
Department of Engineering Technology and the Department of Electrical and Computer Engineering, University of Houston, Houston, TX 77004, USA.

Sep 18, 2026, cs.ROJ/K move · Enter open · S save

Stability-aware Residual Reinforcement Learning Framework for Robotic Manipulator Disturbance Compensation

Jihong Kim, Joonhyuk Kwon, Hwa Soo Kim, TaeWon Seo+1

School of Mechanical Engineering, Hanyang University, Seoul 04763, Republic of Korea · Department of Mechanical System Design, Kyonggi University, Suwon 16227, South Korea · School of Electrical Engineering, Kookmin University, Seoul 02707, South Korea

Robotic Control RL Control Robust Control Residual Policy Learning Dynamical Systems

Open paper AskPDF arXiv

Although conventional controllers and disturbance observers (DOBs) are the standard for precision tracking in manipulators, they suffer from parameter uncertainty, nonlinear friction, and compound disturbances. This study proposes a residual reinforcement learning DOB framework that pairs an analytical observer with an RL policy. The deterministic baseline operates within a reliable region, whereas the RL policy explicitly targets the residuals that the model cannot capture. To make this compensation disturbance-aware, an estimator network aligns the observation history with a privileged disturbance context, organizing the latent space by disturbance regime and enabling rapid adaptation across disturbance transitions. To guarantee stability, we derived and enforced a state-dependent action bound on the RL policy from an input-to-state stability (ISS) analysis such that the closed loop provably confines the tracking error to a certified envelope for arbitrary policy outputs. Experiments on a 6-DOF manipulator demonstrated consistent improvements in disturbance estimation and tracking, including a 27.8% tracking-error reduction on real hardware under zero-shot sim-to-real transfer and a 38.0% reduction under a base-vibration disturbance that was not observed during training.

Abstract

Explore similar work

Stability-aware Residual Reinforcement Learning Framework for Robotic Manipulator Disturbance Compensation

Learning-Based Dynamics Modeling and Robust Control for Tendon-Driven Continuum Robots

Robust Operational Space Control with Conformal Disturbance Bounds for Safe Redundant Manipulation