Reinforced Iterative Learning Control for elastic joint arms: Guaranteed cross-trajectory generalization under model uncertainties

Yuri De Santis; Michele Pierallini; Franco Angelini; Manolo Garabini

cs.ROOct 20, 2026

Reinforced Iterative Learning Control for elastic joint arms: Guaranteed cross-trajectory generalization under model uncertainties

Authors: Yuri De Santis, Michele Pierallini, Franco Angelini, Manolo Garabini

Organizations: Centro di Ricerca “Enrico Piaggio”, Università di Pisa, Largo Lucio Lazzarino 1, Pisa, 56122, Italy · Dipartimento di Ingegneria dell’Informazione, Università di Pisa, Via G. Caruso 16, Pisa, 56122, Italy

Abstract

Trajectory tracking for compliant robots is challenging due to nonlinear dynamics and modeling uncertainties, often requiring learning-based methods to avoid explicit parameter identification. Iterative Learning Control (ILC) ensures theoretically high precision and preserves the passive compliance of the robot. However, traditional ILC requires a separate learning phase for each new trajectory. Conversely, Reinforcement Learning (RL) generalizes to new trajectories, but practically suffers from high sample complexity and limited tracking precision. This work proposes Reinforced Iterative Learning Control (RILC) that combines the strengths of both approaches to achieve fast adaptation and guaranteed tracking convergence across diverse trajectories. By establishing a concurrent coupling between the two controllers, RILC overcomes the high initial errors and slow adaptation of traditional ILC, enabling high-performance tracking from the very first trials. In this architecture, ILC provides high-quality samples, while RL contributes as a generalizing best-effort policy across iterations. Additionally, a theoretical proof is presented to show that the RL component preserves ILC stability and, under specific conditions, can also reduce the error bound asymptotically. The efficacy of this framework is demonstrated through simulation and experiments on an underactuated 4-degree-of-freedom serial elastic robot using different trajectories and modeling mismatches.

Figures & tables

Figure 1: Reinforced Iterative Learning Control (RILC) scheme. The RILC combines Reinforcement Learning (RL) and Iterative Learning Control (ILC) terms (green block). Each process (blue blocks) is populated by the iterations required for convergence to a predefined desired trajectory (grey block). The ILC is updated using past iterations $0,\cdots,j-1$ , while the RL control is determined by the current ILC action and information gathered from the current process $P_{k}$ . A buffer stores previously executed processes $0,\cdots,P_{k-1}$ , and it is used to update the RL policy.

Symbol	Description
$j$	Iteration index
$x_{j}$	State vector at iteration $j$
$y_{j}$	Output
$e_{j}$	Tracking error $y_{d}-y_{j}$
$\bar{u}_{j}$	ILC feedforward control input
$\tilde{u}_{\nu}(\xi_{j})$	RL control contribution, parametrized by $\nu$

Table 1: Principal notation used throughout the paper.

Panel of Figure 2 — Figure 2: Fig.s 2a and 2b represent the control scheme of Reinforced Iterative Learning Control (RILC) used in the training phase and experiments, respectively. In simulations, a stochastic policy (to explore the sample space) and parameters from Tab. 2 are employed, while a deterministic policy is deployed in real-world experiments. For further details, see Alg. 1 .

RL Parameters	Value
low level control frequency [ $\mathrm{Hz}$ ]	$500$
high level control frequency [ $\mathrm{Hz}$ ]	$200$
policy frequency [ $\mathrm{Hz}$ ]	$50$
RL control limits [ $\mathrm{N}$ ]	$[-2,2]$
actuator position limits [ $\mathrm{rad}$ ]	$[-\pi,\pi]$
number of policy update	$200$

Table 2: Parameters for Reinforced Iterative Learning Control (RILC) training with PPO algorithm. We denote with

\mathcal{U}(a,b)

the uniform distribution with the bound

a

and

b

Figure 3: Reward functions evolution of baseline Reinforcement Learning (RL) and Reinforced Iterative Learning Control (RILC) approaches w.r.t. policy updating. The total ’Reward’, ( 46 )-( 50 ), and partial cumulative reward dependent on the error, i.e., ’Reward (Error)’, ( 46 ), are computed as the mean across all training environments during the simulation.

Table 3: RMSE statistics for Iterative Learning Control (ILC), Norm-Optimal Iterative Learning Control (NOILC), Combined Iterative Learning Control (C-ILC), Reinforcement Learning (RL), and Reinforced Iterative Learning Control (RILC), computed over 3000 trajectories. Suffixes ’-N’ and ’-M’ denote nominal and mismatched conditions, while numbers indicate the iteration index.

Panel of Figure 4 — Figure 4: Simulation results analysis for different controllers: Iteative Learning Control (ILC), Norm-Optimal Iterative Learning Control (NOILC), Combined Iterative Learning Control (C-ILC), Reinforcement Learning (RL), Reinforced Iterative Learning Control (RILC). (a) RMSE comparison over 3000 randomized trajectories: each point indicates a $y_{\textnormal{d}}(t_{\textnormal{f}})$ , color-mapped by the resulting RMSE. (b)-(c) Tracking performance and RMSE evolution for the specific trajectory $y_{\textnormal{d}}(t_{\textnormal{f}})=[2.0,-1.0]^{\top}$ . The suffixes ’N’ and ’M’ denote nominal and mismatched conditions, while the numerical suffixes indicate the iteration index.

Figure 5: RMSE and crossover analysis for the four real-robot target configurations. Dashed and dash-dotted lines show the fitted first-order models

\varepsilon_{j}=\varepsilon_{0}\Omega^{j}

for Iterative Learning Control (ILC) and Reinforced Iterative Learning Control (RILC). The vertical line marks the crossover iteration

j^{*}

of the fitted models.

Panel of Figure 6 — Figure 6: Experimental analysis for the tracking task with target $y_{\textnormal{d}}(t_{\textnormal{f}})=[2.0,-1.0]^{\top}$ . (a) Decomposition of control contributions: Feedback (FB), Model-Based (MB), Iterative Learning Control (ILC), Reinforcement Learning (RL), and Total control, at iterations $0$ -th and $20$ -th. (b) Output tracking evolution comparing the baseline ILC and the proposed RILC approach.

Panel of Figure 7 — Figure 7: Photo-sequence of Reinforced Iterative Learning Control (RILC) on tracking trajectory with $y_{\textnormal{d}}(t_{\textnormal{f}})=[2.0,-1.0]^{\top}$ .

Target	$\bar{\varepsilon}_{0}$	$\varepsilon_{0}$	$\bar{\Omega}$	$\Omega$	$j^{*}$
$[1.0,\,-1.0]^{\top}$	0.660	0.140	0.666	0.812	7.9
$[1.0,\,1.0]^{\top}$	0.682	0.063	0.762	0.973	9.8
$[2.0,\,-1.0]^{\top}$	0.885	0.261	0.625	0.794	5.1
$[\pi/2,\,\pi/2]^{\top}$	0.989	0.094	0.730	0.884	12.2

Table 4: Crossover analysis for the real-robot experiments. Parameters

\bar{\varepsilon}_{0}

and

\varepsilon_{0}

are the initial tracking errors for ILC and RILC, respectively;

\bar{\Omega}

and

\Omega

are the corresponding fitted contraction rates; and

j^{*}

is the crossover iteration.

Panel of Figure 8 — Figure 8: Fig. 8a is the photo configuration of the disturbance applying. Fig. 8b shows RMSE over iteration, a disturbance is applied to the robot in iteration $11$ -th. Comparison of Reinforcement Learning (RL), Iterative Learning Control (ILC), and Reinforced Iterative Learning Control (RILC), approaches for the desired trajectory with $y_{\textnormal{d}}(t_{\textnormal{f}})=[2.0,-1.0]^{\top}$ .

Figure 9: Output tracking evolution for desired trajectory with

y_{\textnormal{d}}(t_{\textnormal{f}})=[2.0,-1.0]^{\top}

, without and with (w/) disturbance, in iterations

12

-th,

13

-th,

22

-nd. Comparison of Reinforcement Learning (RL), Iterative Learning Control (ILC), and Reinforced Iterative Learning Control (RILC), approaches.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure 10: Simulation results comparison between ILC and RILC for link-side position tracking $y(t)=q(t)$ for final position $y_{\textnormal{d}}(t_{\textnormal{f}})=[2.0,-1.0]^{\top}$ . (a) Motor and link position trajectories. (b) RMSE evolution over iterations.

Robot Trajectory Tracking

Robot trajectory tracking develops methods that make robots follow desired paths or motions accurately despite disturbances, uncertainty, and physical constraints.

See all 131 papers

Refinement of Accelerated Demonstrations via Incremental Iterative Reference Learning Control for Fast Contact-Rich Imitation Learning
Apr 18, 2026Koki Yamane, Cristian C. Beltran-Hernandez, Steven Oh +2Robot Trajectory Tracking Contact-Rich Robotic Manipulation
OMRON SINIC X Corporation, Bunkyo-ku, Tokyo 113-0033, Japan · Waseda University, Shinjuku-ku, Tokyo, 169-8050, Japan · University of Tsukuba, Tsukuba, Ibaraki 305-8573, Japan
Quasi-Periodic Gaussian Process Predictive Iterative Learning Control
Feb 20, 2026Unnati Nigam, Radhendushka Srivastava, Faezeh Marzbanrad +1Iterative Learning Control Robotic Control
IITB-Monash Research Academy, IIT Bombay, Mumbai, India · Department of Mathematics, IIT Bombay, Mumbai, India · Department of Electrical and Computer Systems Engineering, Monash University, Clayton, Melbourne, Australia
Iterative Learning Control-Informed Reinforcement Learning for Batch Process Control
Mar 16, 2026Runze Lin, Ziqi Zhuo, Junghui Chen +2Iterative Learning Control RL Control
State Key Laboratory of Industrial Control Technology, Institute of Cyber-Systems and Control, Zhejiang University, Hangzhou 310027, China · Department of Chemical Engineering, Chung-Yuan Christian University, Taoyuan 32023, Taiwan, R.O.C.

Apr 18, 2026J/K move · Enter open · S save

Refinement of Accelerated Demonstrations via Incremental Iterative Reference Learning Control for Fast Contact-Rich Imitation Learning

Koki Yamane, Cristian C. Beltran-Hernandez, Steven Oh, Masashi Hamaya+1

OMRON SINIC X Corporation, Bunkyo-ku, Tokyo 113-0033, Japan · Waseda University, Shinjuku-ku, Tokyo, 169-8050, Japan · University of Tsukuba, Tsukuba, Ibaraki 305-8573, Japan

Robot Trajectory Tracking Contact-Rich Robotic Manipulation Learning from Demonstration Robot Imitation Learning Iterative Learning Control Adaptive Control

Open paper AskPDF arXiv

Fast execution of contact-rich manipulation is critical for practical deployment, yet providing fast demonstrations for imitation learning (IL) remains challenging: humans cannot demonstrate at high speed, and naively accelerating demonstrations alters contact dynamics and induces large tracking errors. We present a method to autonomously refine time-accelerated demonstrations by repurposing Iterative Reference Learning Control (IRLC) to iteratively update the reference trajectory from observed tracking errors. However, applying IRLC directly at high speed tends to produce larger early-iteration errors and less stable transients. To address this issue, we propose Incremental Iterative Reference Learning Control (I2RLC), which gradually increases the speed while updating the reference, yielding high-fidelity trajectories. We validate on real-robot whiteboard erasing and peg-in-hole tasks using a teleoperation setup with a compliance-controlled follower and a 3D-printed haptic leader. Both IRLC and I2RLC achieve up to 10x faster demonstrations with reduced tracking error; moreover, I2RLC improves spatial similarity to the original trajectories by 22.5% on average over IRLC across three tasks and multiple speeds (3x-10x). We then use the refined trajectories to train IL policies; the resulting policies execute faster than the demonstrations and achieve 100% success rates in the peg-in-hole task at both seen and unseen positions, with I2RLC-trained policies exhibiting lower contact forces than those trained on IRLC-refined demonstrations. These results indicate that gradual speed scheduling coupled with reference adaptation provides a practical path to fast, contact-rich IL.

Abstract

Robot Trajectory Tracking

Explore similar work

Refinement of Accelerated Demonstrations via Incremental Iterative Reference Learning Control for Fast Contact-Rich Imitation Learning

Quasi-Periodic Gaussian Process Predictive Iterative Learning Control

Iterative Learning Control-Informed Reinforcement Learning for Batch Process Control