Reinforced Iterative Learning Control for elastic joint arms: Guaranteed cross-trajectory generalization under model uncertainties
Authors: Yuri De Santis, Michele Pierallini, Franco Angelini, Manolo Garabini
Organizations: Centro di Ricerca “Enrico Piaggio”, Università di Pisa, Largo Lucio Lazzarino 1, Pisa, 56122, Italy · Dipartimento di Ingegneria dell’Informazione, Università di Pisa, Via G. Caruso 16, Pisa, 56122, Italy
Trajectory tracking for compliant robots is challenging due to nonlinear dynamics and modeling uncertainties, often requiring learning-based methods to avoid explicit parameter identification. Iterative Learning Control (ILC) ensures theoretically high precision and preserves the passive compliance of the robot. However, traditional ILC requires a separate learning phase for each new trajectory. Conversely, Reinforcement Learning (RL) generalizes to new trajectories, but practically suffers from high sample complexity and limited tracking precision. This work proposes Reinforced Iterative Learning Control (RILC) that combines the strengths of both approaches to achieve fast adaptation and guaranteed tracking convergence across diverse trajectories. By establishing a concurrent coupling between the two controllers, RILC overcomes the high initial errors and slow adaptation of traditional ILC, enabling high-performance tracking from the very first trials. In this architecture, ILC provides high-quality samples, while RL contributes as a generalizing best-effort policy across iterations. Additionally, a theoretical proof is presented to show that the RL component preserves ILC stability and, under specific conditions, can also reduce the error bound asymptotically. The efficacy of this framework is demonstrated through simulation and experiments on an underactuated 4-degree-of-freedom serial elastic robot using different trajectories and modeling mismatches.
Figures & tables
Figure 1: Reinforced Iterative Learning Control (RILC) scheme. The RILC combines Reinforcement Learning (RL) and Iterative Learning Control (ILC) terms (green block). Each process (blue blocks) is populated by the iterations required for convergence to a predefined desired trajectory (grey block). The ILC is updated using past iterations 0,⋯,j−1 , while the RL control is determined by the current ILC action and information gathered from the current process Pk . A buffer stores previously executed processes 0,⋯,Pk−1 , and it is used to update the RL policy.
Symbol
Description
j
Iteration index
xj
State vector at iteration j
yj
Output
ej
Tracking error yd−yj
uˉj
ILC feedforward control input
u~ν(ξj)
RL control contribution, parametrized by ν
Table 1: Principal notation used throughout the paper.
Figure 2: Fig.s 2a and 2b represent the control scheme of Reinforced Iterative Learning Control (RILC) used in the training phase and experiments, respectively. In simulations, a stochastic policy (to explore the sample space) and parameters from Tab. 2 are employed, while a deterministic policy is deployed in real-world experiments. For further details, see Alg. 1 .
RL Parameters
Value
low level control frequency [ Hz ]
500
high level control frequency [ Hz ]
200
policy frequency [ Hz ]
50
RL control limits [ N ]
[−2,2]
actuator position limits [ rad ]
[−π,π]
number of policy update
200
Table 2: Parameters for Reinforced Iterative Learning Control (RILC) training with PPO algorithm. We denote with U(a,b) the uniform distribution with the bound a and b .
Figure 3: Reward functions evolution of baseline Reinforcement Learning (RL) and Reinforced Iterative Learning Control (RILC) approaches w.r.t. policy updating. The total ’Reward’, ( 46 )-( 50 ), and partial cumulative reward dependent on the error, i.e., ’Reward (Error)’, ( 46 ), are computed as the mean across all training environments during the simulation.
Table 3: RMSE statistics for Iterative Learning Control (ILC), Norm-Optimal Iterative Learning Control (NOILC), Combined Iterative Learning Control (C-ILC), Reinforcement Learning (RL), and Reinforced Iterative Learning Control (RILC), computed over 3000 trajectories. Suffixes ’-N’ and ’-M’ denote nominal and mismatched conditions, while numbers indicate the iteration index.
Figure 4: Simulation results analysis for different controllers: Iteative Learning Control (ILC), Norm-Optimal Iterative Learning Control (NOILC), Combined Iterative Learning Control (C-ILC), Reinforcement Learning (RL), Reinforced Iterative Learning Control (RILC). (a) RMSE comparison over 3000 randomized trajectories: each point indicates a yd(tf) , color-mapped by the resulting RMSE. (b)-(c) Tracking performance and RMSE evolution for the specific trajectory yd(tf)=[2.0,−1.0]⊤ . The suffixes ’N’ and ’M’ denote nominal and mismatched conditions, while the numerical suffixes indicate the iteration index.
Figure 5: RMSE and crossover analysis for the four real-robot target configurations. Dashed and dash-dotted lines show the fitted first-order models εj=ε0Ωj for Iterative Learning Control (ILC) and Reinforced Iterative Learning Control (RILC). The vertical line marks the crossover iteration j∗ of the fitted models.
Figure 6: Experimental analysis for the tracking task with target yd(tf)=[2.0,−1.0]⊤ . (a) Decomposition of control contributions: Feedback (FB), Model-Based (MB), Iterative Learning Control (ILC), Reinforcement Learning (RL), and Total control, at iterations 0 -th and 20 -th. (b) Output tracking evolution comparing the baseline ILC and the proposed RILC approach.
Figure 7: Photo-sequence of Reinforced Iterative Learning Control (RILC) on tracking trajectory with yd(tf)=[2.0,−1.0]⊤ .
Target
εˉ0
ε0
Ωˉ
Ω
j∗
[1.0,−1.0]⊤
0.660
0.140
0.666
0.812
7.9
[1.0,1.0]⊤
0.682
0.063
0.762
0.973
9.8
[2.0,−1.0]⊤
0.885
0.261
0.625
0.794
5.1
[π/2,π/2]⊤
0.989
0.094
0.730
0.884
12.2
Table 4: Crossover analysis for the real-robot experiments. Parameters εˉ0 and ε0 are the initial tracking errors for ILC and RILC, respectively; Ωˉ and Ω are the corresponding fitted contraction rates; and j∗ is the crossover iteration.
Figure 8: Fig. 8a is the photo configuration of the disturbance applying. Fig. 8b shows RMSE over iteration, a disturbance is applied to the robot in iteration 11 -th. Comparison of Reinforcement Learning (RL), Iterative Learning Control (ILC), and Reinforced Iterative Learning Control (RILC), approaches for the desired trajectory with yd(tf)=[2.0,−1.0]⊤ .
Figure 9: Output tracking evolution for desired trajectory with yd(tf)=[2.0,−1.0]⊤ , without and with (w/) disturbance, in iterations 12 -th, 13 -th, 22 -nd. Comparison of Reinforcement Learning (RL), Iterative Learning Control (ILC), and Reinforced Iterative Learning Control (RILC), approaches.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Simulation results comparison between ILC and RILC for link-side position tracking y(t)=q(t) for final position yd(tf)=[2.0,−1.0]⊤ . (a) Motor and link position trajectories. (b) RMSE evolution over iterations.
Robot trajectory tracking develops methods that make robots follow desired paths or motions accurately despite disturbances, uncertainty, and physical constraints.
Fast execution of contact-rich manipulation is critical for practical deployment, yet providing fast demonstrations for imitation learning (IL) remains challenging: humans cannot demonstrate at high speed, and naively accelerating demonstrations alters contact dynamics and induces large tracking errors. We present a method to autonomously refine time-accelerated demonstrations by repurposing Iterative Reference Learning Control (IRLC) to iteratively update the reference trajectory from observed tracking errors. However, applying IRLC directly at high speed tends to produce larger early-iteration errors and less stable transients. To address this issue, we propose Incremental Iterative Reference Learning Control (I2RLC), which gradually increases the speed while updating the reference, yielding high-fidelity trajectories. We validate on real-robot whiteboard erasing and peg-in-hole tasks using a teleoperation setup with a compliance-controlled follower and a 3D-printed haptic leader. Both IRLC and I2RLC achieve up to 10x faster demonstrations with reduced tracking error; moreover, I2RLC improves spatial similarity to the original trajectories by 22.5% on average over IRLC across three tasks and multiple speeds (3x-10x). We then use the refined trajectories to train IL policies; the resulting policies execute faster than the demonstrations and achieve 100% success rates in the peg-in-hole task at both seen and unseen positions, with I2RLC-trained policies exhibiting lower contact forces than those trained on IRLC-refined demonstrations. These results indicate that gradual speed scheduling coupled with reference adaptation provides a practical path to fast, contact-rich IL.
Koki Yamane, Cristian C. Beltran-Hernandez, Steven Oh +2
OMRON SINIC X Corporation, Bunkyo-ku, Tokyo 113-0033, Japan · Waseda University, Shinjuku-ku, Tokyo, 169-8050, Japan · University of Tsukuba, Tsukuba, Ibaraki 305-8573, Japan
Repetitive motion tasks are common in robotics, but performance can degrade over time due to environmental changes and robot wear and tear. Iterative learning control (ILC) improves performance by using information from previous iterations to compensate for expected errors in future iterations. This work incorporates the use of Quasi-Periodic Gaussian Processes (QPGPs) into a predictive ILC framework to model and forecast disturbances and drift across iterations. Using a recent structural equation formulation of QPGPs, the proposed approach enables efficient inference with complexity O(p3) instead of O(i2p3), where p denotes the number of points within an iteration and i represents the total number of iterations, specially for larger i. This formulation also enables parameter estimation without loss of information, making continual GP learning computationally feasible within the control loop. By predicting next-iteration error profiles rather than relying only on past errors, the controller achieves faster convergence and maintains this under time-varying disturbances. We benchmark the method against both standard ILC and conventional Gaussian Process (GP)-based predictive ILC on three tasks, autonomous vehicle trajectory tracking, a three-link robotic manipulator, and a real-world Stretch robot experiment. Across all cases, the proposed approach converges faster and remains robust under injected and natural disturbances while reducing computational cost. This highlights its practicality across a range of repetitive dynamical systems.
IITB-Monash Research Academy, IIT Bombay, Mumbai, India · Department of Mathematics, IIT Bombay, Mumbai, India · Department of Electrical and Computer Systems Engineering, Monash University, Clayton, Melbourne, Australia
A significant limitation of Deep Reinforcement Learning (DRL) is the stochastic uncertainty in actions generated during exploration-exploitation, which poses substantial safety risks during both training and deployment. In industrial process control, the lack of formal stability and convergence guarantees further inhibits adoption of DRL methods by practitioners. Conversely, Iterative Learning Control (ILC) represents a well-established autonomous control methodology for repetitive systems, particularly in batch process optimization. ILC achieves desired control performance through iterative refinement of control laws, either between consecutive batches or within individual batches, to compensate for both repetitive and non-repetitive disturbances. This study introduces an Iterative Learning Control-Informed Reinforcement Learning (IL-CIRL) framework for training DRL controllers in dual-layer batch-to-batch and within-batch control architectures for batch processes. The proposed method incorporates Kalman filter-based state estimation within the iterative learning structure to guide DRL agents toward control policies that satisfy operational constraints and ensure stability guarantees. This approach enables the systematic design of DRL controllers for batch processes operating under multiple disturbance conditions.
Runze Lin, Ziqi Zhuo, Junghui Chen +2
State Key Laboratory of Industrial Control Technology, Institute of Cyber-Systems and Control, Zhejiang University, Hangzhou 310027, China · Department of Chemical Engineering, Chung-Yuan Christian University, Taoyuan 32023, Taiwan, R.O.C.