Runtime-Incremental Transformer for Reinforcement-Learning-Based Adaptive Control
Authors: Giansalvo Cirrincione, Adriano Fagiolini
Organizations: Laboratoire LTI, Université de Picardie Jules Verne, Amiens, France · MIRPALab, Department of Engineering, University of Palermo, 90128 Palermo, Italy
Learning-based adaptive control of robotic manipulators with non-observable friction memory has been addressed by attention- based meta-controllers whose number of attention heads is fixed before training and is tuned by costly offline search. At long memory horizons, such fixed-capacity controllers are prone to catastrophic failures on a sizeable fraction of training seeds. The present paper introduces a runtime mechanism that grows and prunes the heads of the attention block during reinforcement learning, governed by two signals: the effective rank of the on-policy context distribution, which triggers growth when representational capacity becomes insufficient, and the per-head output magnitude, which flags redundant heads for removal. Policy continuity at growth events and a quantitative bound at prune events are established analytically. On a two- link manipulator with Stribeck friction, the proposed mechanism attains full success across all memory regimes, eliminating the long-horizon failure mode and removing the need for offline tuning of the head count.
Figures & tables
Figure 1: Decision logic of the runtime controller Crt at each check step. A grow event is triggered when the empirical effective rank ρ^α exceeds the current head count scaled by (1+εgrow) ; a prune event is triggered when the relative output πi of the weakest active head falls below εprune . At most one event fires per check; consecutive events are separated by a cooldown of Δgrace steps.
INCRT-1L (static)
Runtime-INCRT
τz [s]
Δ%
Fails
Δ%±CI95
Succ.
Δ (pts)
1
−40.0%
0/10
−45.82%±
7.09%
10/10
−5.8
2
−35.0%
0/10
−50.85%±
8.38%
10/10
−15.8
5
+5.0%
4/10
−54.15%±
5.43%
10/10
−59.1
Table 1: Relative RMSE improvement Δ% over the computed-torque baseline ( success : eval. RMSE <0.10 rad; intervals are 95% Student- t CIs, n=10 ).
Figure 2: Mean relative RMSE improvement Δ% (with 95% Student- t confidence intervals) of Runtime-INCRT (blue, n=10 ) and static INCRT-1L [ 11 ] (grey) over the computed-torque baseline, for each memory regime τz∈{1,2,5} s. Error bars for INCRT-1L are not reported in the parent work.
Figure 3: Per-seed distribution of Δ% for Runtime-INCRT across all 30 main-campaign runs ( n=10 per regime). Each dot represents one seed; the horizontal bar marks the mean. Seed variance is smallest at τz=5 s — the regime where INCRT-1L was most unstable — confirming the stabilising effect of the runtime mechanism.
Figure 4: Evolution of the active head count K(t) during training for a representative seed at each memory horizon τz∈{1,2,5} s. Vertical dashed lines mark grow events. In all three cases the controller performs seven growth steps and saturates at K=Kmax=8 around step 1.9×104 , well before the end of training; the growth schedule is effectively task-invariant.
Kmax
Δ%±CI95
Std
n
2
−57.21%±
8.64%
6.96%
5
4
−48.97%±
9.81%
7.90%
5
8
−50.84%±
8.38%
11.71%
10
16
−43.82%±
12.78%
10.30%
5
Static INCRT-1L ( K=7 )
Δ%=−35.0%
Table 2: Effect of Kmax at τz=2 s. The Kmax=8 row reuses the n=10 main-campaign seeds. Dashed reference: static INCRT-1L with BO-tuned K=7 ( Δ%=−35.0% ).
Figure 5: Effect of the head-count cap Kmax on Runtime-INCRT performance at τz=2 s (25 runs total; n=5 except Kmax=8 where n=10 ). Left: mean Δ% with 95% CI; black dots show individual seeds. The dashed red line is the static INCRT-1L reference ( K=7 , BO-tuned, Δ%=−35.0% ). Right: per-seed standard deviation. Smaller Kmax yields better mean performance and lower variance.
Δgrace
Δ%±CI95
Std
Fails
Sched. [k]
1 000
−41.16%±
24.45%
19.69%
1/5
13.5
2 000
−51.66%±
15.01%
12.09%
0/5
19.5
5 000
−36.61%±
38.31%
30.85%
1/5
37.5
Table 3: Sensitivity to Δgrace at τz=2 s (5 seeds per value). Failure : eval. RMSE >0.10 rad. Schedule length in units of 103 steps.
Figure 6: Sensitivity of Runtime-INCRT to the cooldown parameter Δgrace at τz=2 s (5 seeds per value). Left: mean Δ% with 95% CI. Right: per-seed standard deviation. Performance is non-monotone: both the fast ( 1000 steps) and the slow ( 5000 steps) schedule degrade the mean and inflate variance relative to the default ( 2000 steps), confirming a characteristic curriculum time-scale.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Role
Architecture
Kmax
8
head-count cap
Kinit
1
initial active heads
Kmin
1
minimum active heads
dk
16
per-head key/value dim.
dmodel
128
=Kmax⋅dk
Appendix
Table 4: Runtime-INCRT hyperparameters (same values across all τz∈{1,2,5} s).
Adaptive control of Euler-Lagrange systems is challenging when friction is governed by a finite-horizon internal state that is not directly observable from joint measurements. In this setting, the measured closed-loop state is no longer Markovian, and standard certainty-equivalence adaptive laws may lose their convergence guarantees. The paper proposes a meta-control architecture in which the gains of a computed-torque controller are generated by a self-attention block processing a short window of recent motion history. The number of attention heads is selected before policy training through a surrogate analysis of the autocovariance of the memory-state gradient along the temporal window. This surrogate is based on a temporal adaptation of an incremental rank-tracking framework previously developed by the authors. The selected head count is then fixed and used as an architectural hyperparameter in a reinforcement-learning stage, where the policy is trained under a shielded admissibility constraint. The approach is tested on a 2-DOF manipulator with nonlinear friction and variable payload. In the short and matched memory regimes, the single-layer attention-only meta-controller outperforms a deeper Transformer baseline, with tracking-error reductions of 12 and 19 percentage points, respectively. The reported effect sizes are large, with d approximately -1.1 and -2.1, and Mann-Whitney p < 0.05 in both cases. In the long memory regime, however, the advantage disappears. Four out of ten training runs show either divergence or payload-invariant policy collapse, revealing a weakness in the static Phase-1 head-count prescription. This motivates moving rank-tracking inside the reinforcement-learning loop, allowing attention heads to be pruned or grown at runtime instead of fixed before training.
Giansalvo Cirrincione, Adriano Fagiolini
Laboratoire LTI, Université de Picardie Jules Verne, Amiens, France · MIRPALab, Department of Engineering, University of Palermo, 90128 Palermo, Italy
Minimum attention applies the least action principle in changes of control concerning state and time, first proposed by Brockett. The involved regularization is highly relevant in emulating biological control, such as motor learning. We apply minimum attention in reinforcement learning (RL) as part of the rewards and investigate its connection to meta-learning and stabilization. Specifically, model-based meta-learning with minimum attention is explored in high-dimensional nonlinear dynamics. Ensemble-based model learning and gradient-based meta-policy learning are alternately performed. Empirically, minimum attention improves fast adaptation in few shots and reduces variance from perturbations of the model and environment, compared to model-free and model-based RL baseline, and yields consistent gain when integrated into modern world models (DreamerV3, MAMBA). Furthermore, the minimum attention demonstrates an improvement in energy efficiency.
Shashank Gupta, Pilhwa Lee
Department of EECS University of Michigan Ann Arbor, MI 48109 · Department of Mathematics Morgan State University Baltimore, MD 21251
Most robotic systems rely on fixed-frequency discrete-time controllers, creating a trade-off between the efficiency of low-frequency control and the responsiveness of high-frequency feedback. As a result, systems typically default to high control rates for robustness, at the cost of wasted inference and unnecessary actuation. Addressing this, we introduce Time-Adaptive Robotic Control (TARC), a reinforcement learning framework in which the policy jointly predicts a control action and its duration of application. TARC learns temporally extended actions by optimizing task performance under soft or hard constraints on the number of control switches, enabling adaptive modulation of control rates. We evaluate TARC on two robotic hardware platforms: a high-speed RC car and the Unitree Go1 quadruped, and on a vision-language action model in simulation, where each query incurs a costly transformer forward pass. Across all settings, TARC matches the performance of high-frequency discrete-time controllers while operating at less than half their control frequency. Unlike fixed-rate controllers, TARC adapts its control frequency online, allocating high-frequency feedback only when required.