Runtime-Incremental Transformer for Reinforcement-Learning-Based Adaptive Control

Giansalvo Cirrincione; Adriano Fagiolini

cs.ROSep 11, 2026

Runtime-Incremental Transformer for Reinforcement-Learning-Based Adaptive Control

Authors: Giansalvo Cirrincione, Adriano Fagiolini

Organizations: Laboratoire LTI, Université de Picardie Jules Verne, Amiens, France · MIRPALab, Department of Engineering, University of Palermo, 90128 Palermo, Italy

Abstract

Learning-based adaptive control of robotic manipulators with non-observable friction memory has been addressed by attention- based meta-controllers whose number of attention heads is fixed before training and is tuned by costly offline search. At long memory horizons, such fixed-capacity controllers are prone to catastrophic failures on a sizeable fraction of training seeds. The present paper introduces a runtime mechanism that grows and prunes the heads of the attention block during reinforcement learning, governed by two signals: the effective rank of the on-policy context distribution, which triggers growth when representational capacity becomes insufficient, and the per-head output magnitude, which flags redundant heads for removal. Policy continuity at growth events and a quantitative bound at prune events are established analytically. On a two- link manipulator with Stribeck friction, the proposed mechanism attains full success across all memory regimes, eliminating the long-horizon failure mode and removing the need for offline tuning of the head count.

Figures & tables

Panel of Figure 1 — Figure 1: Decision logic of the runtime controller $\mathcal{C}_{\mathrm{rt}}$ at each check step. A grow event is triggered when the empirical effective rank $\hat{\rho}_{\alpha}$ exceeds the current head count scaled by $(1+\varepsilon_{\mathrm{grow}})$ ; a prune event is triggered when the relative output $\pi_{i}$ of the weakest active head falls below $\varepsilon_{\mathrm{prune}}$ . At most one event fires per check; consecutive events are separated by a cooldown of $\Delta_{\mathrm{grace}}$ steps.

$\tau_{z}$ [s]	$\Delta\%$	Fails	$\Delta\%\pm\mathrm{CI}_{95}$		Succ.	$\Delta$ (pts)
	INCRT-1L (static)		Runtime-INCRT
1	$-40.0\%$	0/10	$-45.82\%$ $\pm$	$7.09\%$	10/10	$-5.8$
2	$-35.0\%$	0/10	$-50.85\%$ $\pm$	$8.38\%$	10/10	$-15.8$
5	$+5.0\%$	4/10	$-54.15\%$ $\pm$	$5.43\%$	10/10	$-59.1$

Table 1: Relative RMSE improvement

\Delta\%

over the computed-torque baseline ( success : eval. RMSE

<0.10

rad; intervals are

95\%

Student-

t

CIs,

n=10

Figure 2: Mean relative RMSE improvement $\Delta\%$ (with $95\%$ Student- $t$ confidence intervals) of Runtime-INCRT (blue, $n=10$ ) and static INCRT-1L [ 11 ] (grey) over the computed-torque baseline, for each memory regime $\tau_{z}\in\{1,2,5\}$ s. Error bars for INCRT-1L are not reported in the parent work.

Figure 3: Per-seed distribution of $\Delta\%$ for Runtime-INCRT across all 30 main-campaign runs ( $n=10$ per regime). Each dot represents one seed; the horizontal bar marks the mean. Seed variance is smallest at $\tau_{z}=5$ s — the regime where INCRT-1L was most unstable — confirming the stabilising effect of the runtime mechanism.

Figure 4: Evolution of the active head count $K(t)$ during training for a representative seed at each memory horizon $\tau_{z}\in\{1,2,5\}$ s. Vertical dashed lines mark grow events. In all three cases the controller performs seven growth steps and saturates at $K=K_{\max}=8$ around step $1.9\times 10^{4}$ , well before the end of training; the growth schedule is effectively task-invariant.

$K_{\max}$	$\Delta\%\pm\mathrm{CI}_{95}$		Std	$n$
2	$-57.21\%$ $\pm$	$8.64\%$	$6.96\%$	5
4	$-48.97\%$ $\pm$	$9.81\%$	$7.90\%$	5
8	$-50.84\%$ $\pm$	$8.38\%$	$11.71\%$	10
16	$-43.82\%$ $\pm$	$12.78\%$	$10.30\%$	5
Static INCRT-1L ( $K=7$ )
$\Delta\%=-35.0\%$

Table 2: Effect of

K_{\max}

\tau_{z}=2

s. The

K_{\max}=8

row reuses the

n=10

main-campaign seeds. Dashed reference: static INCRT-1L with BO-tuned

K=7

(

\Delta\%=-35.0\%

Figure 5: Effect of the head-count cap $K_{\max}$ on Runtime-INCRT performance at $\tau_{z}=2$ s (25 runs total; $n=5$ except $K_{\max}=8$ where $n=10$ ). Left: mean $\Delta\%$ with $95\%$ CI; black dots show individual seeds. The dashed red line is the static INCRT-1L reference ( $K=7$ , BO-tuned, $\Delta\%=-35.0\%$ ). Right: per-seed standard deviation. Smaller $K_{\max}$ yields better mean performance and lower variance.

$\Delta_{\mathrm{grace}}$	$\Delta\%\pm\mathrm{CI}_{95}$		Std	Fails	Sched. [k]
1 000	$-41.16\%$ $\pm$	$24.45\%$	$19.69\%$	$1/5$	$13.5$
2 000	$-51.66\%$ $\pm$	$15.01\%$	$12.09\%$	$0/5$	$19.5$
5 000	$-36.61\%$ $\pm$	$38.31\%$	$30.85\%$	$1/5$	$37.5$

Table 3: Sensitivity to

\Delta_{\mathrm{grace}}

\tau_{z}=2

s (5 seeds per value). Failure : eval. RMSE

>0.10

rad. Schedule length in units of

10^{3}

steps.

Figure 6: Sensitivity of Runtime-INCRT to the cooldown parameter $\Delta_{\mathrm{grace}}$ at $\tau_{z}=2$ s (5 seeds per value). Left: mean $\Delta\%$ with $95\%$ CI. Right: per-seed standard deviation. Performance is non-monotone: both the fast ( $1\,000$ steps) and the slow ( $5\,000$ steps) schedule degrade the mean and inflate variance relative to the default ( $2\,000$ steps), confirming a characteristic curriculum time-scale.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value	Role
Architecture
$K_{\max}$	$8$	head-count cap
$K_{\mathrm{init}}$	$1$	initial active heads
$K_{\mathrm{min}}$	$1$	minimum active heads
$d_{k}$	$16$	per-head key/value dim.
$d_{\mathrm{model}}$	$128$	$=K_{\max}\cdot d_{k}$

Appendix

Table 4: Runtime-INCRT hyperparameters (same values across all

\tau_{z}\in\{1,2,5\}

s).

Robotic Control

Robotic control studies how machines use sensing and computation to produce reliable movement and interaction in changing environments.

See all 584 papers

Temporal Attention for Adaptive Control of Euler-Lagrange Systems with Unobservable Memory
May 7, 2026Giansalvo Cirrincione, Adriano FagioliniRL Control Robot Trajectory Tracking
Laboratoire LTI, Université de Picardie Jules Verne, Amiens, France · MIRPALab, Department of Engineering, University of Palermo, 90128 Palermo, Italy
Meta-reinforcement learning with minimum attention
May 22, 2025Shashank Gupta, Pilhwa LeeMeta-Reinforcement Learning Policy Learning
Department of EECS University of Michigan Ann Arbor, MI 48109 · Department of Mathematics Morgan State University Baltimore, MD 21251
TARC: Time-Adaptive Robotic Control
Oct 27, 2025Arnav Sukhija, Lenart Treven, Jin Cheng +3Robotic Control RL Control
ETH Zurich

May 7, 2026J/K move · Enter open · S save

Temporal Attention for Adaptive Control of Euler-Lagrange Systems with Unobservable Memory

Giansalvo Cirrincione, Adriano Fagiolini

Laboratoire LTI, Université de Picardie Jules Verne, Amiens, France · MIRPALab, Department of Engineering, University of Palermo, 90128 Palermo, Italy

RL Control Robot Trajectory Tracking Adaptive Control

Open paper AskPDF arXiv

Adaptive control of Euler-Lagrange systems is challenging when friction is governed by a finite-horizon internal state that is not directly observable from joint measurements. In this setting, the measured closed-loop state is no longer Markovian, and standard certainty-equivalence adaptive laws may lose their convergence guarantees. The paper proposes a meta-control architecture in which the gains of a computed-torque controller are generated by a self-attention block processing a short window of recent motion history. The number of attention heads is selected before policy training through a surrogate analysis of the autocovariance of the memory-state gradient along the temporal window. This surrogate is based on a temporal adaptation of an incremental rank-tracking framework previously developed by the authors. The selected head count is then fixed and used as an architectural hyperparameter in a reinforcement-learning stage, where the policy is trained under a shielded admissibility constraint. The approach is tested on a 2-DOF manipulator with nonlinear friction and variable payload. In the short and matched memory regimes, the single-layer attention-only meta-controller outperforms a deeper Transformer baseline, with tracking-error reductions of 12 and 19 percentage points, respectively. The reported effect sizes are large, with d approximately -1.1 and -2.1, and Mann-Whitney p < 0.05 in both cases. In the long memory regime, however, the advantage disappears. Four out of ten training runs show either divergence or payload-invariant policy collapse, revealing a weakness in the static Phase-1 head-count prescription. This motivates moving rank-tracking inside the reinforcement-learning loop, allowing attention heads to be pruned or grown at runtime instead of fixed before training.

Abstract

Robotic Control

Explore similar work

Temporal Attention for Adaptive Control of Euler-Lagrange Systems with Unobservable Memory

Meta-reinforcement learning with minimum attention

TARC: Time-Adaptive Robotic Control