Meta-reinforcement learning with minimum attention

Shashank Gupta; Pilhwa Lee

cs.LGMay 22, 2025

Meta-reinforcement learning with minimum attention

Authors: Shashank Gupta, Pilhwa Lee

Organizations: Department of EECS University of Michigan Ann Arbor, MI 48109 · Department of Mathematics Morgan State University Baltimore, MD 21251

Abstract

Minimum attention applies the least action principle in changes of control concerning state and time, first proposed by Brockett. The involved regularization is highly relevant in emulating biological control, such as motor learning. We apply minimum attention in reinforcement learning (RL) as part of the rewards and investigate its connection to meta-learning and stabilization. Specifically, model-based meta-learning with minimum attention is explored in high-dimensional nonlinear dynamics. Ensemble-based model learning and gradient-based meta-policy learning are alternately performed. Empirically, minimum attention improves fast adaptation in few shots and reduces variance from perturbations of the model and environment, compared to model-free and model-based RL baseline, and yields consistent gain when integrated into modern world models (DreamerV3, MAMBA). Furthermore, the minimum attention demonstrates an improvement in energy efficiency.

Figures & tables

Figure 1 : Meta-learning from model and environmental perturbations. (a) Half-Cheetah with varying body masses ( Lee et al., 2020 ) , (b) Adaptation to uphill and downhill, (c) One hind leg crippled ( Nagabandi et al., 2019 )

Panel of Figure 2 — Figure 2 : (a) Training of Half-Cheetah. There is a reduction of variance from minimum attention, (b) Comparison of learning curves among model-free RL (SAC, ( Haarnoja et al., 2018 ) ) and model-based RL (MB-MPO, ( Clavera et al., 2018 ) , the proposed MB-MPO + minimum attention, weighting factor $\alpha=0.05$ and $1$ ). The experiment is run with 10 episodes.

Panel of Figure 3 — Figure 3 : Meta-Training with different models, (a) Hopper, (b) Walker2D, (c) Humanoid. The model-based RL (MB-MPO) is employed with and without minimum attention ( $\alpha=1.0$ ) and compared with SAC.

	MB-MPO	Ours ( $\alpha=1.0$ )	Ours ( $\alpha=0.01$ )	Ours ( $\alpha=0.05$ )
Meta-training
Average total rewards	6692 $\pm$ 318	9721 $\pm$ 128	7545 $\pm$ 123	8385 $\pm$ 179
Average feedback normalized	783 $\pm$ 135	567 $\pm$ 53	590 $\pm$ 88	637 $\pm$ 56
Average feedforward normalized	5.16 $\pm$ 0.11	5.01 $\pm$ 0.07	5.02 $\pm$ 0.10	5.05 $\pm$ 0.07
Average energy normalized	4.54 $\pm$ 0.13	3.85 $\pm$ 0.05	3.94 $\pm$ 0.03	3.91 $\pm$ 0.02
Meta-testing

Table 1 : Half-Cheetah: Meta-training : total reward, feedback (

||\partial u/\partial x||^{2}

, Jacobian), feedforward (

||\partial u/\partial t||^{2}

, jerk), and energy (

||u||^{2}

) at final stage of iterations (200K). Comparison of the model-based RL (MB-MPO) without and with regularization of minimum attention. The bold typeset represents the best outperformance in the variation of

\alpha

. Meta-testing : the agent of Half-Cheetah, one leg crippled, at final stage of iterations (200K).

	Half-Cheetah	Hopper	Walker2D	Humanoid
MB-MPO training	6692 $\pm$ 318	2475 $\pm$ 3.81	2399 $\pm$ 694	575 $\pm$ 177
MB-MPO meta-testing	6356 $\pm$ 132	467 $\pm$ 100	523 $\pm$ 174	315 $\pm$ 67
MB-MPO + Min Attn training	9721 $\pm$ 128	2825 $\pm$ 1.25	3038 $\pm$ 148	498 $\pm$ 38
MB-MPO + Min Attn meta-testing	6822 $\pm$ 109	485 $\pm$ 61	1123 $\pm$ 152	580 $\pm$ 34

Table 2 : Comparison of the model-based RL (MB-MPO) with regularization of minimum attention in meta-training and testing. Total reward at the epoch of 200K. For Hopper, the thigh is paralyzed. For Walker2D, the right thigh is paralyzed.

HalfCheetah main results
Setting	Metric	MB-MPO	Ours (MA)
Meta-train	Reward $\uparrow$	6692 $\pm$ 318	9721 $\pm$ 128
	Feedback $\downarrow$	783 $\pm$ 135	567 $\pm$ 53
	Energy $\downarrow$	4.54 $\pm$ 0.13	3.85 $\pm$ 0.05
Meta-test	Reward $\uparrow$	6356 $\pm$ 132	6822 $\pm$ 109
	Feedback $\downarrow$	673 $\pm$ 75.6	625 $\pm$ 34.7

Table 3 : Summary of main results. Left: MA improves HalfCheetah reward while reducing feedback and energy in meta-training and crippled-back meta-testing. Right: MA improves MAMBA and DreamerV3 baselines.

Figure 4 : Control decomposition $u(x,t)=K(t)x+v(t)$ under crippled-back perturbation, comparing vanilla and MA policies through $\|K(t)\|_{F}$ and $\|v(t)\|_{2}$ .

Panel of Figure 5 — Figure 5 : Feedback sensitivity heatmaps for HalfCheetah. We visualize $\|\partial u/\partial x\|^{2}$ over three state-space projections: (a) forward velocity vs. torso height, (b) forward velocity vs. vertical velocity, and (c) forward velocity vs. torso angular velocity. Minimum attention reduces localized sensitivity peaks and yields a more distributed feedback profile in task-relevant regions.

Panel of Figure 6 — Figure 6 : Heatmaps of feedforward ( $||\partial u/\partial t||^{2}$ ) for HalfCheetah. Our method concentrates propulsive jerk and introduces a novel, proactive stabilization behaviour against torso roll at low speeds.

Panel of Figure 7 — Figure 7 : Half-Cheetah: Time profiles for the feedback, feedforward, and energy in training. The comparison of feedback ( $||\partial u/\partial x||^{2}$ , Jacobian), feedforward ( $||\partial u/\partial t||^{2}$ , Jerk), and energy ( $||u||^{2}$ ) between MB-MPO and MB-MPO with minimum attention ( $\alpha=1.0$ ).

Panel of Figure 8 — Figure 8 : Half-Cheetah: Meta-testing under OOD perturbations: (a) body mass change (Easy OOD tasks), (b) crippled legs (Medium OOD tasks), (c) uphill/downhill terrain (Difficult OOD tasks). We compare SAC, MB-MPO, and MB-MPO + minimum attention (MA, $\alpha=1.0$ ) over 10 evaluation steps. MA improves adaptation performance and reduces variance across perturbation settings.

Metric	Ours (Full MA)	Jacobian-only	Temporal-only
Reward $\uparrow$	9721 $\pm$ 128	6484 $\pm$ 148	6475 $\pm$ 135
Feedback $\downarrow$	567 $\pm$ 53	128 $\pm$ 24.5	736.2 $\pm$ 84.9
Feedforward $\downarrow$	5.01 $\pm$ 0.07	9.54 $\pm$ 0.35	4.94 $\pm$ 0.04
Energy $\downarrow$	3.85 $\pm$ 0.05	3.00 $\pm$ 0.03	3.09 $\pm$ 0.03

Table 4 : Ablation study on HalfCheetah (200K iterations,

\alpha=1.0

). We compare the full minimum attention (MA) regularizer against spatial-only (Jacobian) and temporal-only components. Bold indicates the best value per row.

Panel of Figure 9 — Figure 9 : Half-Cheetah: Dependency on the ensemble model size and imaginary rollouts. When the ensemble model size of $M$ increases to 20, the total reward also increases. The imaginary sample size $N$ provides the benefit of increased total reward to $N=128$ , but decreases when $N$ is turned to 256.

Figure 10 : Half Cheetah: Evolution of spatial sensitivity (feedback norm, $||\partial u/\partial x||^{2}$ ). Top Row (MB-MPO): Converges to a broadly reactive strategy. Bottom Row (Ours): Converges to a sparse, structured strategy, focusing sensitivity on a critical phase.

Figure 11 : Half Cheetah: Evolution of temporal sensitivity (feedforward norm/jerk, $||\partial u/\partial t||^{2}$ ). Top Row (MB-MPO): Progresses from chaotic flailing to a moderately jerky running gait. Bottom Row (Ours): The regularizer shapes the policy to be smoother overall, concentrating its action changes into a single, precise phase of the gait cycle.

	MB-MPO	MA ( $\alpha=1.0$ )	MA ( $\alpha=0.01$ )	MA ( $\alpha=0.05$ )
Early stage of epoch: 10K
Average total rewards	-1084 $\pm$ 73	672 $\pm$ 67	-427 $\pm$ 68	-579 $\pm$ 170
Average feedback normalized	38 $\pm$ 3.95	13 $\pm$ 0.35	36 $\pm$ 5.91	26 $\pm$ 1.91
Average feedforward normalized	7.11 $\pm$ 0.06	7.33 $\pm$ 0.04	5.99 $\pm$ 0.05	6.86 $\pm$ 0.06
Average energy normalized	3.63 $\pm$ 0.017	3.20 $\pm$ 0.016	3.27 $\pm$ 0.15	3.50 $\pm$ 0.010
Middle stage of epoch: 100K		.

Table 5 : Half-Cheetah: Meta-training on the maturation of meta-learning

	MB-MPO	Ours ( $\alpha=1.0$ )	Ours ( $\alpha=0.01$ )	Ours ( $\alpha=0.05$ )
Early stage of epoch: 10K
Average total rewards	-1245 $\pm$ 801	99 $\pm$ 180	90 $\pm$ 102	100 $\pm$ 139
Average feedback normalized	13.6 $\pm$ 9.23	21.5 $\pm$ 1.36	19.2 $\pm$ 6.8	15.3 $\pm$ 1.65
Average feedforward normalized	10.5 $\pm$ 0.07	8.12 $\pm$ 0.15	5.99 $\pm$ 0.14	8.07 $\pm$ 0.08
Average energy normalized	4.25 $\pm$ 0.057	3.30 $\pm$ 0.045	3.20 $\pm$ 0.021	3.50 $\pm$ 0.032
Middle stage of epoch: 100K

Table 6 : Half-Cheetah: Meta-testing on the maturation of meta-learning

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Training Steps	Mamba Baseline	+ MA ( $\alpha=0.05$ )	+ MA ( $\alpha=0.01$ )
30M (Final)	2453 $\pm$ 110	2553 $\pm$ 76	2498 $\pm$ 123
10M (Mid)	1985 $\pm$ 156	2127 $\pm$ 121	1965 $\pm$ 136
2.5M (Early)	467 $\pm$ 288	759 $\pm$ 231	415 $\pm$ 245

Appendix

Table S1 : Full evaluation of MAMBA on Humanoid-dir. Results report Average Total Rewards

\pm

standard deviation across 3 random seeds. We evaluate two regularization strengths (

\alpha

) against the unregularized baseline.

Task	Training Steps	DreamerV3 Baseline	+ MA ( $\alpha=0.05$ )
HalfCheetah-run	100K (Early)	257 $\pm$ 56	308 $\pm$ 59
	250K (Mid)	408 $\pm$ 86	394 $\pm$ 81
	500K (Final)	575 $\pm$ 123	625 $\pm$ 77
reacher-hard	100K (Early)	917 $\pm$ 35	906 $\pm$ 42
	250K (Mid)	874 $\pm$ 20	930 $\pm$ 17
	500K (Final)	942 $\pm$ 20	931 $\pm$ 17

Appendix

Table S2 : Full evaluation of DreamerV3 across three DeepMind Control tasks. Results report Average Total Rewards

\pm

standard deviation across 10 random seeds. Note that the benefit of minimum attention varies by task and training stage.

Panel of Figure 20 — Figure S1 : DreamerV3 training curves across three DeepMind Control tasks, comparing baseline against minimum attention ( $\alpha=0.05$ ). Shaded bands show $\pm 1$ standard deviation across 10 seeds.

Figure S2 : Feedback–feedforward decomposition during policy maturation for HalfCheetah-v3. We visualize

\|K(t)\|_{F}

and

\|v(t)\|_{2}

in the control law

u(x,t)=K(t)x+v(t)

for early and mature policies. The comparison shows how training changes both the feedback and feedforward components over rollout time. MA regularization produces a more constrained dispersion profile, suggesting that the learned controller matures toward a less variable control structure.

Figure S3 : Feedback–feedforward decomposition for HalfCheetah-v3 under mass perturbation, where the body mass is scaled by

1.5

. Rows compare vanilla and minimum-attention (MA) policies, and columns show the feedback intensity

\|K(t)\|_{F}

and feedforward bias

\|v(t)\|_{2}

. The trends are consistent with the crippled-back perturbation in the main paper: MA maintains a more regularized feedback–feedforward profile while preserving temporal adjustments needed to adapt to altered dynamics.

	MB-MPO	Ours ( $\alpha=1.0$ )
Early stage of epoch: 10K
Average total rewards	47 $\pm$ 2	232 $\pm$ 5.75
Average feedback normalized	168 $\pm$ 5.26	290 $\pm$ 1.82
Average feedforward normalized	0.45 $\pm$ 0.026	0.11 $\pm$ 0.003
Average energy normalized	0.26 $\pm$ 0.0027	0.55 $\pm$ 0.022
Middle stage of epoch: 100K

Appendix

Table S3 : Hopper: Meta-training

	MB-MPO	Ours ( $\alpha=1.0$ )
Early stage of epoch: 10K
Average total rewards	234 $\pm$ 3.39	278 $\pm$ 3.15
Average feedback normalized	99.7 $\pm$ 2.15	137 $\pm$ 5.25
Average feedforward normalized	0.11 $\pm$ 0.003	0.03 $\pm$ 0.001
Average energy normalized	1.52 $\pm$ 0.058	1.14 $\pm$ 0.015
Middle stage of epoch: 100K

Appendix

Table S4 : Hopper: Meta-testing

Figure S4 : Hopper: Heatmap of feedback. The feedback ( $||\partial u/\partial x||^{2}$ ) in the projection of the states to (a) Torso-angle versus Torso-height, (b) Torso-angle versus Torso X-velocity, and (c) Torso-angular velocity versus Torso-angle.

Figure S5 : Hopper: Heatmap of feedforward. The feedforward ( $||\partial u/\partial t||^{2}$ ) in the projection of the states to (a) Torso-angle versus Torso-height, (b) Torso-angle versus Torso X-velocity, and (c) Torso-angular velocity versus Torso-angle.

Panel of Figure 27 — Figure S6 : Hopper: Time profiles for the feedback, feedforward, and energy in meta-training. The comparison of feedback ( $||\partial u/\partial x||^{2}$ , Jacobian), feedforward ( $||\partial u/\partial t||^{2}$ , Jerk), and energy ( $||u||^{2}$ ) between MB-MPO and MB-MPO with minimum attention ( $\alpha=1.0$ ).

Panel of Figure 28 — Figure S7 : Hopper. Left Column: Phase-space analysis of control effort for Hopper’s balancing motion. Top Row (MB-MPO): A "panic" strategy with an extreme feedback hotspot ( $>$ 2000) for backward leans. Bottom Row (Ours): A smooth, proactive strategy with distributed, low-gain feedback, eliminating the need for high-gain emergency corrections. Right Column: Control effort during a Hopper trajectory. Top (MB-MPO): Shows massive, periodic feedback spikes at the bottom of each hop. Bottom (Ours): Exhibits a more regular gait with dramatically lower and smoother control effort.

	MB-MPO	Ours ( $\alpha=1.0$ )
Early stage of epoch: 10K
Average total rewards	325 $\pm$ 226	615 $\pm$ 45.76
Average feedback normalized	3 $\pm$ 0.33	60.6 $\pm$ 1.74
Average feedforward normalized	7.76 $\pm$ 0.84	0.35 $\pm$ 0.07
Average energy normalized	9.69 $\pm$ 2.48	7.98 $\pm$ 0.84
Middle stage of epoch: 100K

Appendix

Table S5 : Walker2D: Meta-training

	MB-MPO	Ours ( $\alpha=1.0$ )
Early stage of epoch: 10K
Average total rewards	51 $\pm$ 29	45 $\pm$ 7.91
Average feedback normalized	4.62 $\pm$ 0.21	3.1 $\pm$ 0.7
Average feedforward normalized	6.47 $\pm$ 0.25	3.05 $\pm$ 0.19
Average energy normalized	5.00 $\pm$ 0.42	3.94 $\pm$ 0.29
Middle stage of epoch: 100K

Appendix

Table S6 : Meta-testing of Walker2D

Figure S8 : Walker2D: Heatmap of feedback. The feedback ( $||\partial u/\partial x||^{2}$ ) in the projection of the states to (a) Torso-angle versus Torso-height, (b) Torso-angle versus Torse X-velocity, and (c) Torso-angular velocity versus Torso-angle.

Figure S9 : Walker2D: Heatmap of feedforward. The feedforward ( $||\partial u/\partial t||^{2}$ ) in the projection of the states to (a) Torso-angle versus Torso-height, (b) Torso-angle versus Torse X-velocity, and (c) Torso-angular velocity versus Torso-angle.

Panel of Figure 33 — Figure S10 : Walker2D: Time profiles for the feedback, feedforward, and energy in meta-training. The comparison of feedback ( $||\partial u/\partial x||^{2}$ , Jacobian), feedforward ( $||\partial u/\partial t||^{2}$ , Jerk), and energy ( $||u||^{2}$ ) between MB-MPO and MB-MPO with minimum attention ( $\alpha=1.0$ ).

	MB-MPO	Ours ( $\alpha=1.0$ )
Early stage of epoch: 10K
Average total rewards	297 $\pm$ 10	245 $\pm$ 10
Average feedback normalized	39.9 $\pm$ 1.7	2.48 $\pm$ 0.13
Average feedforward normalized	0.13 $\pm$ 0.006	0.877 $\pm$ 0.06
Average energy normalized	1.08 $\pm$ 0.036	0.52 $\pm$ 0.016
Middle stage of epoch: 100K

Appendix

Table S7 : Humanoid: Meta-training

	MB-MPO	Ours ( $\alpha=1.0$ )
Early stage of epoch: 10K
Average total rewards	314 $\pm$ 19	359 $\pm$ 27
Average feedback normalized	59.1 $\pm$ 5.24	15.9 $\pm$ 1.06
Average feedforward normalized	0.285 $\pm$ 0.003	0.136 $\pm$ 0.002
Average energy normalized	1.01 $\pm$ 0.082	0.053 $\pm$ 0.0081
Middle stage of epoch: 100K

Appendix

Table S8 : Meta-testing of Humanoid

Figure S11 : Humanoid: Heatmap of feedback. The feedback ( $||\partial u/\partial x||^{2}$ ) in the projection of the states to (a) Torso-height versus Torso-forward-velocity, and (b) Right-hip angle versus left-hip angle.

Figure S12 : Humanoid: Heatmap of feedforward. The feedforward ( $||\partial u/\partial t||^{2}$ ) in the projection of the states to a) Torso-height versus Torso-forward-velocity, and b) Right-hip angle versus left-hip angle.

Panel of Figure 38 — Figure S13 : Humanoid: Time profiles for the feedback, feedforward, and energy in meta-training. The comparison of feedback ( $||\partial u/\partial x||^{2}$ , Jacobian), feedforward ( $||\partial u/\partial t||^{2}$ , jerk), and energy ( $||u||^{2}$ ) between MB-MPO and MB-MPO with minimum attention ( $\alpha=1.0$ ).

Explore similar work

Temporal Attention for Adaptive Control of Euler-Lagrange Systems with Unobservable Memory
May 7, 2026Giansalvo Cirrincione, Adriano FagioliniAdaptive Control Lagrangian Methods
Laboratoire LTI, Université de Picardie Jules Verne, Amiens, France · MIRPALab, Department of Engineering, University of Palermo, 90128 Palermo, Italy
On Training in Imagination
May 7, 2026Nadav Timor, Ravid Shwartz-Ziv, Micah Goldblum +2Model-Based Reinforcement Learning Rollout
Weizmann Institute of Science · New York University · Columbia University
MInTRL: Off-policy Intervention can boost On-policy RL
Sep 14, 2026Mingyu Chen, Yefan Tao, Gerald Friedland +2Off-Policy Learning
Amazon Web Services · Boston University

May 7, 2026, cs.LGJ/K move · Enter open · S save

Temporal Attention for Adaptive Control of Euler-Lagrange Systems with Unobservable Memory

Giansalvo Cirrincione, Adriano Fagiolini

Laboratoire LTI, Université de Picardie Jules Verne, Amiens, France · MIRPALab, Department of Engineering, University of Palermo, 90128 Palermo, Italy

Adaptive Control Lagrangian Methods Torque Friction Temporal Attention

Open paper AskPDF arXiv

Adaptive control of Euler-Lagrange systems is challenging when friction is governed by a finite-horizon internal state that is not directly observable from joint measurements. In this setting, the measured closed-loop state is no longer Markovian, and standard certainty-equivalence adaptive laws may lose their convergence guarantees. The paper proposes a meta-control architecture in which the gains of a computed-torque controller are generated by a self-attention block processing a short window of recent motion history. The number of attention heads is selected before policy training through a surrogate analysis of the autocovariance of the memory-state gradient along the temporal window. This surrogate is based on a temporal adaptation of an incremental rank-tracking framework previously developed by the authors. The selected head count is then fixed and used as an architectural hyperparameter in a reinforcement-learning stage, where the policy is trained under a shielded admissibility constraint. The approach is tested on a 2-DOF manipulator with nonlinear friction and variable payload. In the short and matched memory regimes, the single-layer attention-only meta-controller outperforms a deeper Transformer baseline, with tracking-error reductions of 12 and 19 percentage points, respectively. The reported effect sizes are large, with d approximately -1.1 and -2.1, and Mann-Whitney p < 0.05 in both cases. In the long memory regime, however, the advantage disappears. Four out of ten training runs show either divergence or payload-invariant policy collapse, revealing a weakness in the static Phase-1 head-count prescription. This motivates moving rank-tracking inside the reinforcement-learning loop, allowing attention heads to be pruned or grown at runtime instead of fixed before training.

Abstract

Explore similar work

Temporal Attention for Adaptive Control of Euler-Lagrange Systems with Unobservable Memory

On Training in Imagination

MInTRL: Off-policy Intervention can boost On-policy RL