Meta-Learning and Meta-Reinforcement Learning -- Tracing the Path towards DeepMind's Adaptive Agent

Björn Hoppmann; Christoph Scholz

cs.AIFeb 23, 2026

Meta-Learning and Meta-Reinforcement Learning -- Tracing the Path towards DeepMind's Adaptive Agent

Authors: Björn Hoppmann, Christoph Scholz

Abstract

Humans are highly effective at utilizing prior knowledge to adapt to novel tasks, a capability that standard machine learning models struggle to replicate due to their reliance on task-specific training. Meta-learning overcomes this limitation by allowing models to acquire transferable knowledge from various tasks, enabling rapid adaptation to new challenges with minimal data. This survey provides a rigorous, task-based formalization of meta-learning and meta-reinforcement learning and uses that paradigm to chronicle the landmark algorithms that paved the way for DeepMind's Adaptive Agent, consolidating the essential concepts needed to understand the Adaptive Agent and other generalist approaches.

Figures & tables

Figure 1: Meta-learning of 2-way 1-shot animal classification tasks. The current meta-knowledge $\varphi$ is the prior for one-shot learning of each particular classification task. During meta-training, the meta-optimizer receives all $N$ query set losses of the adapted models to update meta-knowledge $\varphi$ . Meta-validation evaluates the training progress on new classification problems every $l$ meta-epochs, while meta-testing on unseen classifications takes place after meta-training.

Panel of Figure 2 — Figure 2: General Meta-Training. In each iteration a new task $T_{i}$ is sampled from the family $p(T)$ . The meta-variable $\varphi$ is the prior for individual $K$ shot fine-tuning of each task. The resulting parameters $\theta_{i}^{\prime}$ of each task are used to update $\varphi$ .

Panel of Figure 3 — Figure 3: General Meta-Testing Paradigm. For each $T_{j}$ sampled from the set of test tasks the parameters $\theta_{j}(\varphi)$ are fine-tuned in $K$ shots, before the resulting $\theta_{j}^{\prime}$ get evaluated on the task’s test set via the task-specific loss $\mathcal{L}_{j}$ to yield the performance.

Figure 4: Meta-Reinforcement learning to race on tracks with varying weather conditions. Starting from the meta-knowledge $\varphi$ , $K$ episodes of fine-tuning on the particular racing track yield performance measures. The meta-optimizer uses these measures to update meta-knowledge $\varphi$ . Meta-validation evaluates the training progress on unseen racing tracks every $l$ meta-epochs. After meta-training, the meta-policy adapts to test tracks to evaluate the quality of prior $\varphi$ .

Panel of Figure 5 — Figure 5: The MAML meta-training scheme.

Component	Advantages	Drawbacks	Algos
Gradient-based	• Easy to implement • Usable for meta-learning and meta-RL • Many open-source repos available	• Weak generalization and OOD performance • Only PG methods for meta-RL	MAML, FO-MAML
Context-based	• Dynamically adjust to tasks without model training • Benefitial in sparse-reward tasks • Implicit goal of base Bayes-optimal behaviour	• Highly dependent on the quality of context	CAVIA, DREAM, MAESN, Meta-Cure, PEARL, RL 2
RNN-based	• Sequence processing, context-based by design • Simple memory-based architecture	• Poor asymptotic performance • Sensitive to architecture, context length, and RL algorithm choice	RL 2 , VariBAD
Bayesian Inference	• Explicit uncertainty quantification • Tailored for Bayes-optimal behaviour	• Complex implementation • Often exhibit unstable learning	MAESN, Meta-Cure, PEARL, VariBAD, MELTS
Model-based RL on Task-Level	• Decisions can be based on the dynamics model’s predictions; better long-horizon planing	• Higher architectural and computational complexity	VariBAD, ADA
Transformer-based	• Powerful contextualization • Meta-learner per design • Very high sample efficiency and adaptation speed • Particularly high OOD performance	• Require vast amounts of data and computational resources for (meta-)training • Perform worse on tasks with high task uncertainty	TrMRL, ADA, AMAGO, GATO, ECET

Table 1: Advantages and disadvantages of the different components introduced along with the landmarks in Section 3 . The last column lists algorithms containing the respective component.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: General Multi-Task-Learning Paradigm:

Panel of Figure 7 — Figure 7: The PEARL meta-training scheme. In each MDP $M_{I}$ , PEARL collects $K$ episodes of experience and stores them in replay buffer $\beta_{i}$ . It samples the context variable $Z$ at the beginning of each episode. After inner learning, minibatches are sampled from the replay buffer to update meta-parameters $\varphi,\theta_{\pi}$ and $\theta_{Q}$ .

Meta-Learning

Meta-learning trains models across tasks so they can adapt quickly to new problems with few examples, limited feedback, or no parameter updates.

See all 164 papers

METTA: Memory-Enhanced Test-Time Adaptation for Meta-Reinforcement Learning
Oct 22, 2026Ali Najar, Hosein Hasani, Amir Kooshan Fattah +2Meta-Reinforcement Learning Test-Time RL
Department of Computer Engineering Sharif University of Technology
Knowledge Reutilization in Meta-Reinforcement Learning
Jun 16, 2026Yuan Meng, Bo Wang, Juan de los Rios Ruiz +4Cross-Embodiment Robot Learning Meta-Reinforcement Learning
School of Computation, Information and Technology, Technical University of Munich, Munich, Germany · State Key Laboratory for Novel Software Technology, Nanjing University, Suzhou, China · Department of Computer Science and Technology, Tsinghua University, Beijing, China
Harness Learning Enables Generalizable Test-Time Adaptation
Sep 28, 2026Alvin Zhang, Xuecheng Liu, Zixuan Wang +6Test-Time Adaptation Meta-Learning

Oct 22, 2026J/K move · Enter open · S save

METTA: Memory-Enhanced Test-Time Adaptation for Meta-Reinforcement Learning

Ali Najar, Hosein Hasani, Amir Kooshan Fattah, Yasamin Rajabi+1

Department of Computer Engineering Sharif University of Technology

Meta-Reinforcement Learning Test-Time RL Memory-Augmented RL

Open paper AskPDF arXiv

Meta-reinforcement learning aims to train agents that adapt quickly to unseen tasks from limited interaction, but this requires representing task evidence over long interaction histories. Classical recurrent agents compress experience into hidden states, while transformer-based agents retrieve from past trajectories at quadratic memory and computation cost. In this work, we propose Memory-Enhanced Test-Time Adaptation (METTA) for meta-reinforcement learning. It decomposes the fast adaptation phase into two temporal scales: a causal neural memory performs fine-grained within-episode adaptation with linear sequence complexity, while compact episode-level summaries support long-range cross-episode task inference without attending over the full interaction history. Moreover, equipping agents with neural memory provides sufficient expressive capacity and test-time training capability, enabling effective adaptation to unseen held-out tasks and task families at meta-test time. On unseen held-out Meta-World ML10/45 test environments, METTA achieves a success rate over

50\%

while maintaining a low training budget and short wall-clock time. On long-context sparse-reward PointMaze tasks, it outperforms transformer-based baselines by exhibiting more structured exploration behavior. These results show that METTA provides an efficient path toward more adaptive and scalable long-context meta-RL.

Abstract

Meta-Learning

Explore similar work

METTA: Memory-Enhanced Test-Time Adaptation for Meta-Reinforcement Learning

Knowledge Reutilization in Meta-Reinforcement Learning

Harness Learning Enables Generalizable Test-Time Adaptation