Reinforcement learning remains sample inefficient, particularly when agents must adapt over time to related but distinct tasks, as in continual learning settings. Efficiency can be improved by transferring knowledge from a pretrained teacher policy to guide exploration in new but related tasks. However, if the new task sufficiently differs from the teacher's training task, the transferred guidance may be sub-optimal and bias exploration toward low-reward behaviors. We propose an energy-based transfer learning method that uses out-of-distribution detection to selectively issue guidance, enabling the teacher to intervene only in states within its training distribution. We theoretically link energy scores to state-visitation patterns under on-policy reinforcement learning. Empirically, our method improves sample efficiency and performance across both single-task and multi-task settings.
Figures & tables
Figure 1: Overview of energy-based transfer learning . As the student interacts with the environment, the teacher: 1) checks if each state is in-distribution or out-of-distribution by comparing the state’s energy score to a pre-defined energy threshold ; 2) If the state exceeds the energy threshold , then it is considered in-distribution for the teacher and an expert action is suggested to the student.
Figure 2: Environments used for empirical experiments. See Section 5 for detailed descriptions.
Figure 3: Minigrid (10 seeds). (a) Empirical energy score distributions with respect to the teacher policy. The source task (blue) shows the teacher’s training distribution. The target task (orange + purple), measured during transfer, is bimodal: one mode overlaps with the source (shared sub-task, in-distribution), while the other does not (non-shared sub-task, out-of-distribution). (b) Guidance quality: fraction of transferable target states correctly identified as ID and advised (orange), and fraction of non-transferable target states incorrectly identified as ID and advised (purple), with the decay schedule disabled. An ideal gating signal yields an orange bar substantially higher than the purple bar. (c) Relative transfer (% vs. scratch) across q and decay schedules (50%: guidance reaches 0 at mid-training). q=−1 denotes always advising (AA baseline). (d) Learning curves: EBTL vs. baselines. Guidance ends at mid-training with q=0.5 .
Figure 4: Lang-Minigrid . (10 seeds). Guidance ends at mid-training with q=0.5 . Task variation is specified through language instructions: in Alternating-Goal, the instruction indicates the target corner (e.g., go to the top left corner ), while in Unlocked-to-Locked, it specifies sequential subgoals (e.g., pick up the key, then go to the goal ).
Figure 5: (10 seeds). Learning curves with (solid) and without (dashed) energy regularization in Minigrid environments.
Figure 6: OOD score distributions across detection methods in Minigrid. Each panel shows the score distributions of the source task (blue) and target task (orange: shared sub-task, purple: non-shared sub-task) under the teacher policy. A good OOD detector should separate blue and purple while keeping blue and orange overlapping. (a) MaxP, (b) Entropy, and (c) ODIN, requiring OOD samples for hyperparameter calibration. (d) (Left) The energy score without regularization (no OOD samples) already matches ODIN, and (Right) with regularization (OOD samples used during teacher training) achieves substantially sharper separation.
Optimality
EBTL (0.3)
EBTL (0.5)
EBTL (0.7)
AA
JSRL
KSRL
Fine-tuning
96%
41.2±4.9
46.1±4.5
40.3±4.7
6.1±5.6
−3.2±10.3
17.5±6.7
−32.2±12.9
70%
26.8±4.6
31.2±5.7
27.0±5.4
17.0±4.3
24.8±5.4
14.2±6.1
−104.3±8.6
45%
16.7±4.6
14.3±4.3
18.9±9.2
13.9±6.2
15.9±3.7
21.5±5.7
−80.6±3.0
15%
9.2±5.9
12.8±4.7
13.7±4.9
12.5±7.1
5.0±6.4
22.9±4.3
−110.5±4.4
Table 1: Robustness to suboptimal teachers . Relative transfer improvement (%) over training from scratch (mean ± 95% CI, 10 seeds) for Alternating Goal. Guidance ends at mid-training.
Figure 7: Overcooked (5 seeds). (Cols 1 & 3) Energy scores: source (blue) vs. target (orange + purple). Target is bimodal: one mode overlaps source (ID), one does not (OOD). (Cols 2 & 4) Learning curves: EBTL vs. baselines ( q=0.5 for Recipe Shift, q=0.2 for Recipe + Layout Shift).
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Setup
Recipe (2 → 3)
Recipe (1 → 3)
Recipe + Layout (2 → 2)
Simple Room
19,008,000
9,004,800
12,000,000
Ring Room
2,400,000
10,003,200
18,000,000
Appendix
Table 2: Selected teacher checkpoints (in environment steps) for each Overcooked setup and source-target configuration.
Setup
Recipe (2 → 3)
Recipe (1 → 3)
Recipe + Layout (2 → 2)
Simple Room
20M
20M
12M
Ring Room
35M
35M
20M
Appendix
Table 3: Transfer horizons (in millions of environment steps) used for student training in each Overcooked setup and configuration. Each experiment is run with 5 random seeds.
Hyperparameter
Value
Learning rate
0.0005
Discount factor ( γ )
0.9
GAE lambda ( λ )
0.8
Policy clip parameter
0.2
Value function clip parameter
10.0
Value loss coefficient
0.5
Appendix
Table 4: Hyperparameters used for all GridWorld experiments.
Table 12
Figure 8: 10 seeds. JSRL hyperparameter tuning in MiniGrid.
Figure 9: 10 seeds. KSRL hyperparameter tuning in MiniGrid.
Figure 10: Results (3 seeds). In each environment, the left column shows the empirical density of the teacher’s state energy ϕ . Blue traces the source rollouts used to train the teacher. Orange and green are target rollouts during transfer and typically form a bimodal pattern: one component overlaps the source shared task (in distribution), while the other occupies a separate region (non-shared task, out of distribution). q is set to 0.3 in both settings.
Figure 11: Actor-Critic architectures used in our experiments. (a) MiniGrid. (b) Overcooked.
Metric
ANOVA p-value
Kruskal–Wallis p-value
Jensen–Shannon
0.1138
0.1592
Kullback–Leibler
0.2457
0.1799
Total Variation
0.1728
0.2322
Hellinger Distance
0.1247
0.1592
Appendix
Table 7: Statistical test results (p-values) for divergence between False OOD and True OOD energy distributions across different (min,mout) settings.
Table 8: Choice of ID states for setting the quantile q . “Post-train” derives q from evaluation rollouts of the converged teacher (no exploration); “In-train” derives q from exploration-time states during teacher training. Values are student transfer performance relative to training from scratch (mean ± 95% CI)
Figure 12: 10 seeds. Effect of replacing rollout-based OOD batches with randomly sampled OOD batches. q=−1 advise in all states (AA Baseline).
Teacher
Without energy loss
With energy loss
Alternating Room
100,000
60,000
Unlocked-to-Locked
260,000
220,000
Appendix
Table 9: Training steps to convergence (mean over 10 seeds; lower is better).
Figure 13: 10 seeds. Comparison of five decay schedules. q=−1 advise in all states (AA Baseline).
Figure 14: Heatmaps of average energy quantiles under the teacher for Alternating Goal. Left: source (goal in Room 1). Right: target (goal in Room 3). Higher quantiles indicate greater teacher familiarity.
Figure 15: 10 seeds. Advice schedules where advice ends at (a) 25%, (b) 50%, and (c) 75% of training in the Unlocked-to-Locked environment. q=−1 advise in all states (AA Baseline).
Figure 16: Learning curves across q (10 seeds). EBTL for q∈{0.2,…,0.7} vs. baselines; near-identical across the range. Guidance ends at mid-training.
Figure 17: IAA reproduction and decay-schedule comparison. 10 seeds. Top: exponential decay (as in the original IAA paper). Bottom: linear decay (the EBTL default). Under exponential decay we reproduce the IAA result, but standard action advising without filtering also exceeds the baseline, suggesting the schedule itself contributes substantially to the gain. Under linear decay, only EBTL remains clearly above the baseline. EBTL is the best performing method under both schedules.
Figure 18: IAA with extended burn-in periods. 10 seeds. Linear Decay. Learning curves for IAA across burn-in start points (25% and 50% of training), advice decay endpoints (50%, 75%, and 100% of training), and thresholds ( ϵ∈{0.1,0.15} ).
Figure 19: IAA value-gap separation in the Unlocked-to-Locked setting , shown across ϵ∈{0.05,0.1,0.15,0.3,0.45,0.9} (one per row); left column freezes the convolutional backbone, right column fine-tunes the full network. Each panel tracks three groups of states over the course of the student’s learning: ID (in-distribution, collected during teacher training), Transferable OOD (states encountered during the student’s learning where the teacher should issue advice), and Non-transferable OOD (states where the teacher should not ). A reliable signal should place the ID and Transferable-OOD curves above the Non-transferable-OOD curve, indicating that good advice is issued more often than bad advice.
Figure 20: IAA value-gap separation in the Alternating-Goal setting , across ϵ∈{0.05,0.1,0.15,0.3,0.45,0.9} (one per row); left column freezes the convolutional backbone, right column fine-tunes the full network. Curves and the expected ordering are as in Figure 19 : a reliable signal keeps the ID and Transferable-OOD curves above the Non-transferable-OOD curve.
Reinforcement learning has achieved remarkable success in learning complex control policies, yet its applicability remains limited due to sample inefficiency and poor generalization across tasks. In this work, we propose RepMT-SAC, a framework for multi-task RL that enables efficient knowledge sharing and robust transfer to new tasks. RepMT-SAC uses spectral MDP decomposition to capture transferable dynamics, structuring the value function into a task-agnostic core with a minimal task-specific adjustment. This design allows for strong zero-shot performance on in-distribution tasks and rapid few-shot adaptation to out-of-distribution tasks. We evaluate RepMT-SAC on quadcopter trajectory-following tasks across in-distribution and out-of-distribution contexts, demonstrating that it outperforms baselines by up to 30%.
Aryan Naveen, Haitong Ma, Haldun Balim +1
Massachusetts Institute of Technology · Harvard School of Engineering and Applied Sciences
Transfer-oriented reinforcement learning requires evaluating algorithms along dimensions that go beyond standard sample efficiency. We focus on two dimensions: practical efficiency, which asks whether conclusions about algorithm suitability change under wall-clock rather than interaction-based budgets, and robustness under dynamics mismatch, which asks how different learning paradigms respond to variability in the training distribution induced by domain randomization. We provide two insights to reinforcement-learning practitioners. First, comparing the sample efficiency of different algorithms is often an insufficient criterion in transfer-oriented settings. The wall-clock time required to train a decent policy is an important consideration for practitioners, and we find that the sample-inefficient PPO algorithm can produce a performant policy faster than relatively more sample-efficient algorithms such as SAC and TD-MPC2, validating the common understanding of massively parallel training paradigms. Second, domain randomization can help different kinds of algorithms learn robust policies. In particular, although PPO, SAC, and TD-MPC2 represent different RL paradigms - on-policy, off-policy, and model-based learning and planning, respectively - we find that domain randomization affects all three algorithms in a similar way. To the best of our knowledge, this is the first controlled comparison of the effect of domain-randomization coverage on PPO, SAC, and TD-MPC2 under the same transfer protocol. Taken together, these two insights highlight the importance of evaluating RL algorithms not only by sample efficiency, but also by practical considerations such as training time and the algorithms' ability to produce usable policies.
Hany Hamed, Abhishek Naik, Colin Bellinger +1
University of Alberta · 3Alberta Machine Intelligence Institute (Amii) · 7Generative Bionics +4
Meta-Reinforcement Learning addresses the critical limitations of conventional Reinforcement Learning in multi-task and non-stationary environments by enabling fast policy adaptation and improved generalization. We introduce a novel Meta-RL framework that integrates a bi-level optimization scheme with a hybrid actor-critic architecture specially designed to enhance sample efficiency and inter-task adaptability. To improve knowledge transfer, we meta-learn a shared state feature extractor jointly optimized across actor and critic networks, providing efficient representation learning and limiting overfitting to individual tasks or dominant profiles. Additionally, we propose a parameter-sharing mechanism between the outer- and inner-loop actor networks, to reduce redundant learning and accelerate adaptation during task revisitation. The approach is validated on a real-world Building Energy Management Systems dataset covering nearly a decade of temporal and structural variability, for which we propose a task preparation method to promote generalization. Experiments demonstrate effective task adaptation and better performance compared to conventional RL and Meta-RL methods.
Théo Zangato, Aomar Osmani, Pegah Alizadeh
Université Sorbonne Paris Nord, France · INSA Rouen, France · Université Lumière Lyon 2, Université Claude Bernard Lyon 1, ERIC, 69007, Lyon, France