Thermodynamics of Reinforcement Learning Curricula

Jacob Adamczyk; Juan Sebastian Rojas; Rahul V. Kulkarni

cs.LGMar 12, 2026

Thermodynamics of Reinforcement Learning Curricula

Authors: Jacob Adamczyk, Juan Sebastian Rojas, Rahul V. Kulkarni

Organizations: Department of Physics, University of Massachusetts Boston, Boston, MA 02125, USA · Department of Mechanical & Industrial Engineering, University of Toronto, Toronto, ON, CA

Abstract

Connections between statistical mechanics and machine learning have repeatedly proven fruitful, providing insight into optimization, generalization, and representation learning. In this work, we follow this tradition by leveraging results from non-equilibrium thermodynamics to formalize curriculum learning in reinforcement learning (RL). In particular, we propose a geometric framework for RL by interpreting reward parameters as coordinates on a task manifold. We show that, by minimizing the excess thermodynamic work, optimal curricula correspond to geodesics in this task space. As an application of this framework, we provide an algorithm, "MEW" (Minimum Excess Work), to derive a principled schedule for temperature annealing in maximum-entropy RL.

Figures & tables

Figure 1: Visualization of the 7 $\times$ 7 Grid World task (inset, left), regret under the linear and geodesic paths (left), and the resulting thermodynamic manifold with the optimal geodesic protocol, which navigates around the phase transition at $\lambda_{1}=\lambda_{2}$ (right). Further details can be found in Appendix B .

Figure 2: ( Humanoid-v5 ) Proposed method (MEW), SAC’s automatic temperature adjustment based on minimum entropy constraint, and two constant temperatures. Base algorithm is ASAC ( Adamczyk et al., 2025 ) .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Comparison of different “thermodynamic speeds” (speed of traversal along the path from $\alpha=0.2\to 0$ ). Across three orders of magnitude, we obtain similar qualitative performance. The constant temperature, i.e. zero speed, $\alpha=0.2$ is included for comparison. The log-log plot of temperatures indicates that a non-trivial protocol is learned. (If variance were constant throughout, a power law, i.e., a straight line, would result.)

Figure 4: Comparison of performance for various “recency thresholds” (cf. $N$ in Algorithm 1 ). For an optimal policy, episodes are of length $1,000$ ; the recency thresholds used to calculate the friction therefore correspond to $1/2$ , $5$ or $50$ episodes. The results shown here and in Figure 3 demonstrate that the proposed method is robust to the required hyperparameters.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Effectiveness of Curriculum Learning Depends on Reward Sparsity and Competing Optima
Oct 22, 2026John J. Vastola, Ann Huang, Satpreet H. Singh +2Curriculum RL Reinforcement Learning
Department of Neurobiology, Harvard Medical School · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Department of Psychology and Center for Brain Science, Harvard University
Bridging Control, Inference, Transport, and Thermodynamics: From Theory to Applications in Learning
Sep 14, 2026Emmy Blumenthal, Nikolas Claussen, Benjamin Eysenbach +4
Joseph Henry Laboratories of Physics, Princeton University, Princeton, New Jersey 08544, USA · Princeton Center for Theoretical Science, Princeton University, Princeton, New Jersey 08544, USA · Department of Computer Science, Princeton University, Princeton, New Jersey 08544, USA
Active Curriculum Refinement for Reinforcement Learning
Aug 26, 2026Zhenya Liu, Yuxin ChenCurriculum RL Curriculum Learning
The University of Chicago, Chicago IL, USA.

Oct 22, 2026J/K move · Enter open · S save

Effectiveness of Curriculum Learning Depends on Reward Sparsity and Competing Optima

John J. Vastola, Ann Huang, Satpreet H. Singh, Samuel J. Gershman+1

Department of Neurobiology, Harvard Medical School · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Department of Psychology and Center for Brain Science, Harvard University

Curriculum RL Reinforcement Learning Sparse-Reward RL Policy Gradient Methods Reward Design for RL

Open paper AskPDF arXiv

While animals and people tend to learn a task more quickly and reliably when they first train on simpler versions of it, curriculum learning's effectiveness in artificial settings varies, and appears substantially greater in reinforcement learning than in supervised learning. In this work, we construct and analyze a minimal model of policy learning to better understand why. We consider an open-loop, episodic target interception task whose parameters---especially how close the agent must be to the target to receive an informative reward signal---can be chosen so that the agent either receives informative rewards frequently ('dense' rewards) or rarely ('sparse' rewards). We mathematically and numerically analyze the dynamics of REINFORCE-based policy learning in both cases. In the dense reward case, curricula have only marginal benefits; in the sparse case, curricula can be required for learning to occur at all. We find that this is especially true when the agent has the conflicting goals of minimizing effort and accumulating high reward, since this can produce distinct reward landscape optima which compete with one another. Our characterization of useful curricula in this setting indicates that they address two issues: they (i) make rewards less sparse to speed up learning, and (ii) shape the reward landscape to steer agents away from bad local optima.

Abstract

Reinforcement Learning

Explore similar work

Effectiveness of Curriculum Learning Depends on Reward Sparsity and Competing Optima

Bridging Control, Inference, Transport, and Thermodynamics: From Theory to Applications in Learning

Active Curriculum Refinement for Reinforcement Learning