Refined Gradient-Based Temperature Optimization for the Replica-Exchange Monte-Carlo Method

Tatsuya Miyata; Shunta Arai; Satoshi Takabe

physics.comp-phJan 20, 2026

Refined Gradient-Based Temperature Optimization for the Replica-Exchange Monte-Carlo Method

Authors: Tatsuya Miyata, Shunta Arai, Satoshi Takabe

Organizations: Institute of Science Tokyo, Ookayama, Tokyo 152-8550, Japan

Abstract

The replica-exchange Monte-Carlo (RXMC) method is a powerful Markov-chain Monte-Carlo algorithm for sampling from multi-modal distributions, which are challenging for conventional methods. The sampling efficiency of the RXMC method depends highly on the selection of the temperatures, and finding optimal temperatures remains a challenge. In this study, we propose a refined online temperature selection method by extending the gradient-based optimization framework proposed previously. Building upon the existing temperature update approach, we introduce a reparameterization technique to strictly enforce physical constraints, such as the monotonic ordering of inverse temperatures, which were not explicitly addressed in the original formulation. The proposed method defines the variance of acceptance rates between adjacent replicas as a loss function, estimates its gradient using differential information from the sampling process, and optimizes the temperatures via gradient descent. We demonstrate the effectiveness of our method through experiments on benchmark spin systems, including the two-dimensional ferromagnetic Ising model, the two-dimensional ferromagnetic XY model, and the three-dimensional Edwards-Anderson model. Our results show that the method successfully achieves uniform acceptance rates and reduces round-trip times across the temperature space. Furthermore, our proposed method offers a significant advantage over recently proposed policy gradient method that require careful hyperparameter tuning, while simultaneously preventing the constraint violations that destabilize optimization.

Figures & tables

Panel of Figure 1 — Figure 1: Evolution of acceptance rates between adjacent replicas during the optimization process for the 2D Ising model ( $L=20$ ). (a) Results using the policy gradient method ( $T=3000$ ). (b) Results using the proposed method ( $T=300$ ).

Panel of Figure 2 — Figure 2: Learning curves for the 2D Ising model ( $L=20$ ). (a) Policy gradient method: The negative reward equivalent to the loss function is plotted against the epoch ( $T=3000$ ). (b) Proposed method: The loss function is plotted against the epoch ( $T=300$ ).

Figure 3: The inverse temperatures at different optimization steps in the 2D Ising model ( $L=20$ ). Starting from the initial geometric progression (epoch 0), the results are shown every 25 steps. The dashed line indicates the inverse temperature where the specific heat is maximal.

Figure 4: Distribution of round-trip times for the 2D Ising model ( $L=20$ ) under five different temperature selection strategies: from above, inverse linear, geometric progression, the strategy resulting from the policy gradient method, the round-trip time minimization strategy, the uniform acceptance rates strategy optimized using the proposed method. The y-axis is presented on a logarithmic scale. Each density is estimated from an evaluation run of $10^{7}$ MCS.

Figure 5: Box plot of the round-trip time distributions for the 2D Ising model ( $L=20$ ) under five strategies. Outliers defined as data points falling beyond 1.5 times the IQR from the box are omitted for clarity.

Figure 6: Evolution of acceptance rates between adjacent replicas during the optimization process for the 2D XY model ( $L=20$ ). Although the optimization was performed for 300 epochs, the results are shown only up to epoch 200 because convergence was already achieved by that point.

Figure 7: The inverse temperatures at different optimization steps in the 2D XY model ( $L=20$ ). Starting from the initial geometric progression (epoch 0), the results are shown every 25 steps. The dashed line indicates the inverse temperature where the specific heat peaks.

Figure 8: Box plot of the round-trip time distributions for the 2D XY model ( $L=20$ ) under three strategies: from above, inverse linear, geometric progression, the uniform acceptance rates strategy optimized using the proposed method, with outliers removed for clarity.

Figure 9: Evolution of acceptance rates between adjacent replicas during the optimization process for the 3D EA model ( $L=8$ ).

Figure 10: The inverse temperatures at different optimization steps in the 3D EA model ( $L=8$ ). Starting from the initial geometric progression (epoch 0), the results are shown every 25 steps. The dashed line indicates the inverse temperature where the specific heat is maximal.

Figure 11: Box plot of the round-trip time distributions for the 3D EA model ( $L=8$ ) under three strategies, with outliers removed for clarity. Result for the round-trip time minimization strategy is not shown because it did not converge within the prescribed number of MCS.

Figure 12: The acceptance rates between adjacent replicas when an inverse temperatures optimized for a single interaction instance is applied to other interaction instances. The index $i$ corresponds to the subscript $i$ of the acceptance rate $A_{i,i+1}$ . The line represents the instance-averaged acceptance rate, and the bands represent the standard deviation. The red dots represent the acceptance rates when applied to the interaction instance used in the optimization.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 13 — Figure 13: Unconstraint optimization results using gradient descent on $\nabla_{\bm{\beta}}f$ in the 2D Ising model ( $L=20$ ). (a) Evolution of acceptance rates between adjacent replicas over 300 epochs. The rate for the pair between replicas 13 and 14 drops to zero, causing stagnation for over 200 epochs. (b) Evolution of inverse temperatures. Starting from a geometric progression (epoch 0), snapshots are plotted every 25 steps. The dashed line indicates the inverse temperature where the specific heat is maximal. The $+$ symbol represents the 13th and 14th replicas.

Panel of Figure 14 — Figure 14: Optimization results using StochasticAD for the 2D Ising model ( $L=10$ ). (a) Evolution of acceptance rates between adjacent replicas over 300 epochs. (b) Evolution of inverse temperatures. Starting from a geometric progression (epoch 0), snapshots are plotted every 25 steps. The dashed line indicates the inverse temperature of maximal specific heat.

Parallel Tempering

Parallel tempering uses interacting replicas at different temperatures to improve sampling and search across multimodal landscapes.

See all 14 papers

Generative Replica-Exchange: A Flow-based Framework for Accelerating Replica Exchange Simulations
Mar 18, 2026Shengjie Huang, Sijie Yang, Jianqiao Yi +6Molecular Dynamics Simulation Parallel Tempering
State Key Laboratory of Bioactive Molecules and Druggability Assessment, International Cooperative Laboratory of Traditional Chinese Medicine Modernization and Innovative Drug Discovery of Chinese Ministry of Education (MOE), School of Pharmacy, Jinan University, #855 Xingye Avenue, Guangzhou, 510632, China · Department of Biochemistry and Molecular Pharmacology, NYU Grossman School of Medicine, New York, NY 10016, USA · Howard Hughes Medical Institute, NYU Grossman School of Medicine, New York, NY 10016, USA +1
Gradient-free Riemannian Langevin Sampler
Jul 8, 2026Ricardo Baptista, Olivier ZahmMarkov Chain Monte Carlo
Department of Statistical Sciences, University of Toronto, Canada · UGA, Inria, CNRS, Grenoble INP*, LJK, 38000 Grenoble, France
Diffusion-warm sampling of the XY model enables fast thermalization at scale
Jun 29, 2026Sehmimul Hoque, Roger Melko, Pooya RonaghMarkov Chain Monte Carlo Diffusion Sampling
Institute for Quantum Computing, University of Waterloo, Waterloo, Ontario N2L 3G1, Canada · Department of Physics and Astronomy, University of Waterloo, Waterloo, Ontario N2L 3G1, Canada · Perimeter Institute for Theoretical Physics, Waterloo, Ontario N2L 2Y5, Canada +1

Mar 18, 2026J/K move · Enter open · S save

Generative Replica-Exchange: A Flow-based Framework for Accelerating Replica Exchange Simulations

Shengjie Huang, Sijie Yang, Jianqiao Yi, Rui Zheng+5

State Key Laboratory of Bioactive Molecules and Druggability Assessment, International Cooperative Laboratory of Traditional Chinese Medicine Modernization and Innovative Drug Discovery of Chinese Ministry of Education (MOE), School of Pharmacy, Jinan University, #855 Xingye Avenue, Guangzhou, 510632, China · Department of Biochemistry and Molecular Pharmacology, NYU Grossman School of Medicine, New York, NY 10016, USA · Howard Hughes Medical Institute, NYU Grossman School of Medicine, New York, NY 10016, USA+1

Molecular Dynamics Simulation Parallel Tempering Markov Chain Monte Carlo

Open paper AskPDF arXiv

Replica exchange (REX) is one of the most widely used enhanced sampling methodologies, yet its efficiency is limited by the requirement for a large number of intermediate temperature replicas. Here we present Generative Replica Exchange (GREX), which integrates deep generative models into the REX framework to eliminate this temperature ladder. Drawing inspiration from reservoir replica exchange (res-REX), GREX utilizes trained normalizing flows to generate high-temperature configurations on demand and map them directly to the target distribution using the potential energy as a constraint, without requiring target-temperature training data. This approach reduces production simulations to a single replica at the target temperature while maintaining thermodynamic rigor through Metropolis exchange acceptance. We validate GREX on three benchmark systems of increasing complexity, highlighting its superior efficiency and practical applicability for molecular simulations.

Abstract

Parallel Tempering

Explore similar work

Generative Replica-Exchange: A Flow-based Framework for Accelerating Replica Exchange Simulations

Gradient-free Riemannian Langevin Sampler

Diffusion-warm sampling of the XY model enables fast thermalization at scale