Refined Gradient-Based Temperature Optimization for the Replica-Exchange Monte-Carlo Method

Tatsuya Miyata; Shunta Arai; Satoshi Takabe

physics.comp-phJan 20, 2026

Refined Gradient-Based Temperature Optimization for the Replica-Exchange Monte-Carlo Method

Authors: Tatsuya Miyata, Shunta Arai, Satoshi Takabe

Organizations: Institute of Science Tokyo, Ookayama, Tokyo 152-8550, Japan

Abstract

The replica-exchange Monte-Carlo (RXMC) method is a powerful Markov-chain Monte-Carlo algorithm for sampling from multi-modal distributions, which are challenging for conventional methods. The sampling efficiency of the RXMC method depends highly on the selection of the temperatures, and finding optimal temperatures remains a challenge. In this study, we propose a refined online temperature selection method by extending the gradient-based optimization framework proposed previously. Building upon the existing temperature update approach, we introduce a reparameterization technique to strictly enforce physical constraints, such as the monotonic ordering of inverse temperatures, which were not explicitly addressed in the original formulation. The proposed method defines the variance of acceptance rates between adjacent replicas as a loss function, estimates its gradient using differential information from the sampling process, and optimizes the temperatures via gradient descent. We demonstrate the effectiveness of our method through experiments on benchmark spin systems, including the two-dimensional ferromagnetic Ising model, the two-dimensional ferromagnetic XY model, and the three-dimensional Edwards-Anderson model. Our results show that the method successfully achieves uniform acceptance rates and reduces round-trip times across the temperature space. Furthermore, our proposed method offers a significant advantage over recently proposed policy gradient method that require careful hyperparameter tuning, while simultaneously preventing the constraint violations that destabilize optimization.

Figures & tables

Panel of Figure 1 — Figure 1: Evolution of acceptance rates between adjacent replicas during the optimization process for the 2D Ising model ( $L=20$ ). (a) Results using the policy gradient method ( $T=3000$ ). (b) Results using the proposed method ( $T=300$ ).

Panel of Figure 2 — Figure 2: Learning curves for the 2D Ising model ( $L=20$ ). (a) Policy gradient method: The negative reward equivalent to the loss function is plotted against the epoch ( $T=3000$ ). (b) Proposed method: The loss function is plotted against the epoch ( $T=300$ ).

Figure 3: The inverse temperatures at different optimization steps in the 2D Ising model ( $L=20$ ). Starting from the initial geometric progression (epoch 0), the results are shown every 25 steps. The dashed line indicates the inverse temperature where the specific heat is maximal.

Figure 4: Distribution of round-trip times for the 2D Ising model ( $L=20$ ) under five different temperature selection strategies: from above, inverse linear, geometric progression, the strategy resulting from the policy gradient method, the round-trip time minimization strategy, the uniform acceptance rates strategy optimized using the proposed method. The y-axis is presented on a logarithmic scale. Each density is estimated from an evaluation run of $10^{7}$ MCS.

Figure 5: Box plot of the round-trip time distributions for the 2D Ising model ( $L=20$ ) under five strategies. Outliers defined as data points falling beyond 1.5 times the IQR from the box are omitted for clarity.

Figure 6: Evolution of acceptance rates between adjacent replicas during the optimization process for the 2D XY model ( $L=20$ ). Although the optimization was performed for 300 epochs, the results are shown only up to epoch 200 because convergence was already achieved by that point.

Figure 7: The inverse temperatures at different optimization steps in the 2D XY model ( $L=20$ ). Starting from the initial geometric progression (epoch 0), the results are shown every 25 steps. The dashed line indicates the inverse temperature where the specific heat peaks.

Figure 8: Box plot of the round-trip time distributions for the 2D XY model ( $L=20$ ) under three strategies: from above, inverse linear, geometric progression, the uniform acceptance rates strategy optimized using the proposed method, with outliers removed for clarity.

Figure 9: Evolution of acceptance rates between adjacent replicas during the optimization process for the 3D EA model ( $L=8$ ).

Figure 10: The inverse temperatures at different optimization steps in the 3D EA model ( $L=8$ ). Starting from the initial geometric progression (epoch 0), the results are shown every 25 steps. The dashed line indicates the inverse temperature where the specific heat is maximal.

Figure 11: Box plot of the round-trip time distributions for the 3D EA model ( $L=8$ ) under three strategies, with outliers removed for clarity. Result for the round-trip time minimization strategy is not shown because it did not converge within the prescribed number of MCS.

Figure 12: The acceptance rates between adjacent replicas when an inverse temperatures optimized for a single interaction instance is applied to other interaction instances. The index $i$ corresponds to the subscript $i$ of the acceptance rate $A_{i,i+1}$ . The line represents the instance-averaged acceptance rate, and the bands represent the standard deviation. The red dots represent the acceptance rates when applied to the interaction instance used in the optimization.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 13 — Figure 13: Unconstraint optimization results using gradient descent on $\nabla_{\bm{\beta}}f$ in the 2D Ising model ( $L=20$ ). (a) Evolution of acceptance rates between adjacent replicas over 300 epochs. The rate for the pair between replicas 13 and 14 drops to zero, causing stagnation for over 200 epochs. (b) Evolution of inverse temperatures. Starting from a geometric progression (epoch 0), snapshots are plotted every 25 steps. The dashed line indicates the inverse temperature where the specific heat is maximal. The $+$ symbol represents the 13th and 14th replicas.

Panel of Figure 14 — Figure 14: Optimization results using StochasticAD for the 2D Ising model ( $L=10$ ). (a) Evolution of acceptance rates between adjacent replicas over 300 epochs. (b) Evolution of inverse temperatures. Starting from a geometric progression (epoch 0), snapshots are plotted every 25 steps. The dashed line indicates the inverse temperature of maximal specific heat.

Parallel Tempering

Parallel tempering uses interacting replicas at different temperatures to improve sampling and search across multimodal landscapes.

See all 14 papers

Mar 18, 2026

Generative Replica-Exchange: A Flow-based Framework for Accelerating Replica Exchange Simulations

Replica exchange (REX) is one of the most widely used enhanced sampling methodologies, yet its efficiency is limited by the requirement for a large number of intermediate temperature replicas. Here we present Generative Replica Exchange (GREX), which integrates deep generative models into the REX framework to eliminate this temperature ladder. Drawing inspiration from reservoir replica exchange (res-REX), GREX utilizes trained normalizing flows to generate high-temperature configurations on demand and map them directly to the target distribution using the potential energy as a constraint, without requiring target-temperature training data. This approach reduces production simulations to a single replica at the target temperature while maintaining thermodynamic rigor through Metropolis exchange acceptance. We validate GREX on three benchmark systems of increasing complexity, highlighting its superior efficiency and practical applicability for molecular simulations.

Shengjie Huang, Sijie Yang, Jianqiao Yi +6

State Key Laboratory of Bioactive Molecules and Druggability Assessment, International Cooperative Laboratory of Traditional Chinese Medicine Modernization and Innovative Drug Discovery of Chinese Ministry of Education (MOE), School of Pharmacy, Jinan University, #855 Xingye Avenue, Guangzhou, 510632, China · Department of Biochemistry and Molecular Pharmacology, NYU Grossman School of Medicine, New York, NY 10016, USA · Howard Hughes Medical Institute, NYU Grossman School of Medicine, New York, NY 10016, USA +1

Molecular Dynamics Simulation Parallel Tempering Markov Chain Monte Carlo

Ask PDF

Jul 8, 2026

Gradient-free Riemannian Langevin Sampler

We address the problem of efficiently sampling multimodal probability distributions, where standard Markov Chain Monte Carlo methods often suffer from poor mixing and mode trapping. To mitigate these issues, we propose Gradient-free Riemannian Langevin Sampler (GRiLS), a novel proposal that improves exploration without requiring gradient evaluations of the target density. Our approach introduces a Riemannian metric which reshapes the local geometry in order to facilitate transitions across modes. The resulting gradient-free MCMC algorithm is particularly suitable for complex, computationally expensive targets where derivatives are unavailable or impractical. The GRiLS proposal requires knowing the mean and covariance of the target density, which we estimate using an ensemble of interacting particles. Empirical results on multimodal benchmarks demonstrate that GRiLS achieves improved mixing compared to existing gradient-based and gradient-free MCMC approaches.

Ricardo Baptista, Olivier Zahm

Department of Statistical Sciences, University of Toronto, Canada · UGA, Inria, CNRS, Grenoble INP*, LJK, 38000 Grenoble, France

Markov Chain Monte Carlo

Ask PDF

Jun 29, 2026

Diffusion-warm sampling of the XY model enables fast thermalization at scale

We introduce a novel technique for scalable sampling of spin-system states with continuous symmetries using diffusion models. By applying our approach to the XY model, a fundamental continuous-spin model in condensed matter physics, we show that our technique addresses the shortfalls of the Markov chain Monte Carlo (MCMC) in generalization to varying system sizes. More specifically, we show that training a temperature-conditioned diffusion model on smaller-size XY model lattices enables the generation of accurate samples in larger lattice sizes. By tracking physically important observables of the model, such as spin correlations, our experiments demonstrate that diffusion sampling followed by a few MCMC steps reduces the thermalization time by an order of magnitude relative to the standard MCMC with random initialization. Our study provides valuable insight as to how generative models can be used to study continuous-state condensed matter systems at scale.

Sehmimul Hoque, Roger Melko, Pooya Ronagh

Institute for Quantum Computing, University of Waterloo, Waterloo, Ontario N2L 3G1, Canada · Department of Physics and Astronomy, University of Waterloo, Waterloo, Ontario N2L 3G1, Canada · Perimeter Institute for Theoretical Physics, Waterloo, Ontario N2L 2Y5, Canada +1

Markov Chain Monte Carlo Diffusion Sampling

Ask PDF

Abstract

Parallel Tempering

Explore similar work

Generative Replica-Exchange: A Flow-based Framework for Accelerating Replica Exchange Simulations

Gradient-free Riemannian Langevin Sampler

Diffusion-warm sampling of the XY model enables fast thermalization at scale