Nonlocal Transition Kernel for Efficient Learning of Restricted Boltzmann Machines

Kaiji Sekimoto; Muneki Yasuda

stat.MLAug 18, 2026

Nonlocal Transition Kernel for Efficient Learning of Restricted Boltzmann Machines

Authors: Kaiji Sekimoto, Muneki Yasuda

Organizations: Graduate School of Science and Engineering, Yamagata University, Japan

Abstract

Learning restricted Boltzmann machines (RBMs) is computationally challenging because it requires expectations whose exact evaluation is generally intractable. The expectations are typically evaluated using a sampling approximation based on blocked Gibbs sampling (BGS), which is a local Markov chain Monte Carlo transition kernel. However, the locality of BGS can lead to poor sampling quality when the RBM has high energy barriers, thereby degrading learning performance. Deep tempering (DT), which performs parallel tempering over a sequence of learnable RBMs including the training RBM, alleviates this locality issue. However, DT algorithmically requires multiple steps to move through the RBM sequence to achieve a nonlocal transition. In this paper, we propose a transition kernel defined over the RBM sequence used in DT. The proposed kernel has a round-trip structure over the sequence, enabling nonlocal moves within a single transition while leaving the RBM sequence invariant. Numerical experiments show that the proposed kernel performs nonlocal transitions more frequently and achieves higher sampling quality with fewer transitions than BGS and DT. We further verify that learning based on the proposed kernel is more stable and mitigates the training failures observed with BGS- and DT-based learning.

Figures & tables

Figure 1: Schematic illustrations of an RBM: (a) a two-layer architecture with a visible layer at the bottom and a hidden layer at the top, and (b) its simplified representation.

Figure 2: Log-likelihood, the MAEs, and the second-largest eigenvalue

\lambda_{2}

versus the training epoch for the GenRBM dataset. The results represent averages over 10 experiments.

Figure 3: Illustration of the RBM sequence with

L=2

used in DT. The bottommost RBM corresponds to the training RBM, and the other RBMs are used for sampling.

Figure 4: Illustration of the sampling procedure for the proposed transition kernel on an RBM sequence with

L=2

. The upward pass sequentially applies the swap operations from lower to upper RBMs, followed by a one-step BGS transition at the topmost RBM, and the downward pass applies the swap operations in the reverse order of the upward pass.

Setting	GenRBM	Islands	Pentagon	Iris	Wine	Seeds
Data dimension $n$	10	25	25	20	65	35
Data size $N_{\mathrm{d}}$	100	100	100	150	178	210
Number of hidden variables $m$	100	100	100	100	100	100
Sample size $N_{\mathrm{s}}$	100	100	100	150	100	100
Learning rate $\varepsilon$	0.01	0.001	0.001	0.002	0.001	0.001

Table 1: Experimental settings for training the RBMs.

Figure 5: Visualization of low-rank synthetic datasets with data dimension $n=25$ : (a) samples and (b) a PCA projection for the islands dataset with $a=0.4$ , and (c) samples and (d) a PCA projection for the pentagon dataset with $\kappa=6$ .

Figure 6: Log-likelihood and MAE versus the training epoch for the GenRBM dataset. The results represent averages over 10 experiments.

Figure 7: Log-likelihoods versus the training epoch for the (a) islands, (b) pentagon, (c) iris, (d) wine, and (e) seeds datasets. The results represent averages over 10 experiments.

Figure 8: Transition trajectories of BGS, DT, and the proposed kernel for the six datasets.

Figure 9: TV distances in Eq. ( 25 ) for the (a) GenRBM, (b) islands, (c) pentagon, and (d) iris datasets. The solid curves and shaded regions represent the medians and the 10th–90th percentile ranges, respectively.

Explore similar work

Training Energy-Based Models with Non-MCMC Samplers and Efficient Temperature Estimation
Dec 2, 2025Kentaro Kubo, Hayato GotoEnergy-Based Models
Corporate Laboratory, Toshiba Corporation, Kawasaki, Kanagawa 212-8582, Japan · RIKEN Center for Quantum Computing (RQC), Wako, Saitama 351-0198, Japan
Thermodynamic Regulation of Finite-Time Gibbs Training in Energy-Based Models: A Restricted Boltzmann Machine Study
Mar 3, 2026Görkem Can SüleymanoğluRestricted Boltzmann Machines Energy-Based Models
Owner and Software Developer, Kuanka Publishing LLC, Turkey. · Ph.D. Student in Economics, Selçuk University, Institute of Social Sciences, Turkey.
Comparison of D-Wave Quantum Annealing and Gibbs Monte Carlo for Sampling from a Probability Distribution of a Restricted Boltzmann Machine
Aug 13, 2025Abdelmoula El-Yazizi, Yaroslav KoshkaMarkov Chain Monte Carlo Quantum Annealing
Department of Electrical and Computer Engineering, Mississippi State University, Starkville, MS 39762, USA

Dec 2, 2025J/K move · Enter open · S save

Training Energy-Based Models with Non-MCMC Samplers and Efficient Temperature Estimation

Kentaro Kubo, Hayato Goto

Corporate Laboratory, Toshiba Corporation, Kawasaki, Kanagawa 212-8582, Japan · RIKEN Center for Quantum Computing (RQC), Wako, Saitama 351-0198, Japan

Energy-Based Models

Open paper AskPDF arXiv

Efficient sampling from Boltzmann distributions over discrete variables is a fundamental operation in a wide range of applications. While fast non-MCMC samplers have recently emerged as promising alternatives to conventional MCMC methods, their practical use for probabilistic learning remains hindered by the difficulty of estimating the effective temperature of the generated samples. In this work, we begin by introducing Langevin simulated bifurcation (LSB), a Boltzmann sampler that enables fast and parallel sampling with accuracy comparable to sequential MCMC methods. To address the challenge of unknown effective temperature, we propose conditional expectation matching (CEM), an efficient estimation method applicable to energy-based models (EBMs) with exploitable conditional independence structures. Building on these components, we further develop a learning framework, termed sampler adaptive learning (SAL), which adaptively adjusts the model temperature to match that of the distribution induced by fast non-MCMC sampling. We demonstrate the effectiveness of LSB, CEM, and SAL on semi-restricted Boltzmann machines (SRBMs), a class of EBMs that are difficult to train using conventional approaches. LSB achieves orders-of-magnitude acceleration over Gibbs sampling while maintaining comparable or higher accuracy, and CEM enables accurate temperature estimation of the resulting distribution with negligible computational overhead. As a consequence, SAL enables efficient training of SRBMs and outperforms conventional Boltzmann machine learning methods on synthetic spin-glass datasets. In addition, the trained models achieve strong performance across multiple tasks. These results establish LSB as a fast and accurate Boltzmann sampler and provide key insights that enable practical applications of fast non-MCMC sampling methods via efficient temperature estimation with CEM.

Abstract

Explore similar work

Training Energy-Based Models with Non-MCMC Samplers and Efficient Temperature Estimation

Thermodynamic Regulation of Finite-Time Gibbs Training in Energy-Based Models: A Restricted Boltzmann Machine Study

Comparison of D-Wave Quantum Annealing and Gibbs Monte Carlo for Sampling from a Probability Distribution of a Restricted Boltzmann Machine