An Energy-Based Perspective on Machine Learning
Organizations: Harvard University
Abstract
Energy-based models (EBMs) define a distribution through a scalar energy, , and are usually presented as one model family among many. This tutorial has a broader goal: beyond introducing EBMs, it uses them as a single lens that connects the different aspects of machine learning. Any EBM is specified by three choices, the energy, which determines what the model represents; the inference procedure, which samples from or optimizes the energy; and the learning method, which fits the energy to data. We first develop these components, including sampling, variational inference, and learning by maximum likelihood, noise contrastive estimation, pseudolikelihood, score matching, and adversarial training, and show that the learning methods share one contrastive structure. We then show that discriminative models, generative models such as GANs, normalizing flows, diffusion and autoregressive models, representation learning methods such as contrastive learning, word2vec, BERT, denoising auto-encoders, and diffusion representations, reinforcement learning with EBM policies and world models, inverse reinforcement learning, and planning, as well as inference-time computation with verifiers, are all EBMs that differ in these three choices, mostly in how they handle the partition function and sampling. This view explains existing methods, organizes them into a common framework, and suggests new algorithms by replacing one component while keeping the others, with composition of energies as a principled way to combine learned models at inference time.
Figures & tables
| Method | Negative sampler | Gap from |
|---|---|---|
| Maximum likelihood | none (exact) | |
| Contrastive divergence | : MCMC steps from data | short chains |
| Persistent CD | from the previous iteration’s samples | chains lag behind |
| Pseudolikelihood | : one Gibbs step from data | one variable at a time |
| Composite likelihood | block Gibbs step from data | one block at a time |
| Score matching | one small Langevin step from data | local, step size |
| Model | Discriminator loss | Entropy | Discriminator at the solution |
|---|---|---|---|
| Original GAN | binary classification | no | constant, |
| Wasserstein GAN | , Lipschitz | no | critic for a distance |
| Entropy-regularized GAN | maximum likelihood | yes | energy, |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Method | Energy | Inference | Learning | Partition function | Sec. |
|---|---|---|---|---|---|
| Logistic regression | linear in | energy minimization | MLE | exact by enumeration | 3.1 |
| Chain CRF | unary and pairwise terms | Viterbi | MLE; max-margin | exact by the forward algorithm | 3.1 |
| SPEN | deep, non-decomposable over | projected gradient descent on relaxed | structured SVM; end-to-end | intractable; no probabilistic model | 3.2 |
| GAN | discriminator as potential | learned generator | adversarial | potential need not be normalized; generator samples the EBM only with entropy regularization | 4.1 |
| Normalizing flow | by change of variables | exact, one pass | exact MLE, no negative phase | for every | 4.2 |
| Diffusion / flow matching | family of EBMs , one per noise level | reverse SDE/ODE; annealed Langevin | denoising score matching; velocity regression | drops out of the score | 4.3 |
Energy-Based Models
Energy-based models assign scalar energies to configurations, defining probabilities and guiding generation, inference, prediction, and associative memory.
See all 99 papers