Learning Through Energy Refinement and Manifold Projection: A Cooperative EBM-AE Framework

Ryad Zemouri

cs.LGSep 12, 2026

Learning Through Energy Refinement and Manifold Projection: A Cooperative EBM-AE Framework

Authors: Ryad Zemouri

Organizations: Hydro-Québec 75, boulevard René-Lévesque Ouest Montréal, Québec, Canada

Abstract

Energy-Based Models (EBMs) provide a flexible framework for generative modeling by learning an energy landscape that assigns low energy values to realistic samples and higher energies to unlikely observations. Despite their theoretical appeal, training EBMs remains challenging due to the computational cost of Langevin sampling and the difficulty of efficiently exploring the learned data manifold. In this work, we propose a cooperative Energy-Based Model and Autoencoder (EBM-AE) framework that combines energy-based refinement with manifold projection. The proposed approach jointly trains an EBM with a denoising autoencoder and introduces an iterative EBM $\rightarrow$ AE $\rightarrow$ EBM sampling procedure in which Langevin dynamics and autoencoder projection alternately refine generated samples. Within this framework, the autoencoder acts as a manifold projection operator that regularizes sampling trajectories, while the EBM performs energy-based refinement toward low-energy regions of the learned distribution. Extensive experiments conducted on the MNIST dataset demonstrate that joint EBM-AE training substantially improves generation quality compared with a conventional autoencoder. Beyond unconditional generation, we evaluate the proposed framework on image inpainting tasks involving structured and random masks. The results show that manifold projection provides the majority of the reconstruction capability, whereas the final energy-based refinement becomes increasingly beneficial as the reconstruction problem becomes more challenging. Taken together, the results indicate that combining manifold projection and energy minimization provides an effective and interpretable framework for generation, reconstruction, and out-of-distribution detection, while offering new insights into the complementary roles of energy-based modeling and representation learning.

Figures & tables

x^{+}\sim p_{D}

Algorithm 1 Cooperative EBM-AE Training

Figure 1: Ablation study conducted over 13 independent experimental runs. The jointly trained models substantially outperform the Vanilla AE, while the complete EBM $\rightarrow$ AE $\rightarrow$ EBM pipeline provides the best overall trade-off between generation quality, sample validity and diversity.

Method	FID $\downarrow$	Entropy $\uparrow$	Unknown (%) $\downarrow$	Confidence $\uparrow$
EBM $\rightarrow$ AE $\rightarrow$ EBM	2834.0	2.2855	2.14	0.9967
AE Joint	3463.0	2.2622	2.34	0.9958
Vanilla AE	18985.0	2.0284	19.25	0.9542

Table 1: Comparison of generation strategies.

Panel of Figure 2 — Figure 2: Qualitative comparison of generated samples and class-wise generation results for the three evaluated methods. The EBM $\rightarrow$ AE $\rightarrow$ EBM framework produces the most realistic samples while preserving a balanced coverage of digit classes.

Selection Criterion	FID	Entropy	Unknown (%)
MC1 (Best Buffer FID)	7743 – 10386	2.11 – 2.25	2.3 – 4.7
MC2 (Best Generated FID)	2834 – 4589	2.26 – 2.34	2.1 – 4.8
MC3 (Best Energy Gap)	8790 – 20593	1.97 – 2.23	2.2 – 11.2
MC4 (Last Checkpoint)	5068 – 16161	2.02 – 2.24	1.9 – 19.0

Table 2: Influence of checkpoint selection.

Panel of Figure 3 — Figure 3: Influence of checkpoint selection. Top row: energy histogram comparison. Bottom row: class-wise generated samples. (a) Best replay-buffer FID (MC1), (b) Best generated-data FID (MC2), (c) Best energy-gap checkpoint (MC3), (d) Final checkpoint (MC4).

Panel of Figure 4 — Figure 4: Relationship between generation quality and the different evaluation metrics. (a) FID versus Unknown Rate. (b) FID versus class-distribution entropy. (c) FID versus generated-sample energy.

Mask Type	Ratio	EBM $\rightarrow$ AE $\rightarrow$ EBM	AE Joint	Vanilla AE
Center	0.20	710	718	1288
Center	0.30	2759	3021	7458
Center	0.50	22230	22679	29746
Random	0.20	760	1048	489
Random	0.30	1940	2441	1700
Random	0.50	6360	6961	9127

Table 3: Inpainting FID performance for different masking configurations.

Mask Type	Ratio	Masked	EBM $\rightarrow$ AE $\rightarrow$ EBM	AE Joint	Vanilla AE
Center	0.20	3.12	0.68	0.76	0.91
Center	0.30	6.85	0.91	1.16	3.41
Center	0.50	11.00	1.43	2.18	7.34
Random	0.20	6.86	1.53	1.84	0.97
Random	0.30	8.58	1.99	2.37	1.46
Random	0.50	13.15	2.63	3.15	3.03

Table 4: Mean generated-sample energy for different masking configurations.

Panel of Figure 5 — Figure 5: Image inpainting results obtained for two levels of reconstruction difficulty. The first row corresponds to a center mask covering 40% of the image, while the second row corresponds to a more challenging scenario where 45% of the image is removed. For each configuration, the left panel presents representative reconstructions obtained with the different methods, whereas the right panel shows the corresponding energy distributions. For moderate masking ratios (40%), all methods are able to reconstruct plausible digits, although the proposed EBM $\rightarrow$ AE $\rightarrow$ EBM framework produces reconstructions that remain closer to the low-energy regions of the learned manifold. As the masking ratio increases to 45%, the reconstruction problem becomes significantly more difficult and the benefit of the proposed energy-guided refinement becomes more apparent. The EBM $\rightarrow$ AE $\rightarrow$ EBM framework consistently generates lower-energy and more coherent reconstructions than both the jointly trained AE and the Vanilla AE baseline.

Experiment	AUROC (%)	AUPR (%)	FPR95 (%)
Best Case 1	96.80	97.81	15.75
Best Case 2	97.40	98.28	9.76
Best Case 3	98.35	98.93	1.21

Table 5: Best OOD detection scores obtained during training.

Experiment	Best Buffer Energy	Best AUROC (%)
test1	-3.7206	98.35
test2	-3.4898	97.91
test3	-3.3692	96.47
test6	-3.2351	97.98
test7	-3.4062	97.48
test9	-3.5439	98.27

Table 6: Relationship between buffer energy and OOD detection performance.

Generative Modeling

Generative modeling learns data distributions to create new samples, from images and text to molecules, simulations, and structured records.

See all 493 papers

May 1, 2026

Learning Multimodal Energy-Based Model with Multimodal Variational Auto-Encoder via MCMC Revision

Energy-based models (EBMs) are a flexible class of deep generative models and are well-suited to capture complex dependencies in multimodal data. However, learning multimodal EBM by maximum likelihood requires Markov Chain Monte Carlo (MCMC) sampling in the joint data space, where noise-initialized Langevin dynamics often mixes poorly and fails to discover coherent inter-modal relationships. Multimodal VAEs have made progress in capturing such inter-modal dependencies by introducing a shared latent generator and a joint inference model. However, both the shared latent generator and joint inference model are parameterized as unimodal Gaussian (or Laplace), which severely limits their ability to approximate the complex structure induced by multimodal data. In this work, we study the learning problem of the multimodal EBM, shared latent generator, and joint inference model. We present a learning framework that effectively interweaves their MLE updates with corresponding MCMC refinements in both the data and latent spaces. Specifically, the generator is learned to produce coherent multimodal samples that serve as strong initial states for EBM sampling, while the inference model is learned to provide informative latent initializations for generator posterior sampling. Together, these two models serve as complementary models that enable effective EBM sampling and learning, yielding realistic and coherent multimodal EBM samples. Extensive experiments demonstrate superior performance for multimodal synthesis quality and coherence compared to various baselines. We conduct various analyses and ablation studies to validate the effectiveness and scalability of the proposed multimodal framework.

Jiali Cui, Zhiqiang Lao, Heather Yu

Basking Ridge, New Jersey, USA Futurewei Technologies Inc · Basking Ridge, New Jersey, USA2026 Futurewei Technologies Inc

Variational Autoencoders Markov Chain Monte Carlo Energy-Based Models Multimodal Generation

Ask PDF

Oct 22, 2026

Scaling Transformer Energy-based Models For High Quality Image Synthesis

Energy-Based Models (EBMs) are a flexible class of generative models that decouple probabilistic modeling from the generation process. This decoupling directly enables compositional generation and inference-time scaling, where sample quality can be refined simply by increasing sampling steps. However, severe challenges in high-dimensional sampling and training instabilities with deep architectures have restricted EBMs to low-dimensional regimes. To resolve these bottlenecks, we introduce the Transformer Energy-Based Model (TEM), a scalable transformer energy function operating entirely within a latent space. On top of this backbone, we extract energies at multiple layers and sum them, composing complementary constraints from local texture to global semantics. Furthermore, two components make this high-quality at scale: Positive Representation Alignment (PRA), which supervises TEM's intermediate representations with a vision foundation model, and advanced sampling strategies that improve sample quality at inference. TEM-XL achieves an FID-50K of 3.82 on class-conditional ImageNet 256

\times

256 without classifier-free guidance, showing that a contrastive EBM can generate high-fidelity images. On ImageNet 512

\times

512, TEM achieves an FID-50K of 8.24, demonstrating that the approach scales to higher resolution. We further validate the core promises of EBMs: compositional generation, zero-shot inpainting, and out-of-distribution detection.

Jungsoo Lee, Alexi Gladstone, Yilun Du

MIT · UIUC · Harvard University

Image Generation Energy-Based Models Transformer Compositional Image Generation

Ask PDF

Jul 29, 2026

Equilibrium Training of Energy-Based Models with Parallel Trajectory Tempering

Energy-Based Models (EBMs) provide an interpretable framework for generative modeling of scientific data, but poor Markov Chain Monte Carlo mixing often limits their reliability. We introduce a training algorithm based on Parallel Trajectory Tempering (PTT), which exploits the continuity of the optimization path to maintain equilibrium sampling throughout learning. This enables stable and fast training on highly multimodal and data-scarce scientific datasets. Combined with reservoir sampling and adaptive optimization, PTT has a computational cost comparable to Persistent Contrastive Divergence, making it a practical replacement for standard training methods. It also provides direct estimates of thermalization times, equilibrium samples from trained models, and accurate log-likelihoods at essentially no additional cost. Experiments on Restricted Boltzmann Machines show that PTT consistently outperforms existing EBM training approaches. On discrete tabular data, it also surpasses state-of-the-art deep generative models, yielding higher-quality samples and greater robustness to overfitting and limited data. Our results make equilibrium maximum-likelihood training of EBMs practical and computationally efficient.

Nicolas Béreux, Aurélien Decelle, Cyril Furtlehner +1

Université Paris-Saclay, CNRS, INRIA, LISN, 91190 Gif-sur-Yvette, France · Escuela Técnica Superior de Ingenieros Industriales, Universidad Politécnica de Madrid, 28006 Madrid, Spain · Departamento de Física Teórica & IPARCOS, Universidad Complutense de Madrid, 28040 Madrid, Spain.

Markov Chain Monte Carlo Energy-Based Models Parallel Tempering

Ask PDF

Abstract

Generative Modeling

Explore similar work

Learning Multimodal Energy-Based Model with Multimodal Variational Auto-Encoder via MCMC Revision

Scaling Transformer Energy-based Models For High Quality Image Synthesis

Equilibrium Training of Energy-Based Models with Parallel Trajectory Tempering