cs.LGSep 4, 2026

Newton Matching for Generative Modeling: A Unified Framework for Fine-Tuning and Sampling

Authors: Zeyang Li, Yunan Wang, Paolo Giaretta, Navid Azizan

Organizations: Massachusetts Institute of Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We develop Newton Matching, a unified framework for fine-tuning and sampling in generative modeling. The target is π∝μeτrπ\proptoμe^{τr}, where rr is the reward, τ>0τ>0 the inverse temperature, and μμ denotes the pretrained model's terminal density for fine-tuning or the constant 11 for sampling. We shift the paradigm from isolated losses to iterative optimization over canonical models: population minimizers of standard conditional matching for terminal densities. Under compatible smooth-realization assumptions, canonical velocities form a manifold diffeomorphic to the density manifold. Transporting the Fisher-Rao metric and mixture connection to this manifold, we show that the reverse-KL Hessian equals the metric, so the Newton direction coincides with the negative Fisher-Rao gradient. At terminal density ρρ, each stage takes a tangential step generated by the regularized reward r−1τlog⁡(ρ/μ)r-\frac1τ\log(ρ/μ), followed by terminal-density-preserving canonicalization. This canonical retraction yields an exact finite-stepsize density characterization. For the ideal iteration, we prove strict reverse-KL descent away from the target for 0<η≤τ0 < η\le τ, global convergence under mild conditions, and local quadratic convergence for full steps (η=τη=τ). Covariance and gradient forms, each with forward or reverse regression-pair constructions, yield sample-wise tangential-update losses with the same population minimizer, without importance sampling or full-trajectory backpropagation. We develop approximate updates and define critical-point consistency as vanishing tangential displacement if and only if ρ=πρ=π. We recover representative methods as exact realizations, critical-point-consistent approximations, or objective-altering variants, enabling modular algorithm design. Our work advances the theory and algorithms of reinforcement learning for generative models.

Figures & tables

Figure 1
Figure 2: Nonuniqueness of velocity representations for terminal densities. The canonical velocity field vρv^{\rho} and a distinct noncanonical velocity field v^\hat{v} both transport p0=N(0,I)p_{0}=\mathcal{N}(0,I) to the same terminal density ρ\rho .
Figure 3: Global geometry of the canonical construction.
Figure 4: Geometric structures and retraction operations under the canonical diffeomorphism. The tangent representation, Fisher–Rao metric, and mixture connection on P\mathscr{P} are transported to Vcan\mathscr{V}_{\mathrm{can}} through C:P→∼Vcan\mathcal{C}:\mathscr{P}\xrightarrow{\sim}\mathscr{V}_{\mathrm{can}} . Conversely, the canonical retraction on Vcan\mathscr{V}_{\mathrm{can}} corresponds through C−1=T∣Vcan:Vcan→∼P\mathcal{C}^{-1}=\left.\mathcal{T}\right|_{\mathscr{V}_{\mathrm{can}}}:\mathscr{V}_{\mathrm{can}}\xrightarrow{\sim}\mathscr{P} to the induced retraction on P\mathscr{P} .
Figure 5: Canonical retraction and its value-ascent certificate. Starting from the canonical velocity field vρ=C(ρ)v^{\rho}=\mathcal{C}(\rho) , the finite-stepsize tangential update vˉ=vρ+ηΓρ,f\bar{v}=v^{\rho}+\eta\Gamma^{\rho,f} generally leaves the canonical manifold Vcan\mathscr{V}_{\mathrm{can}} . Then, vˉ\bar{v} is projected back to the canonical manifold as R(vˉ)\mathcal{R}(\bar{v}) while preserving the terminal density q=T(vˉ)q=\mathcal{T}(\bar{v}) . The induced update satisfies the value-ascent certificate EX∼q[f(X)]≥EX∼ρ[f(X)]\mathbb{E}_{X\sim q}[f(X)]\geq\mathbb{E}_{X\sim\rho}[f(X)] .
Figure 6: Mechanism of finite-stepsize value ascent. Along the updated density path (νt)t∈[0,1](\nu_{t})_{t\in[0,1]} generated by vˉ=vρ+ηΓρ,f\bar{v}=v^{\rho}+\eta\Gamma^{\rho,f} , the expected posterior-value improvement relative to the density path (ptρ)t∈[0,1](p_{t}^{\rho})_{t\in[0,1]} decomposes into the scaled marginal divergence 1ηKL⁡(νt∥ptρ)\frac{1}{\eta}\operatorname{KL}\left(\nu_{t}\middle\|p_{t}^{\rho}\right) and an accumulated nonnegative quadratic term η∫0tκsEX∼νs[∥∇Vsρ[f](X)∥22]ds\eta\int_{0}^{t}\kappa_{s}\mathbb{E}_{X\sim\nu_{s}}\left[\left\|\nabla V_{s}^{\rho}[f](X)\right\|_{2}^{2}\right]\mathrm{d}s . At t=1t=1 , this decomposition yields the terminal value improvement EX∼q[f(X)]−EX∼ρ[f(X)]≥0\mathbb{E}_{X\sim q}[f(X)]-\mathbb{E}_{X\sim\rho}[f(X)]\geq 0 .
Figure 7: One Newton Matching step. The negative Fisher–Rao gradient and the mixture-connection Newton direction coincide. The canonical retraction with this direction produces a canonical-velocity update that decreases the reverse KL divergence.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Construction of ambient classes, admissible subclasses, and smooth structures.
More in this topic

Generative Modeling

Generative modeling learns data distributions to create new samples, from images and text to molecules, simulations, and structured records.

See all 500 papers

Explore similar work

CardsList
Apr 30, 2026

A unified perspective on fine-tuning and sampling with diffusion and flow models

We study the problem of training diffusion and flow generative models to sample from target distributions defined by an exponential tilting of a base density; a formulation that subsumes both sampling from unnormalized densities and reward fine-tuning of pre-trained models. This problem can be approached from a stochastic optimal control (SOC) perspective, using adjoint-based or score matching methods, or from a non-equilibrium thermodynamics perspective. We provide a unified framework encompassing these approaches and make three main contributions: (i) bias-variance decompositions revealing that Adjoint Matching/Sampling and Novel Score Matching have finite gradient variance, while Target and Conditional Score Matching do not; (ii) norm bounds on the lean adjoint ODE that theoretically support the effectiveness of adjoint-based methods; and (iii) adaptations of the CMCD and NETS loss functions, along with novel Crooks and Jarzynski identities, to the exponential tilting setting. We validate our analysis with reward fine-tuning experiments on Stable Diffusion 1.5 and 3.

Carles Domingo-Enrich, Yuanqi Du, Michael S. Albergo

Microsoft Research New England · Harvard University, Kempner Institute

Stochastic Optimal ControlDiffusion SamplingDiffusion Model Fine-TuningFlow-Based Generative ModelingScore-Based Generative Modeling
AskPDF
Oct 12, 2026

A Survey of Flow Matching in Reinforcement Learning

Flow Matching (FM) has recently emerged as a principled and efficient generative modeling framework for reinforcement learning (RL), enabling expressive, multimodal policy parameterizations via deterministic probability transport. Compared to diffusion-based policies that rely on stochastic denoising chains, FM uses sampling based on ordinary differential equations (ODEs), with learned velocity fields, which can substantially reduce inference latency and simplify the incorporation of RL objectives. As research in flow-based RL rapidly accelerates across offline continuous control, online fine-tuning, and foundation model alignment, the literature has become highly fragmented. In this survey, we provide a comprehensive taxonomy of flow-matching approaches in reinforcement learning. We organize the literature along two axes: the target distribution being modeled (e.g., action policies, value critics, transition dynamics) and the mechanism of RL signal integration (e.g., energy-weighted regression, flow-based policy gradients, and group relative policy optimization). Furthermore, we survey emerging frontiers such as discrete and non-Euclidean action spaces, provide a systematic comparative analysis against Gaussian and diffusion baselines, and outline critical open problems. Ultimately, this survey serves as a foundational roadmap for the next generation of generative reinforcement learning and alignment.

Nabuat Zaman Nahim, Fairoz Nower Khan, Peizhong Ju

Department of Computer Science, University of Kentucky, Lexington, Kentucky 40506, USA

Reinforcement LearningFlow-Based Generative ModelingRL for Generative Models
AskPDF
May 7, 2026

Energy Generative Modeling: A Lyapunov-based Energy Matching Perspective

Generative models based on static scalar energy functions represent an emerging paradigm in which a single time independent potential drives sample generation through its gradient field, eliminating the need for time conditioning entirely. We unify the training and sampling phases of this paradigm, conventionally treated as separate procedures, within a single framework: density transport on the Wasserstein space, cast as a nonlinear control problem in which the Kullback Leibler (KL) divergence serves as a Lyapunov function. Training and sampling are then two instances of this same master dynamics, differing only in initial condition. Within this autonomous framework we develop two analytic results. First, since the Lyapunov certificate is asymptotic, we derive a finite step stopping criterion for Langevin sampling and prove that no Lyapunov certificate exists for the deterministic gradient flow on the same energy landscape. Second, the reformulation brings the toolkit of nonlinear control theory to bear on static scalar energy generative modeling, that is, we show that additive composition of trained scalar energies retains an explicit Gibbs invariant measure and inherits the closed-loop Lyapunov certificate. Beyond these immediate results, this reformulation bridges static scalar energy generative models with the full toolkit of nonlinear control theory, opening the door to barrier functions for constrained generation and contraction metrics for accelerated sampling. Experiments on synthetic distributions validate the theoretical predictions.

Yixuan Wang, Wenqian Xue, Warren E. Dixon

Department of Mechanical and Aerospace Engineering University of Florida

Energy-Based Models
AskPDF