Optimal Control of RKHS Diffeomorphic Flows for Classification and Generative Modeling
Organizations: Department of Mathematics, Tufts University, Medford, MA, USA
Abstract
We develop a unified optimal-control framework that solves supervised classification and likelihood-based density estimation with a single diffeomorphic flow whose velocity lives in a reproducing kernel Hilbert space (RKHS); the two tasks share all structure except the objective. Variable projection eliminates the terminal model (a classifier, or a Gaussian-mixture (GMM) reference), leaving a reduced control problem whose gradient comes from an analytic discrete adjoint. The RKHS parameterization makes the flow's divergence exact and deterministic at the cost of one kernel matrix-vector product, avoiding the stochastic trace estimators and potential-flow restrictions of existing continuous normalizing flows. Because the objective and gradient are deterministic, matrix-free reduced Gauss-Newton (GN) and related second-order solvers are practical for both tasks, which we benchmark against a range of first- and second-order optimizers; the learned map is a diffeomorphism with explicitly controlled regularity. In experiments from dimension 2 to 256, the reduced GN solvers use the fewest iterations of any optimizer tested and, together with joint L-BFGS, the least work; a multiscale kernel improves the density fit, and a trainable GMM reference reduces the deformation needed for multimodal transport at comparable likelihood while lowering the likelihood on real data. The full-model solve is quadratic in the particle count and sub-linear in dimension; a control-point reduction makes the particle-count scaling linear at near-lossless accuracy.
Figures & tables
| acronym | method | globalization |
|---|---|---|
| GD | (Euclidean) Gradient Descent ( ) | Armijo |
| NGD | Natural (preconditioned) GD ( ) | Armijo |
| AAGD | Anderson-Accelerated GD | Armijo |
| AANGD | Anderson-Accelerated NGD | Armijo |
| NAGD | Nesterov-Accelerated GD | Wolfe |
| NANGD | Nesterov-Accelerated NGD | Wolfe |
| nll | reconstruction error | ||||||
|---|---|---|---|---|---|---|---|
| target | reference | train | test | objective | time | ||
| circles | UGM | ||||||
| GMM | |||||||
| M8G | UGM | ||||||
| GMM | |||||||
| pinwheel | UGM | ||||||
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| gradient | ||||
|---|---|---|---|---|
| integrator | theoretical | terminal state | terminal-loss | objective |
| Euler | 1 | |||
| RK2 | 2 | |||
| RK4 | 4 | |||
| Euler | RK2 | RK4 | ||
|---|---|---|---|---|
| test accuracy | 2 | |||
| test accuracy | 32 | |||
| work | 32 |
| problem | (train) | (test) | |||
|---|---|---|---|---|---|
| two moons | 3 | 2 | |||
| noisy circles | 3 | 2 | |||
| three rings | 3 | 3 | |||
| 3D tori | 4 | 2 | |||
| XOR | 3 | 2 |
| problem | ||||||
|---|---|---|---|---|---|---|
| two moons | 300 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 |
| noisy circles | 300 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 |
| three rings | 300 | 77.1 | 99.2 | 99.2 | 99.2 | 100.0 |
| 3D tori | 200 | 96.9 | 98.8 | 100.0 | 100.0 | 100.0 |
| XOR | 300 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 |
| latent dimension ( ) | full model | reduced model ( ) |
|---|---|---|
| 16 (17) | 95.86 | 95.82 ( ) |
| 32 (33) | 96.80 | 96.24 ( ) |
| 64 (65) | 94.96 | 95.02 ( ) |
| method | objective | work | iters | reached |
| GN-CG (VarPro) | 17 | 5/5 | ||
| TR-GN (VarPro) | 21 | 5/5 | ||
| L-BFGS (VarPro) | 28 | 5/5 | ||
| NPRCG (VarPro) | 19 | 5/5 | ||
| L-BFGS | 58 | 4/5 | ||
| NPRCG | 318 | 4/5 |
| variant | work | iters | |||
|---|---|---|---|---|---|
| NGD (VarPro) | 51 | ||||
| AANGD (VarPro) | 36 | ||||
| NPRCG (VarPro) | 19 | ||||
| L-BFGS (VarPro) | 28 | ||||
| GN-CG (VarPro) | 17 | ||||
| TR-GN (VarPro) | 21 |
| accuracy (%) | |||||||
|---|---|---|---|---|---|---|---|
| problem | method | work | iters | reached | train | test | |
| noisy circles | GN-CG (VarPro) | 5 | 5/5 | 100.0 | 100.0 | ||
| TR-GN (VarPro) | 8 | 5/5 | 100.0 | 100.0 | |||
| L-BFGS | 23 | 5/5 | 100.0 | 100.0 | |||
| 3D tori | GN-CG (VarPro) | 8 | 4/5 | 100.0 | 100.0 | ||
| TR-GN (VarPro) | 9 | 4/5 | 100.0 | 100.0 | |||
| accuracy (%) | ||||
|---|---|---|---|---|
| test | train | iterations | time (s) | |
| 500 | 93.9 | 100.0 | 10 | 6.8 |
| 96.2 | 100.0 | 10 | 9.3 | |
| 97.6 | 100.0 | 9 | 20.0 | |
| 98.6 | 100.0 | 8 | 23.9 | |
| 98.9 | 100.0 | 8 | 34.6 | |
| UGM | GMM | |||
|---|---|---|---|---|
| test nll | test nll | |||
| nll | gradient norm | reconstruction error | |||||||
|---|---|---|---|---|---|---|---|---|---|
| reference | scale | train | test | objective | time | ||||
| UGM | S | ||||||||
| M | |||||||||
| GMM | S | ||||||||
| M | |||||||||
| nll | gradient norm | reconstruction error | |||||||
|---|---|---|---|---|---|---|---|---|---|
| reference | scale | train | test | objective | time | ||||
| UGM | S | ||||||||
| M | |||||||||
| GMM | S | ||||||||
| M | |||||||||
| nll | gradient norm | reconstruction error | |||||||
|---|---|---|---|---|---|---|---|---|---|
| reference | scale | train | test | objective | time | ||||
| UGM | S | ||||||||
| M | |||||||||
| GMM | S | ||||||||
| M | |||||||||
| nll | gradient norm | reconstruction error | |||||||
|---|---|---|---|---|---|---|---|---|---|
| reference | scale | train | test | objective | time | ||||
| UGM | S | ||||||||
| M | |||||||||
| GMM | S | ||||||||
| M | |||||||||
Flow-Based Generative Modeling
Flow-based generative modeling learns probability transports from simple noise to data for fast sampling, density estimation, and controllable generation.
See all 619 papers