Lagrangian--Hamiltonian Flows for Video Prediction and Image Generation: A Symplectic Perspective

Jiawei Hu

cs.CVSep 28, 2026

Lagrangian--Hamiltonian Flows for Video Prediction and Image Generation: A Symplectic Perspective

Authors: Jiawei Hu

Organizations: Department of Mathematics and Statistics, Boston University

Abstract

We introduce LHFM, a geometric framework for learning image dynamics. Drawing on structures central to classical mechanics, symplectic geometry, and geometric quantization, LHFM represents each image as an exact Lagrangian graph and models its evolution through image-dependent Hamiltonian flows, which yield a transport--source parameterization of image velocities. Our primary application is deterministic video prediction: LHFM-V is a recurrent model that advances frames by integrating predicted transport and source fields, and achieves the lowest reported FLOP count among the compared recurrent models with similar prediction accuracy. The image variant, LHFM-I, shows that the same construction is compatible with flow matching: in a matched experiment, it attains a lower FID than the flow-matching baseline.

Figures & tables

Figure 1: Hamiltonian deformation of a Lagrangian submanifold. For a fixed image trajectory, the time-dependent ambient Hamiltonian $H_{t}$ transports $L_{0}$ to $L_{t}=\Phi_{t}(L_{0})$ . The surfaces schematically represent higher-dimensional exact Lagrangians.

	Classical mechanics	Semiclassical (WKB)
Representation	Point $(q,p)\in T^{*}Q$	Lagrangian graph with amplitude
	Vanilla Flow Matching	LHFM
Representation	An image point $J$	Graph Lagrangian $L_{J}=\operatorname{graph}(dS_{J})$
Dynamics	A general learned Flow in image space	Hamiltonian flow on $T^{*}Q$

Table 1: Geometric motivation for LHFM. Hamiltonian evolution refers to the ambient cotangent bundle, not the image-space vector field.

Figure 2: Local information encoded by the Lagrangian representation. Arrow directions are computed at $x_{0}$ in Euclidean pixel coordinates; lengths are schematic and not to scale.

Figure 3: Separate roles of source and transport.

Figure 4: Spatial transport versus pixelwise blending. (a,d) Endpoint images: Girl with a Pearl Earring and an AI-generated cat portrait. (b) Pixelwise linear interpolation at $t=0.5$ . (c) Interpolation compatible with the LHFM transport–source formulation at the same time.

Method	FID $\downarrow$	NFE
DDPM (reported)	7.48	274
OT-FM (reported)	6.35	142
VP-FM (reported)	8.06	183
S.I. (reported)	10.27	—
OT-FM (reproduced)	11.527	139.83
VP-FM (Tong et al.)	4.335	525.92

Table 2: Unconditional CIFAR-10 generation. Published results are taken from Table 5 of Tong et al. (2024) . The bottom two rows report our matched 150k-update experiments; boldface highlights LHFM-I.

Figure 5: FID versus sampling budget for the matched 150k-update models on unconditional CIFAR-10 (fixed-step Heun solver). Labels give the FID reduction of LHFM-I relative to I-CFM at each budget.

Method	Params (M)	FLOPs (G)	MSE $\downarrow$	MAE $\downarrow$	SSIM $\uparrow$
ConvLSTM ( Shi et al., 2015 )	15.0	56.8	103.3	182.9	0.707
PredRNN ( Wang et al., 2017 )	23.8	116.0	56.8	126.1	0.867
PredRNN++ ( Wang et al., 2018 )	38.6	171.7	46.5	106.8	0.898
MIM ( Wang et al., 2019b )	38.0	179.2	44.2	101.1	0.910
E3D-LSTM ( Wang et al., 2019a )	51.0	298.9	41.3	86.4	0.910
PhyDNet ( Le Guen & Thome, 2020 )	3.1	15.3	24.4	70.3	0.947

Table 3: Moving MNIST results for recurrent video predictors, which capture temporal dependencies through recurrent hidden-state updates. Baseline numbers are as reported in the original papers.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Standard I-CFM	LHFM-I
Dataset / resolution	CIFAR-10 / $32\times 32$ RGB
Conditioning	Unconditional
Base channels / multipliers	128 / $(1,2,2,2)$
Residual blocks per level (down / up)	$2/3$
Attention resolutions	$16,8$ ; $4$ (bottleneck)
Attention heads / channels per head	$4/64$

Appendix

Table 4: Settings for the matched CIFAR-10 comparison. Shared entries apply to both models; reported results use the 150k-update checkpoints.

Figure 6: Architecture of LHFM-V for deterministic video prediction. Each future frame is produced by two transport–source half-steps, with image feedback and recurrent-state updates. In the diagram, $\tau$ , $u$ , and $r$ correspond to $s$ , $w$ , and $r^{\mathrm{net}}$ in the text, respectively; $w$ is measured in pixels per frame and satisfies $w=Nu$ in the text’s notation. The function $\varphi_{1}(X)=\int_{0}^{1}e^{(1-\sigma)X}\,d\sigma$ represents the source integral in equation 12 .

Setting	LHFM-V
Resolution / channels	$64\times 64$ / 1
Observed / future frames	$10/10$
Encoder widths / residual blocks	$(64,128,448)$ / 3
Recurrent-state channels / resolution	384 / $16\times 16$
Memory frames / attention heads	6 plus history / 4
Memory key / value dimensions	$128/256$

Appendix

Table 5: Architecture and training settings for LHFM-V on Moving MNIST.

Figure 7: CIFAR-10 samples from LHFM-I, generated using a 150k-update EMA checkpoint and 128 Heun steps (256 NFE). The 64-image grid is retained in its original order.

Figure 8: LHFM-V predictions on Moving MNIST (sequence IDs 0–2). For each sequence, rows show the ten observed frames (1–10), the ten ground-truth future frames (11–20), and the corresponding LHFM-V predictions. Time progresses from left to right.

Figure 9: LHFM-V predictions on Moving MNIST (sequence IDs 3–5). Rows and time ordering follow Figure 8 . Each observed sequence produces one deterministic prediction.

Explore similar work

Posterior Augmented Flow Matching
May 1, 2026George Stoica, Sayak Paul, Matthew Wallingford +6Contrastive Flow Matching Collapse
Georgia Tech · University of Washington · Hugging Face +2
Energy-Guided Flow Matching
Aug 6, 2026Haoyang Tong, Yu He, Fang Li +6Contrastive Flow Matching Conditional Flow Matching
MAIS & NLPR, CASIA · JD.com · Xi'an Jiaotong University +1
Learning Hamiltonian Flow Maps: Mean Flow Consistency for Large-Timestep Molecular Dynamics
Jan 29, 2026Winfried Ripken, Michael Plainer, Gregor Lied +5Invertible Flows Hamiltonian
Technical University Berlin · BIFOLD · Free University of Berlin +6

May 1, 2026, cs.CVJ/K move · Enter open · S save

Posterior Augmented Flow Matching

George Stoica, Sayak Paul, Matthew Wallingford, Vivek Ramanujan+5

Georgia Tech · University of Washington · Hugging Face+2

Contrastive Flow Matching Collapse

Open paper AskPDF arXiv

Flow matching (FM) trains a time-dependent vector field that transports samples from a simple prior to a complex data distribution. However, for high-dimensional images, each training sample supervises only a single trajectory and intermediate point, yielding an extremely sparse and high-variance training signal. This under-constrained supervision can cause flow collapse, where the learned dynamics memorize specific source-target pairings, mapping diverse inputs to overly similar outputs, failing to generalize. We introduce Posterior-Augmented Flow Matching (PAFM), a theoretically grounded generalization of FM that replaces single-target supervision with an expectation over an approximate posterior of valid target completions for a given intermediate state and condition. PAFM factorizes this intractable posterior into (i) the likelihood of the intermediate under a hypothesized endpoint and (ii) the prior probability of that endpoint under the condition, and uses an importance sampling scheme to construct a mixture over multiple candidate targets. We prove that PAFM yields an unbiased estimator of the original FM objective while substantially reducing gradient variance during training by aggregating information from many plausible continuation trajectories per intermediate. Finally, we show that PAFM improves over FM by up to 3.4 FID50K across different model scales (SiT-B/2 and SiT-XL/2), different architectures (SiT and MMDiT), and in both class and text conditioned benchmarks (ImageNet and CC12M), with a negligible increase in the compute overhead. Code: https://github.com/gstoica27/PAFM.git.

Abstract

Explore similar work

Posterior Augmented Flow Matching

Energy-Guided Flow Matching

Learning Hamiltonian Flow Maps: Mean Flow Consistency for Large-Timestep Molecular Dynamics