Focus, Align, and Diffuse: Time-Series-Aware Keyframe Diffusion for Cardiac Dynamic Synthesis from Sparse Observations

Junkai Liu; Haofan Wu; Nay Aung; Joao A. C. Lima; Steffen E. Petersen; Le Zhang

eess.IVOct 13, 2026

Focus, Align, and Diffuse: Time-Series-Aware Keyframe Diffusion for Cardiac Dynamic Synthesis from Sparse Observations

Authors: Junkai Liu, Haofan Wu, Nay Aung, Joao A. C. Lima, Steffen E. Petersen, Le Zhang

Organizations: School of Engineering, University of Birmingham · William Harvey Research Institute, Queen Mary University London · Division of Cardiology, Johns Hopkins University School of Medicine

Abstract

Dense cardiac dynamics are essential for assessing cardiac function, yet in practice clinicians often observe only extremely sparse measurements. However, recovering full dynamics from such sparsity is a highly ill-posed problem due to the lack of temporal cues. While auxiliary time-series signals like ECG provide physiological guidance, effectively exploiting them is challenging due to the cross-modal representation gap and phase misalignment. In this work, we present CardioFAD, a Focus--Align--Diffuse paradigm that reformulates sparse cardiac dynamic synthesis as a recoverability--alignability anchor discovery problem. Our core insight is that resolving extreme temporal ambiguity requires a compact set of anchor states that are both sufficient to recover the visual trajectory and reliable for ECG--visual alignment. We therefore learn motion-salient keyframes under two complementary principles: trajectory sufficiency, which preserves trajectory-critical intra-modal motion content, and anchor alignment, which promotes consistent inter-modal correspondence at these anchors. We implement the Focus and Align modules optimized by the two aspects. Diffuse then performs two-stage generation by first synthesizing keyframes and subsequently interpolating the remaining frames. Extensive experiments on cardiac MRI and echocardiography videos demonstrate superior visual quality, temporal coherence, and improved functional consistency under the extreme one-shot regime.

Figures & tables

Figure 1: (a) Challenge: Extreme sparsity causes temporal ambiguity, while ECG suffers from cross-modal misalignments. (b) Solution: We reformulate the task as a recoverability–alignability anchor discovery problem. (c) Principle-to-Performance: Our learned keyframes satisfy both principles, yielding superior synthesis quality.

Figure 2: Overview of CardioFAD. Our pipeline contains three stages: (i) Focus: A tokenizer predicts keyframe probabilities, optimized by the TSP to ensure the selected anchors can reconstruct full intra-modal dynamics. (ii) Align: Guided by the AAP, a keyframe-weighted matrix synchronizes visual and ECG latents, establishing keyframes as cross-modal anchors. (iii) Diffuse: A video DiT performs two-stage synthesis, first generating the keyframes, then interpolating the remaining frames.

Methods	Volume2Seq					Slice2Seq
Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$
VideoComposer Wang et al. (2023)	28.15	0.843	0.107	128.76	1762.34	21.86	0.497	0.289	314.85	4574.61
MotionCtrl Wang et al. (2024b)	28.46	0.825	0.104	119.52	1334.87	21.25	0.488	0.294	365.97	4914.27
Tora Zhang et al. (2025c)	29.74	0.854	0.092	97.30	1025.57	23.10	0.539	0.251	267.92	3109.75
MotionPro Zhang et al. (2025d)	27.88	0.817	0.116	145.39	1887.96	22.09	0.501	0.269	292.17	3756.61
CardioFAD (One-stage)	30.51	0.861	0.082	82.48	1007.63	23.75	0.559	0.231	211.49	2785.30

Table 1: Quantitative results for 3D+t CMR synthesis on UKBB dataset .

Methods	EchoNet-Dynamic					EchoNet-Pediatric (A4C)					EchoNet-Pediatric (PSAX)
Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$
General Video Generation
VideoComposer Wang et al. (2023)	22.78	0.539	0.109	48.25	284.16	27.49	0.633	0.089	34.74	346.31	27.26	0.621	0.084	42.19	389.07
Tora Zhang et al. (2025c)	23.90	0.548	0.102	52.06	241.96	27.86	0.638	0.082	28.94	361.09	28.37	0.638	0.081	35.52	347.21
ECHO Video Generation
HeartBeat Zhou et al. (2024)	21.24	0.527	0.122	58.97	376.51	26.71	0.629	0.088	37.56	415.71	27.65	0.630	0.079	45.02	407.26

Table 2: Quantitative results for 2D+t ECHO synthesis on three datasets .

Figure 3: Qualitative comparisons on CMR (left) and ECHO (right) . Frames are grouped by temporal roles, with top ECG curves providing physiological context. Zoomed-in crops and difference heatmaps are provided. Compared to baselines, CardioFAD achieves higher visual fidelity and temporal consistency, avoiding phase drift particularly at challenging positions.

Table 6

Figure 4: Sparse observation budget . Line charts report FVD, and bar charts report PSNR. When

B=1

(extremely sparse), CardioFAD exhibits its greatest advantage.

Figure 5: Visualization of keyframe selection strategies . Representative CMR (left) and ECHO (right) sequences with ECG overlaid. Inter-frame difference heatmaps are provided to illustrate motion magnitude. Our learned keyframes explicitly target motion-salient phases.

Figure 6: (a) TSP Effectiveness: CardioFAD achieves lower error with fewer frames, indicating higher information density. (b) AAP Effectiveness: Cross-modal retrieval metrics show that AAP successfully concentrates alignment capability on keyframes compared to non-keyframes. (c) Interaction Analysis: Visualizing keyframes in Motion-Correlation space. Removing TSP biases selection towards high-correlation frames (top), while removing AAP leads to high-motion frames (right). CardioFAD (top-right) effectively balances dynamic salience with physiological consistency.

Figure 7: Cardiac trajectory visualization . (a) t-SNE of CMR latents. (b) t-SNE of ECG latents with a trajectory through the temporal centroids. (c) Dataset-level mean ECG-CMR similarity matrix $S$ (top) and the corresponding alignment matrix $\tilde{A}$ (bottom).

Figure 8: Keyframe distribution visualization . (a) GT ED/ES frames (top) and our selected keyframes (bottom) on the cardiac manifold. ED/ES-matching keyframes are outlined in black. (b) Mean keyframe phase density and ED/ES phase histograms (dashed lines: mean phases). (c) Distribution of $\lvert\Delta{\text{frame}}\rvert$ to the nearest keyframe from each ED/ES frame. (d) Keyframe Recall@K.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Datasets	Modality / View	Train.	Val.	Test.	Raw Average Frame Count
CMR Dataset
UKBB [ 52 ]	3D+t CMR	3503	438	438	50
ECHO Datasets
EchoNet-Dynamic [ 49 ]	A4C	7465	1288	1277	177
EchoNet-Pediatric (A4C) [ 56 ]	A4C	2580	336	368	101
EchoNet-Pediatric (PSAX) [ 56 ]	PSAX	3559	448	519	103

Appendix

Table 5: Details of the CMR and ECHO datasets used in our experiments.

Table 13

Methods	Latent MSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$
Uniform keyframes	0.023	25.94	0.572	0.081	40.41	188.36
Random keyframes	0.027	25.17	0.554	0.091	45.59	251.03
GT ED/ES	0.020	26.20	0.572	0.083	42.76	202.41
CardioFAD (One-stage)	–	25.14	0.556	0.089	46.82	232.17
CardioFAD (Frozen Tokenizer)	0.018	27.56	0.592	0.070	31.84	162.93
CardioFAD (Joint Tokenizer)	0.014	27.73	0.596	0.067	26.92	137.70

Appendix

Table 8: Frozen visual tokenizer validation. The visual tokenizer is trained independently and frozen before learning the ECG encoder, selector, alignment module, and diffusion model.

Methods	EchoNet-Pediatric (A4C)			EchoNet-Pediatric (PSAX)
Methods	$R^{2}$ $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	$R^{2}$ $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$
General Video Generation
VideoComposer [ 66 ]	0.68	4.93	6.73	0.62	4.88	7.21
Tora [ 78 ]	0.66	4.89	6.86	0.69	4.59	6.52
ECHO Video Generation
HeartBeat [ 81 ]	0.71	4.62	6.29	0.65	4.40	6.94

Appendix

Table 9: Cardiac functional consistency evaluated by EF estimation for 2D+t ECHO synthesis on EchoNet-Pediatric and its A4C/PSAX subsets [ 56 ] .

Metric	Reduced	Mildly reduced	Preserved-range	All
Metric	$\mathrm{EF}\leq 40$	$40<\mathrm{EF}<50$	$\mathrm{EF}\geq 50$	All
$n$	161	125	991	1,277
Real-video MAE $\downarrow$	4.81	4.49	3.76	3.96
One-stage MAE $\downarrow$	5.84	5.29	4.80	4.98
CardioFAD MAE $\downarrow$	5.10	4.65	3.96	4.17
Bias vs. GT	0.72	0.58	0.15	0.26

Appendix

Table 10: EF-stratified functional preservation on EchoNet-Dynamic.

Metric	All stratified	Reduced EF	High motion
Metric	cases	Reduced EF	heterogeneity
$n$	60	20	15
Area-curve MAE (One-stage) $\downarrow$	0.063	0.078	0.095
Area-curve MAE (CardioFAD) $\downarrow$	0.048	0.051	0.067
Segmental-motion MAE (One-stage) $\downarrow$	0.119	0.140	0.182
Segmental-motion MAE (CardioFAD) $\downarrow$	0.097	0.116	0.141

Appendix

Table 11: Direct motion preservation on EchoNet-Dynamic.

Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$
ECHOPulse [ 34 ]	25.60 ±0.19	0.567 ±0.003	0.081 ±0.004	37.24 ±2.5	194.19 ±11.5
CardioFAD (One-stage)	25.17 ±0.16	0.558 ±0.003	0.090 ±0.003	46.52 ±2.9	235.08 ±9.8
CardioFAD (Ours)	27.68 ±0.20	0.595 ±0.004	0.068 ±0.004	26.48 ±2.2	139.54 ±12.8

Appendix

Table 12: Multi-seed robustness analysis on EchoNet-Dynamic. Results are reported as mean ± standard deviation over three independent random seeds.

Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$
Keyframe Learning Principle
w/o TSP	29.07	0.836	0.098	104.53	1226.37
w/o AAP	30.53	0.861	0.082	81.63	910.04
w/o TSP & w/o AAP	28.21	0.827	0.111	134.54	1507.13
Keyframe Selection Strategy
w/ uniform keyframes	29.86	0.852	0.094	97.41	1028.59

Appendix

Table 13: Principle-guided keyframe learning ablation for CMR Volume2Seq synthesis.

Table 20

Table 21

Dataset	Temporal condition	Patient- specific	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$
UKBB	Paired real ECG	$\checkmark$	32.41	0.885	0.064	59.96	736.01
	Subject-shuffled real ECG	$\times$	31.12	0.870	0.076	91.48	958.23
	Canonical training ECG	$\times$	30.74	0.862	0.083	98.77	1046.28
	Null ECG	$\times$	29.24	0.834	0.099	107.39	1281.46
EchoNet-Dynamic	Correct phase template	$\times$	27.73	0.596	0.067	26.92	137.70
	Random-offset template	$\times$	25.59	0.563	0.085	37.93	217.08

Appendix

Table 18: Attribution of temporal guidance on UKBB and EchoNet-Dynamic. We vary only the temporal condition under otherwise matched settings to disentangle patient-specific ECG information, generic phase guidance, and two-stage keyframe anchoring.

Methods	Avg. LPIPS $\downarrow$	Best-of-8 LPIPS $\downarrow$	Pairwise LPIPS	Norm. motion diversity	Uncertainty– error $\uparrow$	EF MAE $\downarrow$	Within-case EF std. $\downarrow$
CardioFAD (One-stage)	0.091	0.085	0.035	0.40	0.29	5.03	2.21
CardioFAD (Ours)	0.070	0.064	0.025	0.27	0.50	4.21	1.35

Appendix

Table 19: Conditional fidelity–diversity analysis on EchoNet-Dynamic. Eight samples are generated for each fixed sparse observation by varying only the rectified-flow sampling seed.

Observed-frame phase	PSNR $\uparrow$	FVD $\downarrow$	EF MAE $\downarrow$
Near ED	27.92	128.92	4.10
Near ES	27.84	135.04	4.08
Contraction	27.72	140.57	4.23
Relaxation	27.68	144.82	4.26
Random observation	27.73	137.70	4.17

Appendix

Table 20: Sensitivity to the observed-frame phase on EchoNet-Dynamic under matched inference settings.

Figure 9: Robustness to ECG perturbations on UKBB (CMR). FVD (lower is better) under three inference-time ECG corruptions: Gaussian noise injection with probability

p

, random lead dropout with

L

missing leads, and temporal phase jitter with

\delta

shifted frames. CardioFAD (Full) degrades most gracefully and remains consistently best across perturbation severities.

Figure 10: Robustness to pseudo-ECG perturbations on EchoNet-Dynamic. FVD (lower is better) under inference-time perturbations of the pseudo-ECG: global phase shifts

\Delta

, component dropout (zero-masking phase/amplitude/derivative), and random segment masking with ratio

r

. CardioFAD (Full) remains consistently best and degrades most gracefully across perturbation severities.

Methods	CMR (UKBB)		ECHO (EchoNet-Dynamic)
Methods	Latency (s) $\downarrow$	Peak VRAM (GB) $\downarrow$	Latency (s) $\downarrow$	Peak VRAM (GB) $\downarrow$
CardioFAD (One-stage)	6.19	14.82	3.34	4.91
CardioFAD (Ours)	12.27	14.84	6.62	4.93

Appendix

Table 21: Inference cost comparison between the one-stage baseline and CardioFAD on CMR and ECHO. We report end-to-end latency per sample and peak VRAM under standardized settings (batch size

=1

, single A100 GPU, same sampling steps, and mixed precision).

Data	Methods	Tokenizer Training Cost				Synthesis Quality
Data	Methods	Params (M) $\downarrow$	FLOPs (G) $\downarrow$	Time/Epoch (s) $\downarrow$	Total (GPU h) $\downarrow$	PSNR $\uparrow$	FVD $\downarrow$
CMR	w/o TSP	48.01	2008.23	480.7	535.1	25.76	1982.53
	w/o AAP	48.56	2011.48	483.0	536.7	27.25	1627.98
	w/o TSP & AAP	48.01	2008.23	480.6	534.0	25.31	2376.82
	CardioFAD (Full)	48.56	2011.48	483.2	536.9	29.00	1291.87
ECHO	w/o TSP	44.45	247.65	202.3	224.8	27.13	368.04

Appendix

Table 22: Tokenizer training efficiency vs. downstream synthesis quality. We compare principle ablations in tokenizer training against the full model. Synthesis quality is reported as the average across the two CMR evaluation settings and across the three ECHO datasets, respectively.

Figure 11: Qualitative comparisons with variants on CMR.

Figure 12: Qualitative comparisons with variants on ECHO.

Explore similar work

CardioDiT: Latent Diffusion Transformers for 4D Cardiac MRI Synthesis
Mar 26, 2026Marvin Seyfarth, Sarah Kaye Müller, Arman Ghanaat +7Medical Image Synthesis Latent Diffusion Models
Institute for Artificial Intelligence in Cardiovascular Medicine, Medical Faculty of Heidelberg University, Heidelberg University, Germany · Department of Cardiology, Angiology, Pneumology, Heidelberg University Hospital, Heidelberg, Germany · DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim, Heidelberg, Germany +4
One Frame, Full Heartbeat: ECG-Free Cardiac Cine MRI Synthesis via Phase-Conditioned Flow Matching
Oct 7, 2026Shiyi Wang, Ruochen Sun, Peirong Liu +2Medical Image Synthesis Cardiac MRI
Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA · Gordon Center for Medical Imaging, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA · Center for Advanced Medical Computing and Analysis, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA
Temporally Consistent and Controllable Video Generation of 2D Cine CMR via Latent Space Motion Modeling
Jun 8, 2026Yiheng Cao, Gustavo Andrade-Miranda, Jiatian Zhang +2Latent Dynamics Modeling Text-to-Video Generation
SyCoIA - IMT Mines Alès · Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, China · SyCoIA, IMT Mines Ales, Ales, France

Mar 26, 2026J/K move · Enter open · S save

CardioDiT: Latent Diffusion Transformers for 4D Cardiac MRI Synthesis

Marvin Seyfarth, Sarah Kaye Müller, Arman Ghanaat, Isabelle Ayx+6

Institute for Artificial Intelligence in Cardiovascular Medicine, Medical Faculty of Heidelberg University, Heidelberg University, Germany · Department of Cardiology, Angiology, Pneumology, Heidelberg University Hospital, Heidelberg, Germany · DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim, Heidelberg, Germany+4

Medical Image Synthesis Latent Diffusion Models Cardiac MRI Diffusion Transformer

Open paper AskPDF arXiv

Latent diffusion models (LDMs) have recently achieved strong performance in 3D medical image synthesis. However, modalities like cine cardiac MRI (CMR), representing a temporally synchronized 3D volume across the cardiac cycle, add an additional dimension that most generative approaches do not model directly. Instead, they factorize space and time or enforce temporal consistency through auxiliary mechanisms such as anatomical masks. Such strategies introduce structural biases that may limit global context integration and lead to subtle spatiotemporal discontinuities or physiologically inconsistent cardiac dynamics. We investigate whether a unified 4D generative model can learn continuous cardiac dynamics without architectural factorization. We propose CardioDiT, a fully 4D latent diffusion framework for short-axis cine CMR synthesis based on diffusion transformers. A spatiotemporal VQ-VAE encodes 2D+t slices into compact latents, which a diffusion transformer then models jointly as complete 3D+t volumes, coupling space and time throughout the generative process. We evaluate CardioDiT on public CMR datasets and a larger private cohort, comparing it to baselines with progressively stronger spatiotemporal coupling. Results show improved inter-slice consistency, temporally coherent motion, and realistic cardiac function distributions, suggesting that explicit 4D modeling with a diffusion transformer provides a principled foundation for spatiotemporal cardiac image synthesis. Code and models trained on public data are available at https://github.com/Cardio-AI/cardiodit.

Abstract

Explore similar work

CardioDiT: Latent Diffusion Transformers for 4D Cardiac MRI Synthesis

One Frame, Full Heartbeat: ECG-Free Cardiac Cine MRI Synthesis via Phase-Conditioned Flow Matching

Temporally Consistent and Controllable Video Generation of 2D Cine CMR via Latent Space Motion Modeling