Latent Space Is Not Flat: Rethinking Latent Structure for 3D Medical Image Synthesis

Haowen Xue; Hao Chen; Hexuan Hu; Qian Huang; Yi Han; Qing Meng; Zaipeng Xie; Chao Li

cs.CVSep 26, 2026

Latent Space Is Not Flat: Rethinking Latent Structure for 3D Medical Image Synthesis

Authors: Haowen Xue, Hao Chen, Hexuan Hu, Qian Huang, Yi Han, Qing Meng, Zaipeng Xie, Chao Li,

Organizations: Hohai University · University of Cambridge · National University of Defense Technology

Abstract

Latent generative models make 3D medical image synthesis computationally practical by generating in a compressed space. However, we show that the common flat Euclidean assumption induced by $\ell_2$ objectives is imprecise: latent-space geometry is so strongly anisotropic that equal-magnitude errors can produce drastically different decoded distortions. We further find that this anisotropy has a clear feature: sensitive variation concentrates in a low-rank subspace. The dominant low-rank components capture the overall structure, encoding long-range, spatially coordinated variation while remaining resistant to local noise. Its orthogonal residual, in contrast, mainly captures local and image-specific variation. Motivated by this asymmetry, we introduce Latent Structure Flow (LSF). At each block, LSF decomposes the latent state into structure and residual, models structural changes with global context, and predicts residual variation locally while preserving a direct path for the input structure. LSF changes only the generator, leaving the frozen codec and pointwise training objective unchanged. Across cross-modality synthesis and tumor inpainting tasks, LSF outperforms all compared baselines on both global and tumor-specific metrics, demonstrating the benefit of explicitly modeling latent-space structure for 3D medical image synthesis.

Figures & tables

Figure 1: Decoded effect of latent errors of equal norm on one subject: slice with PSNR, lesion crop, and absolute error with MAE. Attenuating the structure causes the largest distortion; residual errors change only local appearance.

Figure 2: Latent Structure Flow. In every block, a basis estimated jointly from the state and the condition splits the input tokens into structure $S_{w}$ and residual $R_{w}$ (Eq. ( 3 )). $S_{w}$ is routed to the global branch and $R_{w}$ to the local branch; the output adds the predicted structural change and output residual to $S_{w}$ (Eq. ( 5 )).

Property	MAISI		KL-VAE		VQ-VAE
Property	$S$	$R$	$S$	$R$	$S$	$R$
(A) Attenuation PSNR (dB)	$24.6_{\pm 0.9}$	$33.1_{\pm 1.3}$	$31.8_{\pm 2.3}$	$36.3_{\pm 1.2}$	$29.6_{\pm 2.3}$	$29.5_{\pm 1.8}$
(A) Noise PSNR (dB)	$35.4_{\pm 1.0}$	$30.5_{\pm 1.4}$	$37.9_{\pm 0.9}$	$33.4_{\pm 1.6}$	$33.6_{\pm 1.9}$	$30.5_{\pm 2.1}$
(B) Smooth error (%)	$58_{\pm 0.8}$	$42_{\pm 0.8}$	$80_{\pm 1.5}$	$20_{\pm 1.5}$	$76_{\pm 4.6}$	$24_{\pm 4.6}$
(B) Gaussian error (%)	$34_{\pm 0.1}$	$66_{\pm 0.1}$	$34_{\pm 0.3}$	$66_{\pm 0.3}$	$33_{\pm 0.3}$	$67_{\pm 0.3}$
(C) Within patient	$0.87_{\pm 0.01}$	$0.25_{\pm 0.06}$	$0.66_{\pm 0.02}$	$0.18_{\pm 0.07}$	$0.84_{\pm 0.01}$	$0.22_{\pm 0.12}$

Table 1: Latent

S

vs.

R

error modes . Perturbations have equal latent norm. (A) Attenuating

S

hurts most. (B) Over-smoothed error falls mostly in

S

; Gaussian error does not. (C)

S

is shared within and across patients,

R

is not. (D)

S

is long-range,

R

is local. (E) One global basis captures

S

but not

R

Figure 3: Choosing the rank. $S$ and $R$ are decoded separately for $r\in\{2,4,8\}$ ; each panel gives its share of the latent energy. The residual retains 30–32% of the latent energy at $r{=}2$ , but only 2–3% at $r{=}8$ , motivating our choice of $r{=}2$ .

	T1n $\to$ T1c					T2w $\to$ T2f
Method	MAE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	Tumor $\downarrow$	Change $\downarrow$	MAE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	Tumor $\downarrow$	Change $\downarrow$
cWDM [ 11 ]	$1.59_{\pm 0.60}$	$26.21_{\pm 2.21}$	$0.921_{\pm 0.02}$	$1.365_{\pm 0.50}$	$1.410_{\pm 0.72}$	$1.57_{\pm 0.46}$	$26.14_{\pm 1.84}$	$0.923_{\pm 0.01}$	$1.348_{\pm 0.48}$	$1.451_{\pm 0.50}$
PMRF [ 19 ]	$1.64_{\pm 0.63}$	$26.14_{\pm 2.31}$	$0.918_{\pm 0.01}$	$1.376_{\pm 0.55}$	$1.616_{\pm 0.82}$	$1.59_{\pm 0.43}$	$26.18_{\pm 1.80}$	$0.916_{\pm 0.01}$	$1.359_{\pm 0.45}$	$1.506_{\pm 0.48}$
MoTFM [ 20 ]	$2.06_{\pm 0.64}$	$24.72_{\pm 2.73}$	$0.901_{\pm 0.07}$	$1.851_{\pm 0.72}$	$1.699_{\pm 0.82}$	$1.70_{\pm 0.54}$	$25.05_{\pm 1.88}$	$0.904_{\pm 0.02}$	$1.354_{\pm 0.45}$	$1.566_{\pm 0.54}$
WFM [ 21 ]	$1.88_{\pm 0.65}$	$24.86_{\pm 2.42}$	$0.904_{\pm 0.03}$	$1.393_{\pm 0.63}$	$1.605_{\pm 0.82}$	$1.63_{\pm 0.49}$	$25.12_{\pm 1.64}$	$0.908_{\pm 0.02}$	$1.383_{\pm 0.47}$	$1.522_{\pm 0.53}$
LSF (ours)	$\mathbf{1.56}_{\pm 0.59}$	$\mathbf{26.29}_{\pm 2.15}$	$\mathbf{0.927}_{\pm 0.01}$	$\mathbf{1.341}_{\pm 0.59}$	$\mathbf{1.357}_{\pm 0.71}$	$\mathbf{1.54}_{\pm 0.40}$	$\mathbf{26.44}_{\pm 1.92}$	$\mathbf{0.939}_{\pm 0.01}$	$\mathbf{1.324}_{\pm 0.45}$	$\mathbf{1.424}_{\pm 0.43}$

Table 2: Cross-contrast translation (mean ±SD ; best in bold). MAE, PSNR and SSIM over the brain (

\times 10^{2}

for MAE); Tumor and Change: MAE in the tumor and in the change region (

\times 10

). Baselines operate in pixel space.

	T1c		T1n		T2f		T2w		Average
Method	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$
LeFusion [ 22 ]	$24.04_{\pm 3.52}$	$0.702_{\pm 0.09}$	$25.50_{\pm 3.75}$	$0.695_{\pm 0.11}$	$18.35_{\pm 4.22}$	$0.547_{\pm 0.12}$	$19.96_{\pm 4.30}$	$0.590_{\pm 0.11}$	$21.96$	$0.633$
DiffTumor [ 23 ]	$25.59_{\pm 2.73}$	$0.738_{\pm 0.07}$	$26.04_{\pm 3.44}$	$0.704_{\pm 0.10}$	$21.12_{\pm 2.85}$	$0.608_{\pm 0.09}$	$22.05_{\pm 3.03}$	$0.636_{\pm 0.08}$	$23.70$	$0.671$
cWDM [ 11 ]	$26.33_{\pm 2.83}$	$0.774_{\pm 0.06}$	$26.28_{\pm 3.14}$	$0.711_{\pm 0.09}$	$20.77_{\pm 3.29}$	$0.598_{\pm 0.10}$	$22.01_{\pm 2.70}$	$0.636_{\pm 0.08}$	$23.85$	$0.680$
LSF (ours)	$\mathbf{26.53}_{\pm 2.57}$	$\mathbf{0.796}_{\pm 0.07}$	$\mathbf{27.00}_{\pm 3.26}$	$\mathbf{0.728}_{\pm 0.10}$	$\mathbf{22.91}_{\pm 2.90}$	$\mathbf{0.672}_{\pm 0.08}$	$\mathbf{24.10}_{\pm 2.66}$	$\mathbf{0.720}_{\pm 0.08}$	$\mathbf{25.14}$	$\mathbf{0.729}$

Table 3: Tumor inpainting (mean ±SD ; best in bold). PSNR (dB) and SSIM inside the whole-tumor region.

Figure 4: Cross-contrast translation cases, T1n $\to$ T1c (top) and T2w $\to$ T2f (bottom): slice, tumor-centered crop (yellow box) and absolute error, with the ground-truth tumor contour in cyan. Annotations give full-volume PSNR and tumor MAE.

	Configuration	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$\rho_{S}\downarrow$
(a)	Global branch only	23.27	0.672	1.55	12.6
(b)	+ local branch	24.29	0.703	1.52	11.5
(c)	+ split: $S\!\to$ global, $R\!\to$ local	24.51	0.712	1.50	8.7
(d)	+ input-token structure $S$ (LSF)	25.14	0.729	1.44	8.0
(e)	(d) with $S\!\to$ local, $R\!\to$ global	23.90	0.697	1.60	10.9
(f)	cWDM [ 11 ] , pixel space	23.85	0.680	1.48	11.8

Table 4: Ablation on tumor inpainting. All metrics are computed in the tumor region; LPIPS

\times 10^{-3}

;

\rho_{S}

in % with

r{=}2

(Gaussian error: 6.25%). cWDM is the strongest pixel-space baseline.

Latent Space Geometry

See all 29 papers

VoxStruct3D: Structure-Leading Flow Matching for Voxel-Space 3D MRI Synthesis
Aug 5, 2026Fang Li, Yang Gao, Shihao Zou +5Flow Matching 3D Medical Imaging
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Faculty of Computer Science and Control Engineering, Shenzhen University of Advanced Technology
Recover Semantics First, Generate Better: Improved Latent Modeling for 3D MRI Reconstruction and Cross-Contrast Synthesis
Jun 16, 2026Yonghao Chen, Sicheng Yang, Rui Tang +13D Medical Imaging Magnetic Resonance Imaging
The Hong Kong University of Science and Technology (Guangzhou) · Xi’an Jiaotong University
LiFT: Lifted Inter-slice Feature Trajectories for 3D Image Generation from 2D Generators
May 18, 2026Xinhe Zhang, Yuyang Zhang, Pengfei Jin +3Image-to-Image Translation Medical Image Synthesis
∗School of Engineering and Applied Sciences, Harvard University · ‡Kempner Institute, Harvard University · †Center for Advanced Medical Computing and Analysis, Massachusetts General Hospital and Harvard Medical School

Aug 5, 2026J/K move · Enter open · S save

VoxStruct3D: Structure-Leading Flow Matching for Voxel-Space 3D MRI Synthesis

Fang Li, Yang Gao, Shihao Zou, Weixin Si+4

State Key Laboratory of Virtual Reality Technology and Systems, Beihang University · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Faculty of Computer Science and Control Engineering, Shenzhen University of Advanced Technology

Flow Matching 3D Medical Imaging Medical Image Synthesis

Open paper AskPDF arXiv

High-fidelity 3D MRI synthesis requires both globally coherent anatomy and fine-grained voxel-level detail. Although latent diffusion makes volumetric generation tractable, its image autoencoder introduces a reconstruction bottleneck that can limit the fine detail recoverable in the final volume. We present VoxStruct3D, a voxel-space flow-matching framework that directly models full-resolution MRI volumes using a clean-data prediction objective. Its Volumetric Voxel Generator (VVG) combines factorized 3D patch embedding with overlapping upsampling, time-modulated residual refinement, and skip fusion, enabling neighboring tokens to jointly reconstruct shared voxel regions and suppress patch-boundary artifacts. To complement direct voxel-space modeling with an explicit anatomical prior, we further introduce a Structure-First, Image-Follows (SFIF) strategy. A frozen pretrained 3D medical encoder and a StructVAE extract compact structure tokens that preserve dominant anatomy, while a structure-leading schedule keeps their trajectory ahead of the image trajectory. Patch-Aligned RoPE spatially aligns the unequal token grids, and asymmetric attention enforces one-way guidance from structure to image. Experiments on pathological and healthy T1-weighted brain MRI datasets show that VoxStruct3D achieves the strongest overall performance across feature-distribution alignment, sample diversity, and perceptual quality, producing anatomically coherent and visually realistic volumes.

Abstract

Latent Space Geometry

Explore similar work

VoxStruct3D: Structure-Leading Flow Matching for Voxel-Space 3D MRI Synthesis

Recover Semantics First, Generate Better: Improved Latent Modeling for 3D MRI Reconstruction and Cross-Contrast Synthesis

LiFT: Lifted Inter-slice Feature Trajectories for 3D Image Generation from 2D Generators