Semantic Latent Alignment for Flow Matching with Invertible Bi-Lipschitz Adapters

Felix Nützel; Mischa Dombrowski; Luca Hagen; Bernhard Kainz

cs.CVOct 15, 2026

Semantic Latent Alignment for Flow Matching with Invertible Bi-Lipschitz Adapters

Authors: Felix Nützel, Mischa Dombrowski, Luca Hagen, Bernhard Kainz

Organizations: Friedrich-Alexander-Universität Erlangen-Nürnberg · Imperial College London

Abstract

Pretrained latent spaces trade compression, reconstruction, and the learnability of the coordinates a downstream generator has to work in, which complicates adapting these spaces to different domains or tasks: continued training is expensive, and without a new alignment objective it often fails to improve downstream generation. We instead leave the encoder and decoder fixed and change only the coordinates between them, using an invertible bi-Lipschitz adapter aligned offline to a semantic teacher and frozen during class-conditional flow matching. Invertibility preserves compatibility with the pretrained autoencoder, while the bi-Lipschitz constraint limits how far alignment can distort its latent geometry. This provides a simple way to improve semantic organization while retaining the compression and reconstruction properties of the original latent space. On two medical imaging datasets, constrained latent adaptation improves feature metrics, seed stability and counterfactual edit strength, while remaining competitive on downstream classification. In contrast, unconstrained adaptation can become highly anisotropic and sensitive despite remaining invertible, highlighting the importance of controlling geometric distortion. Our results suggest that carefully constrained post-hoc adaptation offers a parameter-efficient and tuning-free alternative to jointly modifying the representation during generative training.

Figures & tables

Figure 1: Semantic latent adaptation. Top: an invertible adapter

g_{\theta}

is trained offline on the cached latents

z

of a frozen autoencoder, maximizing the linear CKA between the similarity structure of the adapted latents

g_{\theta}(z)

and that of a frozen semantic teacher

\phi(x)

. The bi-Lipschitz constraint bounds how far this alignment can distort the original latent geometry, and

g_{\theta}^{-1}

returns exactly to the coordinates the decoder expects. Bottom: the adapter is then frozen, and flow matching is trained and sampled in the adapted space. The whole noise path is pushed through

g_{\theta}

, so the source distribution becomes

g_{\theta\#}\mathcal{N}(0,I)

, and generated latents are mapped back by

g_{\theta}^{-1}

before decoding. We evaluate the generators by training classifiers on synthetic data and by counterfactual edits that add or remove a finding.

mimic-cxr CXR-FID
	FID $\downarrow$		IRS $\uparrow$		TSTR
Method	domain	DINOv2	domain	DINOv2	bAcc $\uparrow$	macro $F_{1}$ $\uparrow$
No adaptation	$20.16\,{\scriptstyle\pm 7.50}$	$28.09\,{\scriptstyle\pm 13.74}$	$0.315\,{\scriptstyle\pm.103}$	$0.412\,{\scriptstyle\pm.115}$	$0.557\,{\scriptstyle\pm.021}$	$0.168\,{\scriptstyle\pm.046}$
REPA ( Yu et al., 2025 )	$19.34\,{\scriptstyle\pm 2.90}$	$26.13\,{\scriptstyle\pm 9.16}$	$0.331\,{\scriptstyle\pm.060}$	$0.430\,{\scriptstyle\pm.111}$	$\mathbf{0.590}\,{\scriptstyle\pm.012}$	$\mathbf{0.233}\,{\scriptstyle\pm.010}$
VA-VAE ( Yao et al., 2025 )	$\mathbf{15.05}\,{\scriptstyle\pm 2.92}$	$\underline{23.56}\,{\scriptstyle\pm 9.07}$	$\mathbf{0.414}\,{\scriptstyle\pm.063}$	$\underline{0.463}\,{\scriptstyle\pm.089}$	$0.563\,{\scriptstyle\pm.024}$	$0.179\,{\scriptstyle\pm.032}$
ReaLS ( Xu et al., 2025 )	$19.05\,{\scriptstyle\pm 1.35}$	$185.59\,{\scriptstyle\pm 6.90}$	$0.207\,{\scriptstyle\pm.018}$	$0.121\,{\scriptstyle\pm.002}$	$0.544\,{\scriptstyle\pm.005}$	$0.094\,{\scriptstyle\pm.014}$

Table 1: Generation quality and train-on-synthetic (TSTR) utility. Domain FID uses CheXNet features against the training set on MIMIC-CXR and a CT-trained DenseNet-121 against the test set on CT-RATE, and the DINOv2 columns use ViT-B/14 features. IRS is the image retrieval score normalized by that of the real test set. TSTR trains a DenseNet-121 on

50{,}000

class-balanced synthetic images and evaluates it on the real test set. Mean and sample SD over three flow-matching seeds, with generation and classifier seeds averaged within each flow-matching seed; best in bold, second best underlined.

Table 5

Table 6

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	Embedding	Label CKA	$\times$ pixels	$\times$ latents	Retrieval $\uparrow$	Confound $\downarrow$
mimic	Pixels ( $32\times 32$ )	$0.023$	$1.00$	$0.71$	$0.076$	$0.002$
	MedSigLIP ( Sellergren et al., 2026 ) , pooled	$0.121$	$5.35$	$3.78$	$0.163$	$0.016$
	MedSigLIP ( Sellergren et al., 2026 ) , image embedding	$0.182$	$8.01$	$5.66$	$0.209$	$0.017$
	MedImageInsight ( Codella et al., 2024 ) , pooled	$0.192$	$8.46$	$5.97$	$0.279$	$0.017$
	MedImageInsight ( Codella et al., 2024 ) , image embedding	$\mathbf{0.198}$	$\mathbf{8.75}$	$\mathbf{6.18}$	$\mathbf{0.281}$	$0.016$
ct-rate	Pixels ( $32\times 32$ )	$0.079$	$1.00$	$0.99$	$0.130$	$0.232$

Appendix

Table 9: Teacher selection on

8{,}000

TRAIN images per dataset, shaded rows selected. Label CKA is linear CKA against the one-hot labels; the two ratios divide it by that of a

32\times 32

pixel thumbnail and of the flattened FLUX.2 latents. Retrieval is the same-class rate at

10

neighbours excluding the same patient or volume, and confound is the same-patient or same-volume rate at

10

. The CT-native encoders Curia and FlexiCT-2D provide no image-text projection, so their class token stands in for the image embedding.

Figure 4: Uncurated CT-RATE samples, flow-matching seed $0$ , generation seed $0$ , for all fifteen classes: No Finding, six non-focal findings (bronchial wall thickening through other non-focal), and eight focal findings (linear opacities through other focal). Slices are shown with the posterior side down. Each column is one class; all generated entries of a column start from the same initial noise.

Figure 5: Uncurated MIMIC-CXR samples, flow-matching seed $0$ , generation seed $0$ . Each column is one class; all generated entries of a column start from the same initial noise.

Latent Space Alignment

Latent space alignment maps learned representations into compatible coordinates to enable comparison, transfer, and interoperability across models or data sources.

See all 76 papers

Aligning Latent Geometry for Spherical Flow Matching in Image Generation
May 14, 2026Tuna Han Salih Meral, Kaan Oktay, Hidir Yesiltepe +2Flow Matching Neural Representation Geometry
Virginia Tech · fal
Exploring and Exploiting Stability in Latent Flow Matching
May 8, 2026Rania Briq, Michael Kamp, Ohad Fried +2Flow Matching Efficient Inference
Forschungszentrum Jülich · Technical University Dortmund · Lamarr Institute +4
Geometry Preserving Loss Functions Promote Improved Adaptation of Blackbox Generative Model
Apr 26, 2026Sinjini Mitra, Constantine Kyriakakis, Shenyuan Liang +2Domain Adaptation Latent Space Optimization
1Geometric Media Lab, Arizona State University, Tempe 85281, USA. · Department of Statistics, Florida State University, Tallahassee 32306, USA.

May 14, 2026J/K move · Enter open · S save

Aligning Latent Geometry for Spherical Flow Matching in Image Generation

Tuna Han Salih Meral, Kaan Oktay, Hidir Yesiltepe, Adil Kaan Akan+1

Virginia Tech · fal

Flow Matching Neural Representation Geometry Conditional Image Generation Image Generation Flow-Based Generative Modeling Latent Space Geometry

Open paper AskPDF arXiv

Latent flow matching for image generation usually transports Gaussian noise to variational autoencoder latents along linear paths. Both endpoints, however, concentrate in thin spherical shells, and a Euclidean chord leaves those shells even when preprocessing aligns their radii. By decomposing each latent token into radial and angular components, we show through component-swap probes that decoded perceptual and semantic content is carried predominantly by direction, with radius contributing much less. We therefore project data latents onto a fixed token radius, use the radial projection of Gaussian noise as the spherical prior, finetune the decoder with the encoder frozen, and replace linear interpolation with spherical linear interpolation. The resulting geodesic paths stay on the sphere at every timestep, and their velocity targets are purely angular by construction. Under matched training, the method consistently improves class-conditional ImageNet-256 FID across different image tokenizers, leaves the diffusion architecture unchanged, and requires no auxiliary encoder or representation-alignment objective.

Abstract

Latent Space Alignment

Explore similar work

Aligning Latent Geometry for Spherical Flow Matching in Image Generation

Exploring and Exploiting Stability in Latent Flow Matching

Geometry Preserving Loss Functions Promote Improved Adaptation of Blackbox Generative Model