Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis

Hila Chefer; Patrick Esser; Dominik Lorenz; Dustin Podell; Vikash Raja; Vinh Tong; Antonio Torralba; Robin Rombach

cs.CVMar 6, 2026

Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis

Authors: Hila Chefer, Patrick Esser, Dominik Lorenz, Dustin Podell, Vikash Raja, Vinh Tong, Antonio Torralba, Robin Rombach

Organizations: Black Forest Labs · MIT

Abstract

Strong semantic representations improve the convergence and generation quality of diffusion and flow models. Existing approaches largely rely on external models, which require separate training, operate on misaligned objectives, and exhibit unexpected scaling behavior. We argue that this dependence arises from the model's training objective, which poses a denoising task with little incentive to learn semantic representations. We introduce Self-Flow: a self-supervised flow matching paradigm that integrates representation learning within the generative framework. Our key mechanism, Dual-Timestep Scheduling, applies heterogeneous noise levels across tokens, creating an information asymmetry that forces the model to infer missing information from corrupted inputs. This drives learning strong representations alongside generative capabilities without external supervision. Our method generalizes across modalities and enables multi-modal training while following expected scaling laws, achieving superior image, video, and audio generation.

Figures & tables

Panel of Figure 2 — Figure 2: Motivation. (a) Scaling DINO (v2-B $<$ v2-L $<$ v3-H+) paradoxically degrades the generations quality using REPA. (b) Diffusion forcing and full masking create a train-inference gap which degrades generations. Our Dual-Timestep Scheduling improves generation even without a self-supervised objective.

Figure 3: Illustration of our method. Given a clean input $x_{0}$ , we draw two timesteps $t,s$ , and a random mask $M$ , then noise each token according to its assigned timestep. The teacher input is noised with $\tau_{min}=\min\{t,s\}$ , creating an information asymmetry compared to the student. The student is trained to simultaneously denoise the input and reconstruct the teacher’s features given its mixed-noised view.

Without external representations
Model	Steps	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Pre. $\uparrow$	Rec. $\uparrow$
SiT-XL/2	7M	8.3	6.30	130.57	0.69	0.67
SRA	4M	7.27	5.87	143.06	0.69	0.68
Ours	4M	5.70	4.97	151.40	0.72	0.67
With external representations
REPA	4M	5.89	5.73	157.66	0.70	0.69

Table 1: Quantitative results on ImageNet 256

\times

256. Our method outperforms REPA, despite its use of DINOv2 which heavily relies on ImageNet for training. Our method is beneficial in combination with representation autoencoders.

Panel of Figure 4 — Figure 4: Autoencoder generalization and improved representations. (a) Our method improves training and generation in RAE ( Zheng et al., 2025 ) , demonstrating compatibility with semantic latent spaces. (b) Linear probing confirms that our method learns stronger representations than standard flow matching.

Table 5

Without external representations
Model	Steps	CLAP $\downarrow$	CLAP-M $\downarrow$	CLAP-A $\downarrow$
Vanilla Flow	350K	148.874	0.1695	0.1059
SRA	350K	147.215	0.1664	0.1034
Ours	350K	145.645	0.1634	0.1001
With external representations
w/ MERT	350K	148.883	0.1677	0.1040

Table 4: Quantitative results on audio generation. Our method achieves the best FAD scores across multiple CLAP variants.

Panel of Figure 5 — Figure 5: Quantitative results across modalities. Our method significantly outperforms all external and internal alignment methods across text-based image, video, and audio generation. Our method is the only one to outperform REPA on DINOv2 FD (despite REPA directly aligning with DINOv2). Arrows indicate whether lower ( $\downarrow$ ) or higher ( $\uparrow$ ) is better.

Panel of Figure 6 — Figure 6: Scaling behavior. As model size increases (290M → 420M → 625M → 1B parameters), the performance gap between our method and REPA widens (a). Notably, our 625M variant outperforms the 1B REPA model. Our method effectively leverages increased compute, while REPA shows diminishing returns (b).

Figure 7: SIMPLER simulation rollouts. Our method significantly boosts success rate for complex actions such as Move Near (top) and Open and Place (bottom).

Panel of Figure 8 — Figure 8: Multi-modal experiments. (a) We train a single model on three modalities with different weightings to control the trade-off between them. Ours provides consistent improvements (shaded area) across all settings. Axes are inverted so that larger area indicates better performance. (b) Success rates for joint Video-Action prediction. Early on (30k), Ours outperforms FM across all tasks and achieves success in all task categories, whereas FM fails entirely on Open and Place tasks. Later (100k), performance on single-object tasks (Pick Coke Can, Open/Close Drawer) converges, while Ours maintains a significant advantage on complex multi-object and sequential tasks (Move Near, Open and Place).

Figure 9: Typography comparison. Self-Flow significantly outperforms standard flow matching in accurate text rendering.

Panel of Figure 10 — Figure 10: Qualitative comparisons with vanilla flow matching and leading baselines: external (REPA) and internal (SRA).

Panel of Figure 11 — Figure 11: Ablations and Limitations. (a) Removing the self-supervised loss is most detrimental. Removing Dual-Timestep Scheduling or changing the scheduling of the second timestep ( $s$ ) significantly harms the results. (b) While better noise scheduling impacts both methods positively, our method benefits from it more.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 12 — Figure 12: Shifting function and distributions used for sampling timesteps during training and shifting grids for evaluations.

Figure 13: Text-to-image results for baseline selection. We compare SRA and LayerSync to select the leading baseline that does not employ an external encoder. We find that SRA outperforms LayerSync after 400K training steps, and attribute this to SRA’s use of an EMA teacher.

Model	Trainshift	Sampleshift	Masking Ratio	CLAP $\downarrow$	CLAP-M $\downarrow$	CLAP-A $\downarrow$
Vanilla Flow Matching	0.75	6.93	-	148.874	0.1695	0.1059
SRA	0.75	6.93	-	147.215	0.1664	0.1034
Ours	1.0	6.93	0.5	145.645	0.1634	0.1001
REPA MERT	0.75	6.93	-	148.883	0.1677	0.1040

Appendix

Table 5: Quantitative results on audio generation together with hyperparameters obtained as described in Sec. B .

Panel of Figure 14 — Figure 14: Results with early-stopping from all hyperparameter variants considered in the audio experiments. The best hyperparameter runs, as determined by the selection process described in Sec. B , are shown in bold. Our approach compares favorably against all other variants across hyperparameters.

Panel of Figure 15 — Figure 15: Experiments on training timestep behaviors for video training. Left two figures: Performance across different shift parameters in combination with a logit-normal distribution on a Vanilla Flow Matching model trained on WAN2.2 representations. Right two figures: Performance comparison of different approaches with and without the low SNR modification described in Sec. C .

Weights	FID $\downarrow$			FID (framewise) $\downarrow$			FVD $\downarrow$			FAD (CLAP) $\downarrow$			FAD (CLAP-A) $\downarrow$			FAD (CLAP-M) $\downarrow$
$(w_{I},w_{V},w_{A})$	FM	Ours	Rel. Chg.	FM	Ours	Rel. Chg.	FM	Ours	Rel. Chg.	FM	Ours	Rel. Chg.	FM	Ours	Rel. Chg.	FM	Ours	Rel. Chg.
(0.067, 0.319, 0.614)	4.04	3.69	-8.55%	12.2	11.1	-9.59%	72.9	66.3	-9.09%	153	149.8	-2.13%	0.115	0.114	-0.47%	0.178	0.177	-0.22%
(0.105, 0.506, 0.389)	3.51	3.25	-7.34%	10.9	10.2	-6.60%	64.8	61	-5.85%	155.4	154.2	-0.75%	0.12	0.119	-0.74%	0.183	0.182	-0.35%
(0.131, 0.628, 0.241)	3.35	3.19	-4.60%	10.3	9.82	-5.04%	62.7	58	-7.40%	160.1	157.4	-1.66%	0.127	0.127	-0.57%	0.19	0.189	-0.38%
(0.196, 0.079, 0.725)	3.07	2.99	-2.85%	18.1	17.7	-2.01%	116	112	-3.72%	153.9	149.1	-3.08%	0.115	0.112	-3.12%	0.178	0.176	-1.44%
(0.308, 0.123, 0.569)	2.83	2.74	-3.06%	15.6	14.8	-5.10%	101	96.7	-4.33%	156.4	152.6	-2.40%	0.119	0.117	-1.86%	0.182	0.181	-0.74%

Appendix

Table 6: Comparison between performance of Ours and Vanilla Flow Matching (FM) in multimodal runs. For each metric, we show the values achieved by FM and Ours as well as change of Ours relative to FM. All metrics improve across all weightings.

Panel of Figure 16 — Figure 16: Relationship between modality performance and weightings for multimodal runs. See Sec. D .

Panel of Figure 17 — Figure 17: Per-modality performance progress for multimodal runs. The benefits of our approach are present throughout the complete training process.

Panel of Figure 18 — Figure 18: Joint video-action prediction results on SIMPLER. Our self-supervised flow matching framework transfers effectively to embodied AI tasks, learning more efficiently from limited robotics data and showing particular advantages on complex manipulation tasks requiring multi-step reasoning.

Figure 19: Joint video-audio prediction given a conditioning image, initialized from either a video-only or multi-modal (mixed) checkpoint. While multi-modal initialization benefits both formulations, even video-only Self-Flow outperforms multi-modal flow matching (FM).

Panel of Figure 20 — Figure 20: Additional image results. (a) ImageNet convergence comparison. Our method outperforms REPA after $\sim$ 3M training steps, while the experiments in the main paper show an advantage to Self-Flow from the early training steps. This can be attributed to REPA’s alignment with DINOv2, which was heavily trained on ImageNet. (b) Self-Flow with semantic autoencoders. Our self-supervised framework provides consistent improvements even when applied over semantically structured latent spaces such as RAE ( Zheng et al., 2025 ) (on ImageNet, 4(a) ) and FLUX.2 ( Black Forest Labs, 2025 ) (for T2I, 20(b) ), demonstrating complementary benefits.

Panel of Figure 21 — Figure 21: Effect of layer selection. We vary the student layer $l$ (left) and teacher layer $k$ (right) independently, with the other fixed at its default value. Performance is stable near our default selection but degrades when layers are too shallow (weak semantic signal) or when distilling to overly deep student layers (interferes with generation).

Figure 22: Qualitative comparison (text-to-image). Our method produces superior structure, texture, and fine-grained details across prompts of varying complexity.

Figure 23: Qualitative comparison (text-to-image). Our method produces superior structure, texture, and fine-grained details across prompts of varying complexity.

Figure 24: Qualitative comparison (text-to-image). Our method produces superior structure, texture, and fine-grained details across prompts of varying complexity.

Figure 25: Qualitative comparison (text-to-image). Our method produces superior structure, texture, and fine-grained details across prompts of varying complexity.

Figure 26: Qualitative comparison (text-to-image). Our method produces superior structure, texture, and fine-grained details across prompts of varying complexity.

Figure 27: Qualitative comparison (text-to-video). Baselines exhibit structural and temporal artifacts, while our method produces coherent and temporally smooth results.

Figure 28: Qualitative comparison (text-to-video). Baselines exhibit structural and temporal artifacts, while our method produces coherent and temporally smooth results.

Figure 29: Qualitative comparison (text-to-video). Baselines exhibit structural and temporal artifacts, while our method produces coherent and temporally smooth results.

Figure 30: Qualitative comparison (text-to-video). Baselines exhibit structural and temporal artifacts, while our method produces coherent and temporally smooth results.

Figure 31: Qualitative comparison (text-to-video). Baselines exhibit structural and temporal artifacts, while our method produces coherent and temporally smooth results.

Figure 32: Improved typography in text-to-image generation. Results from our scaled 4B parameter multi-modal model after just 100K high-resolution finetuning steps. Our method produces accurate and legible text rendering compared to vanilla flow matching.

Figure 33: Improved typography in text-to-image generation. Results from our scaled 4B parameter multi-modal model after just 100K high-resolution finetuning steps. Our method produces accurate and legible text rendering compared to vanilla flow matching.

Figure 34: Improved typography in text-to-image generation. Results from our scaled 4B parameter multi-modal model after just 100K high-resolution finetuning steps. Our method produces accurate and legible text rendering compared to vanilla flow matching.

Flow-Based Generative Modeling

Flow-based generative modeling learns probability transports from simple noise to data for fast sampling, density estimation, and controllable generation.

See all 538 papers

Laplacian Multi-scale Flow Matching for Generative Modeling
Feb 23, 2026Zelin Zhao, Petr Molodyk, Haotian Xue +1Mixture-of-Transformers Generative Modeling
Georgia Institute of Technology Atlanta, GA 30332, USA
Representation by Design in Generation: Cross-View Class-Token Alignment in Diffusion Transformers
Sep 28, 2026Xiaoyu Wu, Yifei Wang, Chen WeiFlow Matching Diffusion Transformer
Carnegie Mellon University · Rice University
Dino Forcing Flow Models: Do not denoise what you can predict
Oct 8, 2026Arijit Ghosh, Lucas Degeorge, Paul Couairon +3Self-Conditioning Flow-Based Generative Modeling
ENPC, IP Paris · Ecole Polytechnique, IP Paris · AMIAD +1

Feb 23, 2026J/K move · Enter open · S save

Laplacian Multi-scale Flow Matching for Generative Modeling

Zelin Zhao, Petr Molodyk, Haotian Xue, Yongxin Chen

Georgia Institute of Technology Atlanta, GA 30332, USA

Mixture-of-Transformers Generative Modeling Flow-Based Generative Modeling Flow Matching

Open paper AskPDF arXiv

In this paper, we present Laplacian multiscale flow matching (LapFlow), a novel framework that enhances flow matching by leveraging multi-scale representations for image generative modeling. Our approach decomposes images into Laplacian pyramid residuals and processes different scales in parallel through a mixture-of-transformers (MoT) architecture with causal attention mechanisms. Unlike previous cascaded approaches that require explicit renoising between scales, our model generates multi-scale representations in parallel, eliminating the need for bridging processes. The proposed multi-scale architecture not only improves generation quality but also accelerates the sampling process and promotes scaling flow matching methods. Through extensive experimentation on CelebA-HQ and ImageNet, we demonstrate that our method achieves superior sample quality with fewer GFLOPs and faster inference compared to single-scale and multi-scale flow matching baselines. The proposed model scales effectively to high-resolution generation (up to 1024

\times

1024) while maintaining lower computational overhead.

Abstract

Flow-Based Generative Modeling

Explore similar work

Laplacian Multi-scale Flow Matching for Generative Modeling

Representation by Design in Generation: Cross-View Class-Token Alignment in Diffusion Transformers

Dino Forcing Flow Models: Do not denoise what you can predict