Drift Variation Autoencoder: Unifying Generation and Representation Learning through Conditional Posterior Flow Matching

Jiarui Cao

cs.LGAug 25, 2026

Drift Variation Autoencoder: Unifying Generation and Representation Learning through Conditional Posterior Flow Matching

Authors: Jiarui Cao

Organizations: The Chinese University of Hong Kong

Abstract

Stochastic masking, cropping, or modality removal makes deterministic reconstruction an incomplete target: one observation can admit many clean completions. This work takes the corresponding posterior $P(X\mid C)$ as the common statistical object for conditional generation and generatively sufficient representation learning. Drift Variation autoencoder trains a masked encoder $Z=E(C)$ and a conditional flow decoder with one clean-prediction Flow Matching loss. The analysis first decomposes the ideal conditional KL into generator approximation and the representation deficiency $I(X;C\mid Z)$ . It then derives orthogonal risk decompositions for conditional Flow Matching. For an affine Gaussian path, the clean-prediction representation gap is zero if and only if $P(X\mid Z)=P(X\mid C)$ . Thus the encoder-dependent excess clean-prediction risk induced by Flow Matching and the profiled ideal conditional KL have the same posterior-sufficient zero set, without being numerically equal objectives. An exact conditional field with a zero-noise endpoint then generates $P(X\mid Z)$ and hence $P(X\mid C)$ at a joint ideal optimum. The result extends to continuous multimodal product spaces when the complete modality tuple remains the Flow target for every observation mask. On CrossGeom-4, an 18-run controlled benchmark, observable factors have linear-probe $R^2$ of $0.9990$ - $0.9992$ , shuffling the joint model's encoder condition increases conditional error by $13.5\times$ - $15.7\times$ , and joint target attention reduces disagreement on an unobserved factor shared by two outputs by $90.1$ - $92.8\%$ relative to independent target decoders. Visible modalities are also generated and reconstructed, directly validating the full-tuple objective. Unconditional mode balance remains imperfect, delimiting the empirical claim to a controlled multimodal proof of concept.

Figures & tables

Input: encoder $E_{\theta}$ , clean predictor $D_{\phi}$ , augmentation law $A$ , time law $\rho$ , weight $w$
1	Sample a clean example $X$ and augmentation randomness, then form $C=A(X,\Xi_{A})$ .
2	Encode only the observation: $Z=E_{\theta}(C)$ .
3	Sample $X_{0}\sim\mathcal{N}(0,s_{0}^{2}I)$ independently and $t\sim\rho$ .
4	Form $\sigma_{t}=1-(1-\sigma_{\min})t$ and $X_{t}=tX+\sigma_{t}X_{0}$ .
5	Predict $\widehat{X}=D_{\phi}(X_{t},t,Z)$ and compute $w(t)\left\lVert\widehat{X}-X\right\rVert^{2}$ .
6	Backpropagate jointly through $D_{\phi}$ and $E_{\theta}$ ; no stopped target or external teacher is required by the theory.

Table 1: Drift Variation autoencoder training with an affine Gaussian path. Every prose step denotes a minibatch operation.

	Single-input conditional MAE		Unknown target-target MAE			Unknown-factor W1			Uncond. joint
Level	Ind.	Joint	Ind.	Joint	Reduction	Ind.	Joint	Reduction	reduction
L1	.1166	.1176	1.6397	.1186	92.8%	.6776	.2067	69.5%	88.8%
L2	.1145	.1154	1.6278	.1201	92.6%	.6878	.2669	61.2%	89.4%
L3	.1310	.1338	1.5651	.1542	90.1%	.7143	.1648	76.9%	86.6%

Table 2: Joint coupling under full-tuple Flow Matching. Conditional MAE is similar for matched independent (Ind.) and joint decoders. Joint target attention specifically reduces disagreement and distribution error for the unknown factor shared by two outputs. Values are three-seed means; complete standard deviations appear in Appendix G .

Figure 2: Known-factor conditional MAE, disagreement on the jointly sampled unknown factor, and unconditional TV-to-uniform. Joint and independent decoders have similar conditional accuracy, but only joint target attention coordinates residual uncertainty. Error bars are standard deviations across the three seeds, computed with denominator

n

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Required report
Observation	Augmentations, strengths, mask law, content dependence, and all declared side information.
Representation	Encoder/token architecture, pooling, frozen and nonlinear probes, rank statistics, and checkpoint rule.
Flow	Source, schedules, time law, weighting/clipping, endpoint noise, solver, steps, and final-time handling.
Generation	Fidelity, precision/recall, condition consistency, repeated-sample diversity, and sample count.
Multimodal	Visible subsets, full generated targets, per-modality fidelity, visible reconstruction, and joint consistency.
Ablations	Shuffled/zero condition, source construction, time allocation, bottleneck size, and joint versus independent targets.

Appendix

Table 3: Minimum information for evaluating a Drift Variation autoencoder instance.

Level	Decoder	Uncond. TV $\downarrow$	Uncond. joint MAE $\downarrow$	A $\to$ BC $\downarrow$	B $\to$ AC $\downarrow$	C $\to$ AB $\downarrow$	AB $\to$ C $\downarrow$	AC $\to$ B $\downarrow$	BC $\to$ A $\downarrow$
1	independent	0.0892 $\pm$ 0.0111	1.6462 $\pm$ 0.0460	0.1147 $\pm$ 0.0044	0.1177 $\pm$ 0.0072	0.1173 $\pm$ 0.0113	0.0962 $\pm$ 0.0029	0.0950 $\pm$ 0.0081	0.0943 $\pm$ 0.0006
1	joint	0.0990 $\pm$ 0.0128	0.1847 $\pm$ 0.0096	0.1123 $\pm$ 0.0093	0.1212 $\pm$ 0.0076	0.1193 $\pm$ 0.0064	0.0990 $\pm$ 0.0124	0.1011 $\pm$ 0.0136	0.0966 $\pm$ 0.0096
2	independent	0.0724 $\pm$ 0.0022	1.6674 $\pm$ 0.0426	0.1197 $\pm$ 0.0132	0.1132 $\pm$ 0.0130	0.1105 $\pm$ 0.0092	0.0975 $\pm$ 0.0153	0.0952 $\pm$ 0.0112	0.0831 $\pm$ 0.0116
2	joint	0.1082 $\pm$ 0.0186	0.1771 $\pm$ 0.0116	0.1055 $\pm$ 0.0040	0.1204 $\pm$ 0.0118	0.1203 $\pm$ 0.0119	0.0923 $\pm$ 0.0026	0.0968 $\pm$ 0.0009	0.1061 $\pm$ 0.0202
3	independent	0.0542 $\pm$ 0.0132	1.5977 $\pm$ 0.0220	0.1260 $\pm$ 0.0018	0.1543 $\pm$ 0.0112	0.1126 $\pm$ 0.0064	0.1150 $\pm$ 0.0083	0.0952 $\pm$ 0.0006	0.1072 $\pm$ 0.0090
3	joint	0.0861 $\pm$ 0.0172	0.2133 $\pm$ 0.0164	0.1295 $\pm$ 0.0178	0.1583 $\pm$ 0.0143	0.1137 $\pm$ 0.0022	0.1196 $\pm$ 0.0156	0.1196 $\pm$ 0.0200	0.1127 $\pm$ 0.0079

Appendix

Table 4: Conditional factor MAE and unconditional metrics under full-tuple Flow Matching. Every conditional query generates A, B, and C; direction labels identify the visible condition, not the only active targets.

Level	Decoder	Probe $R^{2}\uparrow$	Random R 2	Visible recon. $\downarrow$	Single cond. $\downarrow$	Double cond. $\downarrow$	Unknown joint $\downarrow$	Unknown W1 $\downarrow$	Both modes $\uparrow$	Shuffle $\times\uparrow$
1	independent	0.9991 $\pm$ 0.0001	0.8227 $\pm$ 0.0858	0.0840 $\pm$ 0.0038	0.1166 $\pm$ 0.0059	0.0951 $\pm$ 0.0033	1.6397 $\pm$ 0.0432	0.6776 $\pm$ 0.0117	1.0000 $\pm$ 0.0000	15.6921 $\pm$ 0.6424
1	joint	0.9992 $\pm$ 0.0000	0.8227 $\pm$ 0.0858	0.0918 $\pm$ 0.0066	0.1176 $\pm$ 0.0076	0.0989 $\pm$ 0.0114	0.1186 $\pm$ 0.0099	0.2067 $\pm$ 0.0539	1.0000 $\pm$ 0.0000	15.2957 $\pm$ 1.5546
2	independent	0.9992 $\pm$ 0.0000	-13.2173 $\pm$ 17.3786	0.0809 $\pm$ 0.0089	0.1145 $\pm$ 0.0104	0.0919 $\pm$ 0.0117	1.6278 $\pm$ 0.0313	0.6878 $\pm$ 0.0109	1.0000 $\pm$ 0.0000	16.4248 $\pm$ 1.9564
2	joint	0.9992 $\pm$ 0.0000	-13.2173 $\pm$ 17.3786	0.0912 $\pm$ 0.0069	0.1154 $\pm$ 0.0089	0.0984 $\pm$ 0.0078	0.1201 $\pm$ 0.0107	0.2669 $\pm$ 0.0309	1.0000 $\pm$ 0.0000	15.6814 $\pm$ 0.9940
3	independent	0.9992 $\pm$ 0.0000	0.7396 $\pm$ 0.2158	0.0910 $\pm$ 0.0050	0.1310 $\pm$ 0.0060	0.1058 $\pm$ 0.0054	1.5651 $\pm$ 0.0261	0.7143 $\pm$ 0.0133	0.9991 $\pm$ 0.0012	14.1745 $\pm$ 0.6261
3	joint	0.9990 $\pm$ 0.0001	0.7396 $\pm$ 0.2158	0.1041 $\pm$ 0.0069	0.1338 $\pm$ 0.0099	0.1173 $\pm$ 0.0122	0.1542 $\pm$ 0.0080	0.1648 $\pm$ 0.0516	1.0000 $\pm$ 0.0000	13.5004 $\pm$ 1.1064

Appendix

Table 5: Representation, visible reconstruction, uncertainty, and condition-use evidence. “Both modes” is the fraction of contexts whose 16 samples contain both signs of the shared unknown factor. “Shuffle” is shuffled-condition MAE divided by matched-condition MAE with identical initial ODE noise.

Conditional Generative Modeling

Conditional generative modeling learns to produce data distributions steered by labels, inputs, text, or physical and structural constraints.

See all 387 papers

Oct 7, 2026

MUNITE: Unified Multimodal Latent Inference for Any-to-Any Multimodal Generation

We introduce MUNITE, a latent-variable framework for flexible any-to-any multimodal generation that treats encoding and latent generation as the same inference problem under different amounts of observed evidence. Given any subset of modalities, MUNITE models the conditional distribution over the latent representation associated with the complete observation. Full observation recovers deterministic encoding, no observation recovers the latent marginal, and intermediate subsets define conditional latent inference, all within a single conditional flow model. A shared latent sample captures variation that must remain consistent across generated targets, while modality-specific generative decoders model the remaining uncertainty independently. To learn these conditional distributions from incomplete training examples, we extend conditional flow matching through self-distillation: predictions conditioned on richer available observations supervise the same model conditioned on smaller subsets at the same intermediate latent state. When the richer-evidence trajectory follows the exact conditional flow, this provides the same expected learning signal as full-target denoising. Across PolyMNIST-D-Q, FFHQ64, and image-text-audio, MUNITE achieves competitive or better generation quality and source-target alignment, with higher joint-generation coherence. In particular, it attains the highest coherence in all one-to-many and unconditional image-text-audio comparisons, showing the effectiveness of unified latent inference across diverse multimodal settings.

Kyeongmin Yeo, Minhyuk Sung

KAIST

Multimodal Generation Unified Multimodal Models Conditional Flow Matching

Ask PDF

Sep 26, 2026

From Feed-Forward to Flow: Unifying Reconstruction and Generation Is Easier Than You Think

Reconstruct where the images provide evidence, and generate where they do not: recent success of spatial world models such as Atlas (World Labs Team, 2026) highlights the value of unifying reconstruction and generation in one model. Yet the two have long lived in separate paradigms with distinctive failure modes: feed-forward reconstruction averages ambiguity into blur, while conditional generation invents plausible but scene-inconsistent detail. In this work, we present a unified flow-based formulation for reconstruction and generation, where a shared clean-target predictor performs direct reconstruction at its single-step endpoint and unfolds conditional generation through multi-step flow. A controlled toy study reveals the mechanism: with a single step, the predictor collapses to the conditional mean just like feed-forward methods, favoring consistency over diversity. With multi-step inference, the fidelity of generated details grows with context richness: closer observations reduce ambiguity and yield better-matched details. We further instantiate the formulation in appearance and geometry 3D tasks. JiT-LVSM improves perceptual and distributional quality in novel view synthesis, while JUSt3R retains competitive single-step geometry prediction with additional multi-step inference capabilities that reduces veil and flying-pixel artifacts, producing cleaner surface structure with greater test-time compute. Together, they show that reconstruction and generation can share both a formulation and a backbone, with their behavior governed by denoising configuration---making unification surprisingly simple.

Haoru Wang, Qianfan Shen, Kai Ye +2

Peking University

Flow-Based Generative Modeling Feed-Forward 3D Reconstruction Multi-View 3D Reconstruction Novel View Synthesis

Ask PDF

Sep 3, 2026

Discriminative Flow Matching: Beyond Time-Conditioning in Generative Restoration via Flow-State Representations

Existing Conditional Flow Matching (CFM) formulations describe transport progress using an explicit interpolation coordinate, commonly interpreted as time, assuming that a single global variable adequately represents a sample's position along the generative trajectory. In restoration tasks, however, transport progress is sample-dependent because the initial distribution may exhibit varying statistical dependencies with the target distribution. Thus, samples at the same interpolation coordinate can differ substantially in degradation level, distance to the target distribution, and restoration difficulty. We investigate whether signal representations learned by discriminatively trained models provide a meaningful description of generative transport state in CFM-based restoration. Through systematic latent-space analysis, we show that discriminative representations organize according to degradation severity and follow a consistent trajectory toward the clean-data manifold during generation. Motivated by these observations, we introduce the Discriminative Flow-State Hypothesis, which posits that discriminative representations encode a transport state governing generative restoration. Based on this hypothesis, we propose Discriminative Flow Matching, which conditions the Flow-Matching velocity field on Discriminative Flow-State Representations rather than explicit time coordinates. Experiments on speech enhancement and image denoising show that these representations characterize restoration progress, enable adaptive inference, and consistently outperform CFM and diffusion-related baselines. Our findings suggest that discriminative representations provide an effective state-aware alternative to explicit time conditioning and offer a novel perspective on the relationship between discriminative and CFM-based generative modeling.

Shrishti Saha Shetu, Emanuël A. P. Habets, Andreas Brendel

International Audio Laboratories Erlangen∗ Am Wolfsmantel 33, 91058 Erlangen, Germany · Fraunhofer IIS, Am Wolfsmantel 33, 91058 Erlangen, Germany

Flow Matching Conditional Generative Modeling Image Denoising

Ask PDF

Abstract

Conditional Generative Modeling

Explore similar work

MUNITE: Unified Multimodal Latent Inference for Any-to-Any Multimodal Generation

From Feed-Forward to Flow: Unifying Reconstruction and Generation Is Easier Than You Think

Discriminative Flow Matching: Beyond Time-Conditioning in Generative Restoration via Flow-State Representations