We consider deep multivariate models for heterogeneous collections of random variables. In the context of computer vision, such collections may e.g. consist of images, segmentations, image attributes, and latent variables. When developing such models, most existing works start from an application task and design the model components and their dependencies to meet the needs of the chosen task. This has the disadvantage of limiting the applicability of the resulting model for other downstream tasks. Here, instead, we propose to represent the joint probability distribution by means of conditional probability distributions for each group of variables conditioned on the rest. Such models can then be used for practically any possible downstream task. Their learning can be approached as training a parametrised Markov chain kernel by maximising the data likelihood of its limiting distribution. This has the additional advantage of allowing a wide range of semi-supervised learning scenarios.
Figures & tables
Figure 1. Dependency of the matrix asymmetry on the chain length n ; see Section 5.1 .
Figure 2. GMM results. Top: the learned densities (coloured lines), as well as the ground truth density (shown in black) and the density obtained for the best consistent model (shown as dashed black line). Bottom, left: KL-divergence DKL(p∗(x)∥p(x)) between the learned and the ground truth densities in dependency on the chain length. Bottom, right: the dependency of the proposed inconsistency measure on the chain length.
Figure 3. Conditional generation for MNIST / Fashion MNIST. Columns correspond to classes.
Figure 4. Generation for CelebA, conditioned on the presence of glasses attribute (top) and male attribute (bottom). Each display shows pairs of image and segmentation generated with the attribute switched off (first two rows) and switched on (last two rows).
Figure 5. Segmentation from incomplete images / in-painting. First row: original images with hidden parts shown as black squares, second row: ground truth segmentations, third row: predicted segmentations, fourth row: in-painting.
Figure 6. Image demixing. First two rows: original image pairs, third row: mixed image, last two rows: demixing results.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7. Given a code z , images are obtained by sampling, which alternates between sampling from p(x∣c,z) and p(c∣x,z) , i.e. x∼p(x∣z) . Top row: z∼π(z) (see the text), bottom row: z uniform.
Figure 8. The dependencies of the optimisation objective on the chain length during the training. Top row: log-linear model, second row: symmetrised log-linear model, bottom row: MLP. Left: short optimisation chains, right: long optimisation chains.
Figure 9. Network architectures. Left: for 1D outputs, right: for 2D outputs.
Diffusion models are increasingly used as powerful conditional generators, yet real deployments often involve multiple target distributions arising from different tasks, e.g., diverse prompt domains in text-to-image generation, or multiple environments in robotics with diffusion policies. This naturally leads to a multi-objective learning (MOL) problem. A key challenge is that achieving good Pareto trade-offs can require a generalist model class with substantially larger capacity than what suffices for solving any individual task, thereby increasing statistical cost since sample complexity typically scales with the model complexity. To reconcile this, we develop a principled MOL framework for diffusion models with limited data: a semi-supervised regime where paired (labeled) samples are scarce, but (unlabeled) condition data are abundant. We propose a two-stage training procedure that first fits lightweight specialist models from limited paired data, and then distills them into a generalist model by generating pseudo-samples. We establish generalization bounds showing that the required number of paired samples only depends on the complexity of the specialist model classes. We further extend the theory to diffusion policies for sequential decision making to account for distribution shift in on-policy rollouts. Extensive experiments on robotic control and image restoration tasks are conducted to verify our theoretical results.
This paper studies the problem of learning a joint distribution from marginal observations, which is inherently ill-posed due to the ambiguity of feasible couplings. We propose LUD-MSR, a latent-variable probabilistic framework that models the joint distribution via auxiliary representations and optimizes evidence lower bounds using only marginal data. Under mild assumptions, we establish an upper bound on the distribution approximation error. This analysis reveals a trade-off in representation learning between domain consistency and information preservation. To address this trade-off, we introduce a Multi-Scale image Representation (MSR) mapping that exploits structural similarity at coarse scales while suppressing domain-specific variations. We show that MSR achieves a more favorable balance of this trade-off compared to existing approaches. Experiments on real-world denoising benchmarks, including cryo-electron microscopy (cryo-EM), demonstrate the effectiveness of the proposed framework.
Yihang Zou, Hui Zhang, Zuowei Shen +1
Yau Mathematical Sciences Center and Department of Mathematical Sciences, Tsinghua University, Beijing, China · Qiuzhen College, Tsinghua University, Beijing, China · Department of Mathematics, National University of Singapore, Singapore +1
Self-supervised representation learning often relies on deterministic predictive architectures to align context and target views in latent space. While effective in many settings, such methods are limited in genuinely multi-modal inverse problems, where squared-loss prediction collapses towards conditional averages, and they frequently depend on architectural asymmetries to prevent representation collapse. In this work, we propose a probabilistic alternative based on generative joint modeling. We introduce Gaussian Joint Embeddings (GJE) and its multi-modal extension, Gaussian Mixture Joint Embeddings (GMJE), which model the joint density of context and target representations and replace black-box prediction with closed-form conditional inference under an explicit probabilistic model. This yields principled uncertainty estimates and a covariance-aware objective for controlling latent geometry. We further identify a failure mode of naive empirical batch optimization, which we term the Mahalanobis Trace Trap, and develop several remedies spanning parametric, adaptive, and non-parametric settings, including prototype-based GMJE, conditional Mixture Density Networks (GMJE-MDN), topology-adaptive Growing Neural Gas (GMJE-GNG), and a Sequential Monte Carlo (SMC) memory bank. In addition, we show that standard contrastive learning can be interpreted as a degenerate non-parametric limiting case of the GMJE framework. Experiments on synthetic multi-modal alignment tasks and vision benchmarks show that GMJE recovers complex conditional structure, learns competitive discriminative representations, and defines latent densities that are better suited to unconditional sampling than deterministic or unimodal baselines.