Bringing Generative Learning to Representation Learning: Self-Supervised Transfer Learning as Distribution Matching

Yuling Jiao; Wensen Ma; Defeng Sun; Hansheng Wang; Yang Wang

stat.MLFeb 20, 2025

Bringing Generative Learning to Representation Learning: Self-Supervised Transfer Learning as Distribution Matching

Authors: Yuling Jiao, Wensen Ma, Defeng Sun, Hansheng Wang, Yang Wang

Organizations: School of Artificial Intelligence, Wuhan University, 430072, Wuhan, China · School of Mathematics and Statistics, Wuhan University, Wuhan, 430072, China · Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong SAR, China · Guanghua School of Management, Peking University, 100871, Beijing, China · Department of Mathematics, The University of Hong Kong, Pokfulam, Hong Kong SAR, China

Abstract

Most self-supervised learning objectives defend against collapse but leave the target representation law unspecified. We formulate representation learning as Distribution Matching (DM), learning an augmentation-invariant encoder whose induced law matches an explicit geometric reference. The reference law specifies what the learned representation distribution should look like, whereas a separately chosen discrepancy determines how deviations from this target are measured; here we use Mallows distance. The DM framework reveals a directional inverse: generative learning maps a tractable reference to data, whereas representation learning maps data to a designed reference law. We connect the population objective to class-centre separation and classification error and prove a non-asymptotic neural-sieve guarantee. Simulations and image benchmarks show manifold rectification, fine-grained structure and transfer across label spaces.

Figures & tables

Figure 1: Raw Euclidean distance can conflict with semantic meaning. Although $\bm{x}_{3}$ is a cropped view of $\bm{x}_{1}$ , $\left\|{\bm{x}_{1}-\bm{x}_{2}}\right\|_{2}<\left\|{\bm{x}_{1}-\bm{x}_{3}}\right\|_{2}$ .

Figure 2: The prescribed reference distribution. To draw from $\mathbb{P}_{\mathcal{R}}$ , a component is selected with probability $\alpha_{k}$ , its orthogonal centre $\bm{c}_{k}$ is perturbed by the Gaussian direction in ( 4 ), and the result is normalised to the radius- $R$ hypersphere. Two components are shown; $\epsilon$ controls their within-component spread.

Figure 3: Error versus unlabelled sample size. The DM error decreases over the four values of

n_{S}

; the latent-oracle error is shown as a reference level.

Figure 4: t-SNE visualisation. (a) Standardised observations

X

; (b) normalised DM representations

f(X)

. Colours denote latent classes and are not used in pretraining.

Table 1: Domain-transfer target classification accuracy (%). For each benchmark, the encoder is pretrained on an unlabelled source sample and evaluated on the corresponding target domain using an affine linear probe and a

k

-NN probe. DM uses

K^{\prime}=384

reference components. The reported DM accuracies are higher than those of the listed baselines in each of the six comparisons.

Feature space	Raw $X$	Latent oracle $Z$	DM $f(X)$
Classification error	0.4160	0.3320	0.3310

Table 4: Classification error after non-linear warping. Here

d=512

n_{S}=15,000

and

n_{T}=200

Unlabelled size ( $n_{S}$ )	Latent oracle $Z$	DM $f(X)$
$1,000$	0.2970	0.5890
$5,000$	0.2970	0.4220
$10,000$	0.2970	0.3065
$20,000$	0.2970	0.2905

Table 5: Empirical error as

n_{S}

increases. The same target and test designs are used at every source sample size.

Figure 5: Augmentation-induced semantic bridging. The raw observations $\bm{x}_{1}$ and $\bm{x}_{2}$ may be far apart, while transformations $A_{1},A_{2}\in\mathcal{A}$ isolate views $\bm{x}_{1}^{\star}$ and $\bm{x}_{2}^{\star}$ whose Euclidean distance is at most $\delta$ . The augmentation distance therefore discounts nuisance variation that can be removed by the allowed transformations.

Method	Linear	$5$ -NN	Seconds per epoch
Optimised DM	$92.11$	$89.17$	$72.28$
DM-Batch	$90.22$	$87.16$	$\approx 31$

Table 6: CIFAR-10 representation quality and training speed of optimised DM and DM-Batch. Linear and

5

-NN denote test accuracy (%); epoch time is measured on a single Tesla V100.

Explore similar work

Learning a Flow to Self-Supervised Representations
Sep 24, 2026Yuling Jiao, Wensen Ma, Houduo Qi +1Self-Supervised Representations Imagenet
School of Artificial Intelligence, National Center for Applied Mathematics in Hubei, Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, China. · Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong SAR, China. · Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong SAR, China.
Understanding Self-Supervised Learning via Latent Distribution Matching
May 5, 2026Fabian A Mikulasch, Friedemann ZenkeFuture Latent Representations Bayesian Filtering
Friedrich Miescher Institute for Biomedical Research, 4056 Basel, Switzerland · Faculty of Science, University of Basel, 4033 Basel, Switzerland
Sharp Rates and a One-Line Correction for Spectral Representation Learning
Sep 14, 2026Dier Tang, Jing Yee Tan, Guangyue HanSelf-Supervised Representations Subspace
Department of Mathematics, The University of Hong Kong

Sep 24, 2026, cs.CVJ/K move · Enter open · S save

Learning a Flow to Self-Supervised Representations

Yuling Jiao, Wensen Ma, Houduo Qi, Defeng Sun

School of Artificial Intelligence, National Center for Applied Mathematics in Hubei, Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, China. · Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong SAR, China. · Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong SAR, China.

Self-Supervised Representations Imagenet Representation-Alignment Loss Velocity Field

Open paper AskPDF arXiv

Explicit geometric references offer a direct way to structure self-supervised representations. Existing adversarial distribution-matching formulations, however, require costly encoder-critic optimization. We introduce Flow-Based Distribution Matching (FBDM), a non-adversarial framework that learns this reference-directed geometry through spherical conditional velocity regression. An ETF-inspired reference allows its number of components K' to exceed the auxiliary flow dimension d* while retaining structured geometric separation. We assign both augmented views of each image to the same target, while limiting how many images each reference center can receive. An explicit alignment loss further pulls the two views' representations closer together. Experiments across benchmarks ranging from CIFAR to ImageNet show that FBDM achieves performance nearly on par with DM and remains competitive with existing SSL methods. Matched training-cost comparisons show a 1.48- to 1.83-fold speedup over DM with a negligible increase in GPU memory usage. We also provide a theoretical explanation for the usefulness of the learned representations: under stated conditions, we bound the downstream misclassification rate in terms of the FBDM pretraining loss.

Abstract

Explore similar work

Learning a Flow to Self-Supervised Representations

Understanding Self-Supervised Learning via Latent Distribution Matching

Sharp Rates and a One-Line Correction for Spectral Representation Learning