Generative Residual Factorization

Letian Gong; Yuzhou Hong

cs.CVSep 28, 2026

Generative Residual Factorization

Authors: Letian Gong, Yuzhou Hong

Organizations: Zhejiang University of Science and Technology · Zhejiang Sci-Tech University

Abstract

Under a shared-factor model, the conditional law of the next image patch factors into a posterior over the shared scene factor and a residual kernel given that factor. A sufficient statistic of the past replaces the raw past in the posterior and does not replace the kernel. The conditional entropy splits into residual entropy, which no observation of the factor can remove, and a posterior term, which a better representation of the past can remove. Next-embedding prediction is a directional likelihood on a shallow map, so the fiber of that map is unidentified and a constant embedding remains a minimizer. The same split is an equality in a scalar Gaussian model, evaluated in closed form.

Figures & tables

SNR	$I(x_{1};x_{2})$	$H(x_{2}\mid x_{1})$	$H(x\mid s)$	posterior gap
0.1	0.004	2.614	2.570	0.044
0.5	0.059	1.909	1.766	0.144
2	0.294	1.328	1.072	0.255
8	0.781	0.697	0.379	0.318
32	1.409	0.025	$-0.314$	0.339

Table 1: Denser Gaussian grid, still in nats,

v_{s}=1

. Residual entropy tracks

v_{n}

. Predictive information tracks

\rho

SNR	$I(x_{1};x_{2})$	$H(x_{2}\mid x_{1})$	$H(x\mid s)$	$I(x;s\mid\mathrm{past})$
0.25	0.020	2.203	2.112	0.091
1	0.144	1.622	1.419	0.203
4	0.511	1.020	0.726	0.294
16	1.085	0.364	0.033	0.332

Table 2: Gaussian split in nats at

v_{s}=1

. The signal-to-noise ratio is

v_{s}/v_{n}

Horizon $K$	residual sum	$I(x_{1};x_{2})$
1	0.413	0.127
2	0.825	0.127
4	1.651	0.127
8	3.302	0.127
16	6.604	0.127

Table 3: Residual bill on the binary channel after

s

is known. Each future bit costs

H(x\mid s)=0.4127

nats. Predictive information is not multiplied by

K

Patch	$d$	$I(x_{1};x_{2})$	$H(x\mid s)$
scalar	1	0.144	1.419
CIFAR-10, $4\times 4\times 3$	48	6.902	68.107
MNIST, $7\times 7$	49	7.046	69.527

Table 4: Isotropic Gaussian entropies at SNR

=1

, scaled by patch dimension. These are not estimates from pixels. They show how Corollary 7 turns one coordinate into a patch.

Data	Run	Best probe	Output	Residual MSE	Floor
MNIST	NEPA	87.24 (2)	76.03	0.03183	0.05680
MNIST	pixel	89.61 (2)	86.91	0.03236	0.05680
CIFAR-10	NEPA	31.54 (3)	27.26	0.01763	0.05748
CIFAR-10	pixel	31.68 (6)	31.68	0.01410	0.05748

Table 5: Posterior probe and residual probe. Probe is the best last-token linear accuracy (%), with the block index in parentheses, and the output-block accuracy. Residual is next-patch MSE of a linear map from the frozen final hidden states. Floor is the class-mean next-patch MSE.

Data	Run	0	1	2	3	4	5	6
MNIST	NEPA	15.07	84.53	87.24	85.96	84.23	79.90	76.03
MNIST	pixel	14.07	83.70	89.61	88.03	86.03	86.09	86.91
CIFAR-10	NEPA	19.68	28.20	29.80	31.54	31.11	30.20	27.26
CIFAR-10	pixel	19.73	28.86	28.56	29.96	31.11	31.43	31.68

Table 6: Last-token linear probe (%) at every block. Block 0 is the patch embedding.

Epoch	MNIST NEPA	MNIST pixel	CIFAR NEPA	CIFAR pixel
1	$-0.853$	0.069	$-0.956$	0.036
2	$-0.910$	0.037	$-0.954$	0.021
4	$-0.919$	0.031	$-0.955$	0.014
8	$-0.926$	0.027	$-0.958$	0.012
12	—	—	$-0.959$	0.012

Table 7: Mean training loss by epoch. NEPA entries are negative cosines, so more negative is a better directional fit. Pixel entries are mean squared error on the training batches.

Explore similar work

Conditional Predictive Sufficient Statistics for Visual Representation Learning
Sep 25, 2026Yuzhou HongPixel-Level Prediction Statistical Inference
Zhejiang Sci-Tech University
Marginal Matching Does Not License Factorized Sampling: Auditing Conditional Style Leakage in Factorized Generative Models
Aug 5, 2026Duong Bach, Hai Nguyen Hong, Cuong DoGenerative Models Exploratory Factor Analysis
Smart Health Center (VISHC), VinUniversity, Ha Noi, Viet Nam
Conditioning Residuals for Diffusion Models via Representation Feedback
May 16, 2025Weilai Xiang, Hongyu Yang, Di Huang +1Diffusion Models Generative Models
State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Institute of Artificial Intelligence, Beihang University, Beijing 100191, China · School of Computer Science and Engineering, Beihang University, Beijing 100191, China

Sep 25, 2026, cs.CVJ/K move · Enter open · S save

Conditional Predictive Sufficient Statistics for Visual Representation Learning

Yuzhou Hong

Zhejiang Sci-Tech University

Pixel-Level Prediction Statistical Inference

Open paper AskPDF arXiv

A useful visual representation is a statistic of the observed past that retains the latent factors shared with the future and discards patch-private noise. We formalize this requirement as a conditional predictive sufficient statistic (CPSS). Under a shared-factor model of image patches, the mutual information between the past and the next patch equals the information the past carries about the shared factor, up to a remainder that the next patch itself fails to reveal. Predicting the next patch embedding with a cosine loss is maximum likelihood for a von Mises-Fisher model of that embedding's direction, and is therefore a tractable surrogate for the predictive information. The same population loss is also minimized by a constant embedding, so stop-gradient does not by itself select the sufficient statistic; it only blocks the symmetric gradient that implements the constant solution in one step. The regression target is a shallow embedding, which forces the network output back into that shallow range and leaves the sufficient statistic in intermediate blocks. Small causal Transformers on MNIST and CIFAR-10 are used as diagnostics, not as a leaderboard. On MNIST the future shift and the stop-gradient move probe accuracy by tens of points, and the CPSS readout peaks before the output. On CIFAR-10, with the same short budget and no augmentation, every objective lands near a linear classifier on pixels. What still matches the derivation is the geometry: the CPSS output is a worse readout than its best intermediate block, next-pixel regression does not pay that penalty, and removing the stop-gradient collapses the effective rank of the embedding even when the pretext loss looks perfect.

Abstract

Explore similar work

Conditional Predictive Sufficient Statistics for Visual Representation Learning

Marginal Matching Does Not License Factorized Sampling: Auditing Conditional Style Leakage in Factorized Generative Models

Conditioning Residuals for Diffusion Models via Representation Feedback