FeaTok: Feature-Loss Tokenization for Image Generation

Zhang Xiaofeng; Samuel Lavoie; Nicolas Beltran-Velez; Jianhao Yuan; Felix Friedrich; Jakob Verbeek; Aaron Courville; Michal Drozdzal

cs.CVOct 15, 2026

FeaTok: Feature-Loss Tokenization for Image Generation

Authors: Zhang Xiaofeng, Samuel Lavoie, Nicolas Beltran-Velez, Jianhao Yuan, Felix Friedrich, Jakob Verbeek, Aaron Courville, Michal Drozdzal,

Organizations: Mila - Québec AI Institute · Université de Montréal · FAIR at Meta · Columbia University ·

Abstract

Discrete image generators learn the distribution of tokens produced by an image tokenizer. These token representations should be easy for generators to model, not merely enable faithful image reconstruction. Yet these tokenizers are commonly trained to reconstruct pixels, which prioritizes low-level fidelity and may disrupt representation structure useful for generation. Motivated by feature-predictive self-supervised learning, we introduce feature-loss tokenization (FeaTok), which trains discrete tokens by reconstructing pretrained features rather than pixels. Ablations across SigLIP, SigLIP2, DINOv2, and DINOv3 show better generative performance with FeaTok. In class-conditional generation, FeaTok advances the quality-compute Pareto frontier, reducing gFDr6 at 256x256 by up to 29 percent over the strongest discrete baselines and outperforming the strongest unguided continuous baseline (RAEv2) with 38 times less sampling compute. FeaTok also achieves higher human-preference reward-model scores than prior art without reward-based training and surpasses continuous and discrete text-conditional baselines on three benchmarks with only 500 fine-tuning updates after pre-training.

Figures & tables

Figure 1: Feature-based discrete tokens enable strong generation. FeaTok models advance the quality–compute Pareto frontier for image generation. We categorize generative image models by how pretrained feature-predictive self-supervised learning representations (Def. 1 ) shape the generated representation or the generator: not at all (pixel-based), together with pixel objectives (mixed), or alone (feature-based). Generation performance on ImageNet $256\times 256$ , where all models sample without guidance. Both axes are log-scaled.

Figure 2: FeaTok tokenizer and generation pipeline. Blue denotes the conventional discrete tokenizer and orange our contributions. Only feature losses shape the discrete tokens:

\mathcal{L}_{\rm feat}

trains

D_{\rm feat}

to reconstruct the multi-layer target

y

of the frozen encoder

\hat{E}

from both global and patch tokens, while

\mathcal{L}_{\rm pix}

trains only

D_{\rm pix}

on detached patch tokens. The generator (not shown) models the global token from

Q_{\rm global}

together with the patch tokens from

Q_{\rm patch}

Figure 3: Tokenizer-target and generator-type ablations. (a) With patch-only generation, feature targets improve gFID for SigLIP, DINOv2, and DINOv3. The DINOv3 pixel bar has a break to accommodate its gFID of

2.92

. (b) Spatial dependence between adjacent patch tokens versus patch-only gFID. The feature target raises the dependence and lowers gFID for SigLIP, DINOv2, and DINOv3. Both axes are log-scaled. (c) With the feature target fixed, global-augmented generation improves gFID for all four encoders, achieving better generation than pixel-target baselines.

Figure 4: Feature-target depth. First-layer, last-layer and four-layer-average targets with global-augmented generation. The SigLIP first-layer bar has a break to accommodate its gFID of

2.50

Pixel-based
			without guidance							with guidance
method	size	TFLOPs	gFDr6	incep.	cvnx2	dino2	mae	siglip2	clip	gFDr6	incep.	cvnx2	dino2	mae	siglip2	clip
VAR-d30 ( Tian et al., 2024 )	2.0B	‡ 2.29	12.14	3.09	5.09	11.92	10.21	22.60	19.91	6.75	1.16	1.64	5.30	7.25	11.95	13.22
LlamaGen-3B ( Sun et al., 2024 )	3.1B	4.18	13.70	5.62	8.70	13.33	9.34	23.32	21.91	5.97	1.26	1.40	4.75	5.35	10.46	12.59
TiTok-S-128 ( Yu et al., 2024b )	0.3B	5.08	21.45	5.11	10.51	22.62	14.22	41.96	34.27	15.32	1.19	1.98	14.04	13.53	30.23	30.96
Open-MAGVIT2-XL ( Luo et al., 2024 )	1.5B	1.16	18.70	6.85	12.83	19.89	10.29	34.18	28.13	7.72	1.39	2.24	7.15	5.53	14.98	15.03

Table 1: Comparison with discrete baselines on ImageNet-1k

256\!\times\!256

. FeaTok outperforms all discrete baselines with and without guidance. Size denotes the generator parameter count. TFLOPs reports the sampling cost per image without guidance. Baselines use their released sampling settings, and App. F reports FeaTok at gFID-selected settings. ‡ VAR’s unguided generator cost is estimated as half its guided cost. Bold and underline mark the best and second-best values across all models in each metric column.

Pixel-based
			without guidance							with guidance
method	size	TFLOPs	gFDr6	incep.	cvnx2	dino2	mae	siglip2	clip	gFDr6	incep.	cvnx2	dino2	mae	siglip2	clip
SiT-XL/2 ( Ma et al., 2024 )	675M	57.82	16.05	5.24	8.34	16.77	9.40	30.05	26.52	8.44	1.26	2.02	7.89	5.62	16.14	17.69
Mixed
SiT-XL/2-REPA ( Yu et al., 2025b )	675M	12.69	11.39	3.57	5.10	10.81	7.34	20.70	20.80	5.45	0.85	1.22	4.27	3.85	9.87	12.65
LightningDiT-XL/1 ( Yao et al., 2025 )	675M	57.45	6.22	1.33	2.42	5.77	4.08	11.77	11.95	4.57	0.85	1.09	3.76	3.02	8.47	10.21

Table 2: Comparison with baselines using continuous features on ImageNet-1k at

256\times 256

. Without guidance, FeaTok achieves lower gFDr6 than all listed baselines at a lower sampling cost. Size denotes the generator parameter count, and TFLOPs reports the sampling cost per image without guidance. Bold and underline mark the best and second-best values in each metric column. Guided continuous-baseline results are transcribed from Singh et al. (2026) .

Baselines with continuous features
			without guidance							with guidance
method	steps	TFLOPs	gFDr6	incep.	cvnx2	dino2	mae	siglip2	clip	gFDr6	incep.	cvnx2	dino2	mae	siglip2	clip
RAEv2-XL ( Singh et al., 2026 )	3	1.65	38.90	14.12	11.69	16.68	69.14	78.74	43.06	8.95	2.98	3.15	6.41	10.49	16.15	14.50
RAEv2-XL ( Singh et al., 2026 )	6	3.07	8.49	2.52	2.90	5.32	12.75	14.45	13.01	3.43	1.27	1.29	2.11	4.54	4.92	6.42
Baselines of one-step flow models
Drift-L (latent) ( Deng et al., 2026 )	1	0.79	10.92	0.91	2.03	10.35	6.51	24.12	21.59	10.92	0.91	2.03	10.35	6.51	24.12	21.59

Table 3: Generation with matched inference cost on ImageNet

256\times 256

. TFLOPs are computed without guidance per image. Guided one-step results are from Yang et al. (2026) . Drift models share the same setup w and w/o guidance. Bold and underline mark the best and second-best values in each metric column.

Figure 5: Human-preference rewards on

50

K generated ImageNet images with

95\%

confidence intervals. FeaTok achieves the highest rewards compared to baselines, and also closer to the rewards measured on the training set. Dashed lines represent the mean of the rewards over training-data. HPSv2 scores are displayed multiplied by

100

. FeaTok uses large generators sampled with guidance.

	without guidance							with guidance
method	gFDr6	incep.	cvnx2	dino2	mae	siglip2	clip	gFDr6	incep.	cvnx2	dino2	mae	siglip2	clip
Baselines
RAE-XL † ( Zheng et al., 2026b )	3.03	0.97	1.42	2.07	3.11	4.40	6.19	2.13	0.75	1.10	1.34	2.02	2.74	4.84
BAR-L ( Yu et al., 2026 )	3.82	1.19	1.77	2.86	2.80	8.98	5.30	2.94	0.78	1.02	2.24	2.54	5.81	5.22
FeaTok-L-DINOv3	2.17	1.16	0.74	1.41	1.69	4.63	3.38	1.80	0.94	0.65	1.26	1.30	3.51	3.17

Table 4: Generation on ImageNet-1k

512\times 512

. FeaTok outperforms BAR-L and RAE with and without guidance. Bold and underline mark the best and second-best values in each metric column. † RAE uses autoguidance instead of classifier-free guidance, and its unguided row uses the released configuration with autoguidance disabled.

continuous models
	after pre-training			after fine-tuning
model	GenEval $\uparrow$	GenAI $\uparrow$	DPG $\uparrow$	GenEval $\uparrow$	GenAI $\uparrow$	DPG $\uparrow$
Flux-VAE ( Black Forest Labs, 2024 )	0.417	0.573	77.6	0.783	0.639	79.2
RAE ( Zheng et al., 2026b )	0.584	0.632	80.1	0.815	0.672	80.6
RAEv2 ( Singh et al., 2026 )	0.624	0.638	81.7	0.827	0.680	82.3
discrete models

Table 5: Text-to-image results after pre-training and fine-tuning. FeaTok is broadly comparable to RAEv2, and surpasses all baselines after 500 fine-tuning updates. Continuous-latent baseline results are taken from Singh et al. (2026) .

Figure 6: Text-to-image samples with abbreviated prompts. Full prompts are available in Tab. 17 .

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

name in main text	variant	checkpoint	hidden dim	patch size
SigLIP	ViT-L/16	google/siglip-large-patch16-256	1024	16
SigLIP2	SO400M/16, NaFlex	google/siglip2-so400m-patch16-naflex	1152	16
DINOv2	ViT-L/14, registers	facebook/dinov2-with-registers-large	1024	14
DINOv3	ViT-L/16	facebook/dinov3-vitl16-pretrain-lvd1689m	1024	16

Appendix

Table 6: Pretrained source encoders and checkpoints, shared by the frozen target encoder

\hat{E}

and the initialization of the trainable encoder

E

category	parameter	value
architecture	feature decoder $D_{\rm feat}$	ViT, $24$ layers, width $1024$ (ViT-L encoders),
		$27$ layers, width $1152$ (SigLIP2)
	pixel decoder $D_{\rm pix}$	ViT-L, $24$ layers, width $1024$ , patch size $16$
	quantizer	FSQ, $16$ bits per patch token and $16$ bits for the global token
optimization	optimizer	AdamW, $\beta=(0.9,0.999)$
	learning rate	$10^{-4}$ to $10^{-5}$ , cosine decay

Appendix

Table 7: FeaTok tokenizer architecture and training hyperparameters at

256\times 256

, shared across encoders unless noted. We follow the tokenizer setup of BAR ( Yu et al., 2026 ) and change only the parts specific to FeaTok: the feature decoder and feature losses, the encoder regularizer, the discrete global token, and the stop-gradient that restricts the pixel losses to the pixel decoder. Epochs are over ImageNet-1k, with optimizer steps in parentheses.

category	parameter	value
architecture	encoder input ∗	DINOv3-L on the full $512\times 512$ image, $32\times 32$ grid of $1024$ patch tokens
	feature decoder $D_{\rm feat}$	ViT, $24$ layers, width $1024$
	pixel decoder $D_{\rm pix}$	ViT-L, $24$ layers, width $1024$ , patch size $16$
	quantizer ∗	FSQ, $10$ bits per patch token and $10$ bits for the global token
	memory ∗	gradient checkpointing
optimization	optimizer	AdamW, $\beta=(0.9,0.999)$

Appendix

Table 8: FeaTok tokenizer architecture and training hyperparameters at

512\times 512

, used for FeaTok-L-DINOv3 in Tab. 4 . ∗ Entries that differ from the

256\times 256

recipe in Tab. 7 . Epochs are over ImageNet-1k, with optimizer steps in parentheses.

category	parameter	BAR-B	BAR-L
architecture	layers	$24$	$32$
	width	$768$	$1280$
	attention heads	$12$	$16$
	masked bit modeling head	$3$ layers, width $2048$
conditioning	class condition tokens	$32$
	global token repeats	$32$

Appendix

Table 9: Generator architecture and training hyperparameters at

256\times 256

. We follow the generator setup of BAR ( Yu et al., 2026 ) and add only the global token: its conditioning, its joint condition dropout, and the loss weight

\lambda_{\rm global}

. Epochs are over ImageNet-1k, with optimizer steps in parentheses.

category	parameter	BAR-L
architecture	layers	$32$
	width	$1280$
	attention heads	$16$
	masked bit modeling head	$3$ layers, width $2048$
	sequence ∗	$1$ global token and $1024$ patch tokens, $10$ bits each
conditioning	class condition tokens	$32$

Appendix

Table 10: Generator architecture and training hyperparameters at

512\times 512

, used for FeaTok-L-DINOv3 in Tab. 4 . ∗ Entries that differ from the BAR-L column of the

256\times 256

recipe in Tab. 9 . Epochs are over ImageNet-1k, with optimizer steps in parentheses.

		SigLIP		SigLIP2		DINOv2		DINOv3
tokenizer target	generator type	rFID	gFID	rFID	gFID	rFID	gFID	rFID	gFID
pixel	patch-only	1.40	1.88	1.16	1.68	0.85	1.76	0.72	2.92
pixel + feature	patch-only	1.26	1.74	1.08	1.74	0.83	1.72	0.73	1.98
feature	patch-only	1.01	1.70	0.92	1.80	0.93	1.63	0.99	1.48
pixel + feature	global-augmented	1.26	1.77	1.08	1.65	0.83	1.60	0.73	1.67
first layer	global-augmented	0.93	2.50	1.03	1.85	1.26	1.69	1.03	1.42

Appendix

Table 11: Ablations for FeaTok. Feature-only loss outperforms pixel and combined pixel–feature losses for three pretrained encoders. Jointly modeling global and patch tokens with feature-only supervision achieves the best gFID for all four encoders. We report rFID on the ImageNet validation set and gFID on

50

K ImageNet samples without guidance, with lower values indicating better performance. The tokenizer target specifies which losses train the discrete tokens, and the generator type specifies whether global-augmented generative modeling is used. Feature denotes the normalized average of four encoder layers, whereas first layer and last layer use a single encoder layer as the target. rFID depends only on the tokenizer, so rows that share a tokenizer report the same value. Bold and underline indicate the best and second-best result in each metric column.

source	supervision	rFID $\downarrow$	rFDr6 $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
SigLIP	pixel	1.40	2.68	17.89	0.426
SigLIP	feature	1.01	2.45	17.27	0.418
SigLIP2	pixel	1.16	2.09	18.81	0.466
SigLIP2	feature	0.92	2.04	17.67	0.433
DINOv2	pixel	0.85	1.82	20.04	0.519
DINOv2	feature	0.93	2.06	17.50	0.433

Appendix

Table 12: Reconstruction of regularized tokenizers on ImageNet at

256\times 256

. Feature denotes the four-layer-average target. Bold marks the better result within each source block.

Figure 7: Linear separability and generation. Discrete-token probe accuracy on

25

K test images versus patch-only gFID on

50

K samples without guidance. The gFID axis is log-scaled.

		patch branch			global branch
model	guidance	alloc.	gs	temp	alloc.	gs	temp
base generator ( $0.4$ B)
FeaTok-SigLIP	without	$[4,4,4,4]$	1.0	1.8	$[4,4,4,4]$	1.0	1.4
FeaTok-SigLIP	with	$[4,4,4,4]$	4.0	1.6	$[4,4,4,4]$	1.0	2.0
FeaTok-SigLIP2	without	$[4,4,4,4]$	1.0	1.6	$[3,3,5,5]$	1.0	2.0
FeaTok-SigLIP2	with	$[4,4,4,4]$	4.0	1.6	$[4,4,4,4]$	1.0	2.0

Appendix

Table 13: Sampling configurations of FeaTok for the

256\times 256

results in Tab. 1 .

base generator
	generation@256 w/o guidance					generation@256 w/ guidance
method	gFID $\downarrow$	Prec. $\uparrow$	Rec. $\uparrow$	Dens. $\uparrow$	Cov. $\uparrow$	gFID $\downarrow$	Prec. $\uparrow$	Rec. $\uparrow$	Dens. $\uparrow$	Cov. $\uparrow$
BAR-B ( Yu et al., 2026 )	1.64	0.800	0.614	1.153	0.997	1.13	0.775	0.660	1.060	0.994
FeaTok-B-SigLIP	1.61	0.807	0.621	1.146	0.996	1.27	0.778	0.654	1.068	0.994
FeaTok-B-SigLIP2	1.53	0.802	0.621	1.142	0.995	1.12	0.782	0.655	1.067	0.996
FeaTok-B-DINOv2	1.47	0.782	0.645	1.085	0.996	1.10	0.784	0.659	1.069	0.995

Appendix

Table 14: ImageNet-1k generation at

256\times 256

, without and with guidance, evaluated by gFID and PRDC. Bold and underline mark the highest and second-highest precision, recall, density, and coverage within each discrete generator-scale block, with ties at the displayed precision sharing a rank.

	without guidance							with guidance
method	gFDr6	incep.	cvnx2	dino2	mae	siglip2	clip	gFDr6	incep.	cvnx2	dino2	mae	siglip2	clip
BAR-B ( Yu et al., 2026 )	5.15	1.01	1.70	5.21	5.82	9.40	7.73	4.89	0.69	0.95	4.63	5.89	9.32	7.86
BAR-B (refined decoder)	4.62	0.99	1.65	4.71	5.23	8.39	6.76	4.36	0.67	0.90	4.08	5.35	8.32	6.81
FeaTok-B-SigLIP	5.08	1.01	1.31	5.63	6.11	9.28	7.15	4.63	0.79	1.03	4.77	5.77	8.91	6.51
FeaTok-B-SigLIP2	4.66	0.96	1.29	4.97	5.14	8.85	6.73	4.03	0.70	0.85	3.88	4.71	7.65	6.39
FeaTok-B-DINOv2	4.40	0.92	1.36	4.12	4.93	8.31	6.74	3.87	0.69	0.85	3.37	4.51	7.34	6.43

Appendix

Table 15: gFDr6 on ImageNet-1k

256\times 256

at gFID-selected sampling configurations. BAR uses its released settings, and each FeaTok model uses the configuration of Tab. 14 . Refined-decoder rows replace BAR’s pixel decoder by the one fine-tuned for

40

additional epochs, with the same tokens and sampling settings. Columns follow Tab. 1 . Bold and underline mark the best and second-best values within each generator size in each metric column.

		matched batch $256$			peak e2e over the batch sweep
system	guidance	gen. img/s $\uparrow$	e2e img/s $\uparrow$	GiB $\downarrow$	img/s $\uparrow$	bsz	GiB $\downarrow$
BAR-B ( Yu et al., 2026 )	without	28.93	27.69	10.6	34.32	1024	32.2
BAR-B ( Yu et al., 2026 )	with	17.06	16.59	17.8	17.49	512	32.2
FeaTok -B	without	27.49	26.36	10.4	31.82	1024	31.3
FeaTok -B	with	15.97	15.57	17.4	16.31	512	31.4
BAR-L ( Yu et al., 2026 )	without	16.02	15.61	22.1	17.02	512	36.8

Appendix

Table 16: Sampling throughput and peak GPU memory on one L40S (bf16, uncompiled). Throughput is in generated images per second (img/s, higher is better), and memory is the peak GPU memory in GiB (lower is better). gen. counts generation only, and e2e includes pixel decoding. We report a matched batch size of

256

, and each model/setting’s best end-to-end throughput over the batch sweep, with the batch size (bsz) at which it is reached and its memory. Guided rows use classifier-free guidance for BAR and FeaTok (scale

5.0

for base and

5.3

for large models) and internal guidance (

1.78

) for RAEv2, which needs no extra forward pass. FeaTok rows use the DINOv3 tokenizer, and throughput does not depend on the encoder. RAEv2 uses

100

ODE steps, the setting of its released sampling configuration ( https://github.com/nanovisionx/RAEv2/blob/8a0d238/configs/stage2/sampling/imagenet-dinov3l-k7.yaml ).

Panel of Figure 8 — Figure 8: ImageNet qualitative samples (1/4). Rows: BAR-L, then FeaTok-L with SigLIP2, DINOv2, DINOv3, and SigLIP, at their respective guided settings (App. E ).

Panel of Figure 9 — Figure 9: ImageNet qualitative samples (2/4). Rows: BAR-L, then FeaTok-L with SigLIP2, DINOv2, DINOv3, and SigLIP, at their respective guided settings (App. E ).

Panel of Figure 10 — Figure 10: ImageNet qualitative samples (3/4). Rows: BAR-L, then FeaTok-L with SigLIP2, DINOv2, DINOv3, and SigLIP, at their respective guided settings (App. E ).

Panel of Figure 11 — Figure 11: ImageNet qualitative samples (4/4). Rows: BAR-L, then FeaTok-L with SigLIP2, DINOv2, DINOv3, and SigLIP, at their respective guided settings (App. E ).

Panel of Figure 12 — Figure 12: Text-to-image samples at $256\times 256$ from MJHQ-30K animal (first two rows) and food prompts. Full numbered prompts are in Tab. 17 .

#	prompt
1	a butterfly gently touching the vast surface of the lake, peace, serenity,
2	a photo of a cat and a dog on the beach wearing fancy hats
3	abstract photo of a cat jumping through a flower field on a bright sunny day
4	american traditional tiger drinking a beer, sticker, square
5	cute beautiful flamingo, floral garnish, tropical flowers, watercolor vibrant illustration, png
6	grizzly bear wearing medieval armor looking fierce and roaring

Appendix

Table 17: Full MJHQ-30K ( Li et al., 2024b ) prompts for the panels of Fig. 12 , numbered in the figure’s reading order (left to right, top to bottom); the number appears in front of each panel’s truncated caption. Prompts are reproduced verbatim, including the trailing keyword lists and the original capitalisation and punctuation, since these are part of what the model was conditioned on. Panels

1

2

6

15

13

are the five reproduced in Fig. 6 .

Image Tokenization

Image tokenization converts images into compact discrete or continuous visual units for efficient generation, compression, and multimodal reasoning.

See all 57 papers

Jun 4, 2026

Balancing Image Compression and Generation with Bootstrapped Tokenization

Despite progress in image tokenization, standard methods encode redundant information by mixing all granularities within each token, thus redundancy persists between tokens. The mix of information of different granularity also complicates the training of generators. This paper introduces SelfBootTok, a method that resolves this by cleanly decomposing information into global and local token groups. Through self-bootstrapped learning, the model predicts local details exclusively from global tokens, shifting the burden of visual details from the generator to the tokenizer. Consequently, our generator is far more efficient, requiring only global tokens and reducing computation by approximately 40%, while delivering superior reconstruction and generation. Moreover, this paradigm scales elegantly: by leveraging more data or parameters to self-supervise local representation learning, SelfBootTok achieves a new state-of-the-art gFID score of 1.56 using only 64 tokens.

Haozhe Chi, Jinghan Li, Hao Jiang +4

Peking University · Central Media Technology Institute, Huawei

Learned Image Compression Image Tokenization Visual Tokenization Autoregressive Image Generation Self-Supervised Visual Representation Learning Visual Token Compression

Ask PDF

Mar 31, 2026

MacTok: Robust Continuous Tokenization for Image Generation

Continuous image tokenizers enable efficient visual generation, and those based on variational frameworks can learn smooth, structured latent representations through KL regularization. Yet this often leads to posterior collapse when using fewer tokens, where the encoder fails to encode informative features into the compressed latent space. To address this, we introduce MacTok, a Masked Augmenting 1D Continuous Tokenizer that leverages image masking and representation alignment to prevent collapse while learning compact and robust representations. MacTok applies both random masking to regularize latent learning and DINO-guided semantic masking to emphasize informative regions in images, forcing the model to encode robust semantics from incomplete visual evidence. Combined with global and local representation alignment, MacTok preserves rich discriminative information in a highly compressed 1D latent space, requiring only 64 or 128 tokens. On ImageNet, MacTok achieves a competitive gFID of 1.44 at 256

\times

256 and a state-of-the-art 1.52 at 512

\times

512 with SiT-XL, while reducing token usage by up to 64

\times

. These results confirm that masking and semantic guidance together prevent posterior collapse and achieve efficient, high-fidelity tokenization.

Hengyu Zeng, Xin Gao, Guanghao Li +5

Fudan University

Image Tokenization Visual Token Compression Posterior Collapse

Ask PDF

Oct 14, 2026

Generative Representation Learning for Scalable Latent Tokenization

We introduce an approach for self-supervised representation learning designed to be used as a tokenizer for generative modeling. Our objective supervises a representation loss by factorizing the pointwise mutual information of the data, which we estimate by taking information differences between conditional predictions from a learned generative model. We justify our algorithm theoretically and empirically, demonstrating that our generative approach to representation learning avoids dimensional collapse and feature suppression in cases where commonly used methods like DINO and LeJEPA fail. Moreover, we show our representation learning technique is scalable; expending more compute leads to predictably better latent tokenization. For the first time, we perform a careful analysis of the combined flop efficiency of tokenizer and generative model and demonstrate that our approach is a compute-optimal improvement over direct diffusion for pixel-space generation, obtaining an FID-50K of 2.18 with JiT-L. Applied to ImageNet and the text-to-image dataset GPIC, we show tokenizer encoder scaling laws for diffusability, where extrapolating our approach achieves representations in 7x fewer FLOPs than DINOv2 on ImageNet and 8x fewer FLOPs than WebSSL on GPIC.

Alan Baade, Kyle Sargent, Vincent Sitzmann +4

Stanford University · Massachusetts Institute of Technology · University of Michigan

Representation Learning Image Tokenization Self-Supervised Visual Representation Learning Diffusion-Based Image Generation

Ask PDF

Abstract

Image Tokenization

Explore similar work

Balancing Image Compression and Generation with Bootstrapped Tokenization

MacTok: Robust Continuous Tokenization for Image Generation

Generative Representation Learning for Scalable Latent Tokenization