Image Tokenizers Should Be Trained Between Images

Ole Beißwenger; Jan-Niklas Dihlmann; Hendrik Lensch

cs.CVOct 12, 2026

Image Tokenizers Should Be Trained Between Images

Authors: Ole Beißwenger, Jan-Niklas Dihlmann, Hendrik Lensch

Organizations: University of Tübingen

Abstract

Latent diffusion models generate images as codes of an image tokenizer, whose decoder turns them into the pixel image. Recent tokenizers improve how well these codes reconstruct an image and how easily a generator learns them. Yet a generator does not reproduce the codes of training images, and when a generated code lands outside what the decoder expects, the decoded image contains imperfections. Current methods make the decoder more robust to generated codes by adding noise during training. However, generated codes lie between the training codes and resemble their mixture more than one of them plus noise. To improve generated images, decoders should therefore be trained on codes in the space between training samples. We propose InterpTok, which fine-tunes MAETok to decode mixtures of image codes into natural images. Since no target image exists between two training samples, a discriminator judges whether each decoded mixture looks as natural as its two sources. A mixing ratio loss requires the decoded mixture to keep the mixing ratio of the two codes. Together, they prevent collapse to a trivial solution and yield a sharp and natural composition of both images. InterpTok keeps MAETok's reconstruction quality and improves interpolation quality more than threefold, the best of all tokenizers in a recent benchmark. Beyond interpolation, InterpTok improves the generation FID of a matched diffusion transformer from 11.61 to 8.48 without guidance, and even reaches 8.11 at an eighth of the sampling steps, where the generator on MAETok's codes worsens. The generator trained on our tokenizer also reaches the final result of the one trained on MAETok at half the training steps. Our findings show that the space between image codes can and should be explicitly supervised rather than ignored.

Figures & tables

Figure 1: Overview. Reconstruction trains the tokenizer at the codes of images A and B, never on the line between them. InterpTok trains it there, and the codes on the line between them decode to natural images that change from A toward B, where MAETok superimposes the two images.

DINOv2 encoder
Tokenizer	Latent dims	rFID $\downarrow$	iFID $\downarrow$
DM-VAE	8,192	0.683	8.29
RAE	196,608	0.57 †	7.68 †
Trained encoder
SD-VAE	4,096	0.62 †	59.91 †
EQ-VAE	4,096	0.82 †	47.51 †

Table 1: Reconstruction and interpolation on ImageNet. InterpTok against nine of the thirteen tokenizers compared by Xu et al. (2026) , and Table 30 lists the omitted rows. Among the nine are SD-VAE ( Rombach et al., 2022 ) , EQ-VAE ( Kouzelis et al., 2025 ) , DM-VAE ( Ye et al., 2026 ) , REPA-E ( Leng et al., 2025 ) and SoftVQ-VAE ( Chen et al., 2025b ) . Latent dims counts the values of a code. Bold is the best value among trained encoders, and underlining marks an improvement of InterpTok over MAETok. † Quoted from that comparison or the tokenizer’s own paper.

Figure 3: FID between two codes. (a) From a code to its midpoint with one of its nearest training codes, on a logarithmic axis. (b) The iFID of InterpTok over its fine-tuning steps.

SiT-B
gFID $\downarrow$	250	64	32	16
MAETok	11.61	12.33	13.88	20.56
w/ our decoder	8.56	8.99	9.92	15.33
InterpTok	8.48	8.09	8.11	12.48
SiT-XL released with MAETok
MAETok	2.46	2.78	3.46	7.52

Table 2: Generation against sampling steps. Each SiT-B is class-conditional, trained for 400,000 steps and sampled without guidance with the stochastic sampler of SiT ( Ma et al., 2024 ) . The second row decodes the first row’s generated codes, from the same noise, with the 41,000-step decoder that Table 3 trains on MAETok’s frozen encoder. The SiT-XL rows decode samples of the generator released by Chen et al. (2025a) .

n=50{,}000

per cell. Bold is the best value in its column and block.

encoder and decoder trained
	Params (M)	Steps	iFID $\downarrow$	rFID $\downarrow$	probe $\uparrow$	gFID $\downarrow$
MAETok	174	–	14.42	0.582	73.0	11.61
w/ interpolation	174	41k	5.44	0.556	63.5	18.90
w/ reg.	217	41k	12.86	0.588	77.0	21.15
InterpTok	217	41k	4.30	0.597	77.9	8.48
w/o DINOv3	217	41k	4.53	0.544	65.8	17.58

Table 3: Ablations of InterpTok. The upper rows train encoder and decoder. The lower rows train only the decoder on MAETok’s frozen encoder and are evaluated with MAETok’s generator. Interpolation is the interpolation discriminator with the pixel mixing ratio loss, reg. the regularization losses, DINOv3 the alignment losses and the feature mixing ratio loss, and Inception the Inception-v3 features of the representation Fréchet loss. Bold is the best value in its column and block.

Appendix figures & tables47 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Selected three-image interpolation. MAETok left and InterpTok right, with the source images at the corners, mixtures of two neighboring corners along the edges in steps of a quarter and mixtures of all three inside.

Figure 5: Selected interpolation, agaric.

Figure 6: Selected interpolation, rose hip.

Figure 7: Selected interpolation, volcano.

Figure 8: Selected interpolation, earthstar, first pair.

Figure 9: Selected interpolation, lycaenid.

Figure 10: Selected interpolation, pomegranate.

Figure 11: Selected interpolation, earthstar, second pair.

Figure 12: Selected interpolation, anemone fish.

Figure 13: Selected interpolation, boathouse.

Figure 14: Selected interpolation, sulfur butterfly.

Figure 15: Selected interpolation, yellow lady’s slipper.

Figure 16: Selected interpolation, orange.

Figure 17: Selected interpolation, strawberry.

Tokenizer	generator steps	gFID $\downarrow$
MAETok	100k	24.07
InterpTok	100k	16.26
MAETok	200k	15.88
InterpTok	200k	11.27
MAETok	300k	13.04
InterpTok	300k	9.54

Appendix

Table 4: Generation across generator training budgets, beside the published configurations. On the left, one SiT-B/1D per row on the named tokenizer’s codes, sampled at 250 steps without guidance. On the right, values as published, sampled without guidance, whose generator size, patch size and training budget differ across rows.

n=50{,}000

per cell.

Figure 18: Generation across the generator’s training. The percentages are the gap relative to MAETok’s gFID at each budget, the dashed rule is MAETok’s value at 400,000 steps, and the circle marks the first budget at which InterpTok reaches it.

Tokenizer	SigLIP2 $\downarrow$	DINOv2 $\downarrow$	MAE $\downarrow$	Inception pool $\downarrow$	ADM FID $\downarrow$
MAETok	63.72	344.8	0.825	13.20	11.61
InterpTok (41k)	59.01	309.6	0.775	10.27	8.48
InterpTok (82k)	60.73	323.1	0.851	11.69	9.57

Appendix

Table 5: Generated samples in four feature spaces. Each row is a SiT-B/1D at 400,000 steps on the named tokenizer’s codes, against the 50,000 ImageNet validation images and, for ADM FID, against the ADM reference statistics. Each value is a Fréchet distance in the features of SigLIP2, DINOv2, a masked autoencoder (MAE) ( He et al., 2022 ) or Inception, comparable within a column only.

Panel of Figure 19 — Figure 19: Selected samples without guidance. In each block of three rows, MAETok above, InterpTok (41k) in the middle and InterpTok (82k) below. Columns share a class and an initial noise.

the released MAETok SiT-XL ( Chen et al., 2025a )
Decoder	FID $\downarrow$
MAETok	2.46
$+$ our decoder	2.26

Appendix

Table 6: One generator, two decoders.

n=50{,}000

samples per block, decoded by MAETok’s decoder and by our decoder, comparable within a column only.

Figure 20: UMAP projections of image codes. $n=50{,}000$ ImageNet validation codes per panel, colored by class.

Table 27

Tokenizer	Reconstruction $\downarrow$	Midpoint $\downarrow$
SD-VAE	0.801	135.89
MAETok	0.863	85.21
w/ interpolation	0.865	9.23
InterpTok	1.013	13.61

Appendix

Table 9: MS-COCO reconstruction and decoded midpoints. FID against the 50,000 train2017 sources,

n=50{,}000

per cell.

Tokenizer	SigLIP2 $\downarrow$	MAE $\downarrow$	DINOv2 $\downarrow$
MAETok	85.8	2.03	953
InterpTok (41k)	34.6	0.52	315
InterpTok (82k)	36.4	0.70	311

Appendix

Table 10: Decoded MS-COCO midpoints in three feature spaces ,

n=50{,}000

midpoints of random pairs of different train2017 images, against the COCO sources. Values are comparable within a column only.

Tokenizer	Params (M)	Tokens	rFID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	rFID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
			ImageNet				MS-COCO
SD-VAE	84 †	1024	0.62 †	25.00	0.805	0.0630	0.801	24.76	0.817	0.0609
MAETok	174	128	0.582	23.55	0.760	0.0917	0.863	23.25	0.770	0.0913
InterpTok	217	128	0.597	24.02	0.775	0.0819	1.013	23.65	0.784	0.0817

Appendix

Table 11: Reconstruction on ImageNet and MS-COCO. The ImageNet validation set and MS-COCO train2017,

n=50{,}000

per cell. † Quoted from Chen et al. (2025a) .

FID $\downarrow$
	ImageNet									MS-COCO
	1 code	adjacent codes, same class				codes of random classes				2, random
		$k{=}2$	$3$	$5$	$10$	$k{=}2$	$3$	$5$	$10$
MAETok	0.57	35.12	72.80	121.02	181.21	87.49	143.46	186.22	238.52	85.21
InterpTok	0.57	3.79	6.71	18.59	76.23	13.78	40.81	124.85	219.07	13.61
Inception Score $\uparrow$

Appendix

Table 12: Image quality of decoded mixtures. Equal-weight code mixtures on ImageNet and MS-COCO against each corpus’s own 50,000 originals,

n=50{,}000

tuples per cell. Adjacent codes are those of consecutive validation images of one class. sFID, precision and recall are computed with the ADM evaluator ( Dhariwal & Nichol, 2021 ) , precision and recall after Kynkäänniemi et al. (2019) .

	near pairs				random pairs
Path	PPL	share	$d_{\mathrm{mid}}$	ordered $\uparrow$	PPL	share	$d_{\mathrm{mid}}$	ordered $\uparrow$
SD-VAE	0.796	0.174	0.739	61.4%	0.899	0.186	0.722	58.4%
MAETok	0.665	0.167	0.623	99.8%	0.799	0.172	0.636	99.4%
InterpTok	0.779	0.197	0.603	99.1%	1.056	0.203	0.628	94.8%

Appendix

Table 13: Movement along decoded paths. Path length in the spirit of Karras et al. (2019) with step shares and source distances, over nine equally spaced positions of the straight path,

n=5{,}000

per pair set.

Path	CMMD $\downarrow$
InterpTok
line in the latent space	171.3
autoencoder crossfade	462.4
MAETok
line in the latent space	508.5
autoencoder crossfade	493.4

Appendix

Table 14: Image quality along paths. CMMD in thousandths of the images decoded along the straight path between the codes of two images of different classes, against held-out real images,

n=25{,}000

paths per row. The reconstruction ceiling scores each tokenizer’s reconstructions of real images.

Position	MAETok	InterpTok (41k)
$\alpha=0.625$	10.17	5.35
$\alpha=0.75$	6.12	5.62
$\alpha=0.875$	4.76	5.25

Appendix

Table 15: Position along the path and interpolation geometry , FID on

n=50{,}000

decoded images per cell. In panel b, near pairs each code with its nearest validation code in the latent space of the model, as in Table 31 .

SiT-B on MAETok’s codes, one generator
gFID $\downarrow$	250	64	32	16
MAETok	11.61	12.33	13.88	20.56
$+$ our decoder	8.56	8.99	9.92	15.33
$+$ MAETok’s losses w/ interp. discriminator	9.71	10.16	11.26	16.92
$+$ MAETok’s losses	10.80	11.50	13.05	20.02
SiT-B on each tokenizer’s own codes

Appendix

Table 16: Generation quality with the stochastic and the deterministic sampler. gFID of the class-conditional SiT-B generators of Sec. 4.1 and of the released SiT-XL against the ADM statistics of ImageNet 256 without guidance,

n=50{,}000

per cell. Without interp. and without reg. are the variants of Table 3 trained without interpolation and without the regularization losses, the rows w/ reg. and w/ interpolation there, and

+

marks a decoder placed on the codes of the row above.

a. distance to the average code of its own class, relative to 250 steps
Sampling steps	250	32	16	8
MAETok	1	0.97	0.93	0.85
InterpTok	1	0.96	0.91	0.82
without interp.	1	0.96	0.91	0.82
without reg.	1	0.96	0.91	0.83
MAETok, stochastic	1	0.98	0.96	0.96

Appendix

Table 17: Finished codes and class averages with fewer steps. Distance of the finished generated codes to the average code of their own class and to the nearest average code of another class, relative to 250 steps, and the share of codes whose nearest class average is their own. 50,048 generated codes per generator, with the deterministic sampler unless the row says stochastic. Without interp. and without reg. are the rows w/ reg. and w/ interpolation of Table 3 .

Figure 21: The generator’s prediction over the sampling steps. The solid line is the distance from the generator’s prediction of the finished code to the average code of its own class, and the dashed line the distance to the nearest average code of another class, at each of 250 steps of the deterministic sampler, averaged over 50,048 generations per generator. Both are divided by the mean distance of the finished codes from the average of all validation codes. The dot marks the step from which the prediction of MAETok lies farther from the average code of its own class than from the nearest average code of another class.

Table 38

a. within-class spread, relative to 250 steps
Sampling steps	250	32	16	8
MAETok	1	0.96	0.92	0.84
InterpTok	1	0.95	0.90	0.79
without interp.	1	0.95	0.89	0.78
without reg.	1	0.96	0.91	0.81
b. share of the ten nearest validation codes that carry the code’s label, percent

Appendix

Table 20: Generated codes at four numbers of sampling steps. 50,048 generated codes per generator in blocks a to e and 50,000 in blocks f to i and in the stochastic rows of block b, deterministic sampler unless the block or row says otherwise. Block d is the distance of the generated code at 8 steps from the generated code at 250 steps from the same starting noise, in percent of the length of the code.

	MAETok	InterpTok	without interp.	without reg.
Linear-probe accuracy, percent	73.0	77.9	77.0	63.5
Nearest neighbor of the same class, percent	30	61	65	17
Nearest-neighbor distance, relative	1	1.6	1.6	0.9
Own-class codes among the 100 nearest to a class average, of at most 50	17	37	39	5
Nearest class average is its own, percent	55	73	73	43
in the unnormalized latent space of the decoder

Appendix

Table 21: The latent space of each encoder. The codes of the 50,000 ImageNet validation images, in the normalized latent space of the generator unless a block says otherwise. The linear probe follows the MAE procedure of Table 28 . Without interp. and without reg. are the rows w/ reg. and w/ interpolation of Table 3 . † Quoted from Zheng et al. (2026) under its own procedure.

Sampling steps	250	32	16	8	250	32	16	8
	Class acc., percent				Sim.
MAETok	60	60	61	58	0.42	0.43	0.43	0.45
$+$ our decoder	62	64	66	71	0.42	0.43	0.44	0.47
InterpTok	61	67	73	83	0.43	0.45	0.48	0.56

Appendix

Table 22: Class accuracy and similarity of the decoded images , 50,000 decoded images per cell, deterministic sampler unless a block says otherwise. Class acc. is the share of the decoded images whose top-1 class under a ResNet-50 trained on ImageNet is the conditioned label, and sim. is the mean cosine similarity of the features of those images within a class.

gFID $\downarrow$	seed 0	seed 1
MAETok	11.51	11.38
InterpTok	8.51	8.28

Appendix

Table 23: A second training seed. SiT-B/1D after 400,000 steps, 250 sampling steps without guidance,

n=50{,}000

per cell, one sampling run per generator. Seed 0 is the generator of Table 2 , whose draw there gives 11.61 and 8.48.

41,000 fine-tuning steps
Decoder	250	64	32	16	FD-DINOv2 $\downarrow$	CMMD $\downarrow$	2nd draw
MAETok	2.44	2.76	3.44	7.49	125.6	0.313	2.49
MAETok’s losses	2.61	2.88	3.55	7.81	108.2	0.226	2.63
InterpTok	2.24	2.39	2.79	5.80	88.1	0.171	2.24
13,000 fine-tuning steps
MAETok’s losses w/ noise augmentation	2.70	–	–	–	124.4	0.258	–

Appendix

Table 24: The released SiT-XL of Chen et al. (2025a) decoded by each decoder. gFID at 250, 64, 32 and 16 sampling steps, FD-DINOv2 and CMMD ( Jayasumana et al., 2024 ) at 250 steps, and the gFID of a second draw at 250 steps. Every decoder keeps MAETok’s frozen encoder. The draw of Table 2 gives 2.46 for MAETok’s decoder and 2.26 for ours. A dash marks a value that is not measured. Bold is the best value in its column among MAETok’s decoder and the 41,000-step decoders.

n=50{,}000

per cell.

Panel of Figure 22 — Figure 22: Uncurated samples. SiT-B on MAETok’s codes on the left and on InterpTok’s codes on the right, 250 steps of the stochastic sampler without guidance. One class per row, switch, black widow, bulbul, computer keyboard, rhinoceros beetle, Mexican hairless, American chameleon and loupe, with the first eight samples of each.

Figure 23: Decoded paths under a smaller code. In each row, the two dark-outlined ends decode the codes of two images generated with Z-Image-Turbo ( Z-Image Team et al., 2025 ) and the seven tiles between them mixtures of those two codes.

Figure 24: Decoded mixtures of four codes under a smaller code. Four unrelated squares, one per quadrant, each tile decoded from a mixture of its four corner codes with weights bilinear in the position. Only the dark-outlined corners hold images generated with Z-Image-Turbo ( Z-Image Team et al., 2025 ) .

Tokenizer	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
InterpTok	24.02	0.775	0.0819
MAETok	23.55	0.760	0.0917
Smaller code	19.31	0.615	0.1735

Appendix

Table 25: Reconstruction under a smaller code. ImageNet, 50,000 images, SSIM in MAETok’s convention.

Configuration	Value
Data and pairing
Training images	ImageNet-1k train at $256\times 256$ in $[-1,1]$
Pair sampling	each epoch, every image is paired with its most similar unpaired image among its $k$ nearest neighbors, $k=8,\,8,\,16,\,16,\,32,\,32,\,64,\,64,\,128,\,128$ over epochs 1–10
$\alpha$ sampler	Beta(2,2), one $\alpha$ per mixture
Tokenizer and batches
Base model	MAETok, encoder and decoder trainable

Appendix

Table 26: Tokenizer training configuration. The ImageNet fine-tune behind every reported InterpTok result.

Configuration	Value
Discriminator
Discriminator backbone	frozen DINO ViT-S/16 at 224 px, features from 5 layers, 196 tokens per layer
Discriminator heads	the interpolation discriminator’s head, which scores the decoded mixture beside one source, and the unconditional reconstruction discriminator’s head, both on the shared backbone
Discriminator parameters	25.9M trainable, frozen backbone 21.7M
Adversarial schedule	the discriminator trains from step 0 and its loss reaches the tokenizer after 1,500 steps, one discriminator update per tokenizer update, adaptive weight capped at 1.0
Adaptive weight	the VQGAN ratio $\\|\nabla\mathcal{L}_{\mathrm{rec}}\\|/(\\|\nabla\mathcal{L}_{G}\\|+10^{-4})$ at the decoder’s final linear layer, capped at 1.0

Appendix

Table 27: Discriminator, vision model and generator settings. The frozen backbones act during training only.

Split	Units	Source and pairing	Evaluation target
ImageNet validation	50,000 midpoints	val code to a sampled top-10 nearest neighbor in the train split, spherical path, $\alpha=0.5$ , the released evaluation code at its defaults, FID against the 50,000 validation originals	iFID
ImageNet validation	50,000 images	reconstruction, no pairing, the released evaluation code at its defaults for rFID and our own pixel evaluation, SSIM with a uniform window at data range 2, for the rest	rFID (Tables 1 and 11 ), PSNR, SSIM, LPIPS (Table 11 )
ImageNet validation	50,000 reference images	equal-weight barycenters of $k=2,3,5,10$ codes, same-class and cross-class, straight line	FID, sFID, precision and recall of decoded barycenters (Table 12 )
ImageNet validation	50,000 midpoints	random cross-class pairs under one seed, straight-line midpoint at $\alpha=0.5$ , ADM evaluator against the 50,000 validation originals	FID of decoded midpoints through the fine-tune (Table 8 )
ImageNet validation	5,000 near and 5,000 random pairs	near pairs at DINOv3 cosine 0.6 to 0.8 and random pairs of different classes under one seed, the straight line decoded at nine equally spaced positions, LPIPS with AlexNet between consecutive decodes and to both source reconstructions	perceptual path length, largest-step share, midpoint distance and ordered fraction (Table 13 )
MS-COCO train2017	50,000 images	reconstruction, no pairing	rFID, PSNR, SSIM, LPIPS (Table 11 )

Appendix

Table 28: Evaluation datasets and protocols. One row per measurement, with its corpus, sample count, pairing and reference.

Quantity	reported	ours	rel. diff.	source
iFID, MAETok	14.06 †	14.42	$+$ 2.6%	comparison table released by Xu et al. (2026)
iFID, DM-VAE	8.14 †	8.29	$+$ 1.8%	comparison table released by Xu et al. (2026)
rFID, MAETok, iFID protocol	0.58 †	0.5819	$+$ 0.3%	released leaderboard of Xu et al. (2026)
rFID, MAETok, MS-COCO	4.87 †	5.05	$+$ 3.8%	Table 4 of Chen et al. (2025a)

Appendix

Table 29: Reproducing published measurements. Each row re-runs one published measurement under the protocol that produced it. † The published value.

Item	Value
Evaluators
iFID and rFID	the released evaluation code of Xu et al. (2026) at its defaults
Inception features	the Inception weights of pytorch-fid and of torch-fidelity
Other FID comparisons	the ADM evaluator, against the validation originals for reconstruction and interpolation and the ADM ImageNet 256 reference statistics for generation
Pixel metrics	our own pixel evaluation, SSIM with a uniform window at data range 2, LPIPS with AlexNet
Encoder supervision, corpus and code dimensions

Appendix

Table 30: Evaluators and provenance of the quoted tokenizers. The evaluator of every measurement, and for each quoted tokenizer what supervised it, on which corpus and at how many code dimensions.

Other pixel conventions	MAETok	InterpTok	SD-VAE
SSIM, Gaussian window at data range 1, ImageNet	0.644	0.670	0.708
SSIM, Gaussian window at data range 1, MS-COCO train2017	0.656	0.682	0.724

Appendix

Table 31: Other pixel conventions and other pairings. ImageNet validation at

n=50{,}000

and InterpTok (82k) unless a row states otherwise.

Image Tokenization

Image tokenization converts images into compact discrete or continuous visual units for efficient generation, compression, and multimodal reasoning.

See all 57 papers

What Makes a Good Medical Image Tokenizer? Rethinking Reconstruction and Generation in Medical Image Tokenization
Sep 21, 2026Niklas Bubeck, Yundi Zhang, Vasiliki Sideri-Lampretsa +4Image Tokenization Medical Image Synthesis
School of Computation, Information and Technology, Technical University of Munich, Germany · Munich Center for Machine Learning, Technical University of Munich, Germany · School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Germany +3
MacTok: Robust Continuous Tokenization for Image Generation
Mar 31, 2026Hengyu Zeng, Xin Gao, Guanghao Li +5Image Tokenization Visual Token Compression
Fudan University
Semantic One-Dimensional Tokenizer for Image Reconstruction and Generation
Mar 17, 2026Yunpeng Qu, Kaidong Zhang, Yukang Ding +2Image Tokenization Image Reconstruction
Tsinghua University, Beijing, China · Alibaba Group, Beijing, China

Sep 21, 2026J/K move · Enter open · S save

What Makes a Good Medical Image Tokenizer? Rethinking Reconstruction and Generation in Medical Image Tokenization

Niklas Bubeck, Yundi Zhang, Vasiliki Sideri-Lampretsa, Julian McGinnis+3

School of Computation, Information and Technology, Technical University of Munich, Germany · Munich Center for Machine Learning, Technical University of Munich, Germany · School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Germany+3

Image Tokenization Medical Image Synthesis Image Generation Evaluation Medical Image Representation Learning

Open paper AskPDF arXiv

Latent diffusion models now dominate medical image generation, and every such pipeline rests on a tokenizer that compresses images into the latent codes for image generation to operate on. Thereby, the tokenizer choice bounds every downstream task from reconstruction fidelity and generation quality to the representations available for downstream analysis. Yet, medical imaging pipelines routinely utilize tokenizers from natural imaging on the hypothesis that their behavior carries over. However, this is an assumption never tested in the medical imaging regime, where datasets are orders of magnitude smaller and images exhibit far lower inter-sample variance. We present a systematic evaluation of medical image tokenizers evaluating thirty configurations across ten model families on twelve datasets at three compression factors, spanning reconstruction, generation, latent geometry, downstream classification, and memorization. We find that (1) performance on image reconstruction and generation strongly correlate, unlike prior reports on natural images; (2) modern tokenizers use nearly all of their codebook entries, but still leave most of the latent space unused; (3) training-set memorization is mild and is further suppressed by stronger latent space compression; and (4) discrete quantization can largely preserve downstream classification, with lookup-free schemes being the main exception.

Abstract

Image Tokenization

Explore similar work

What Makes a Good Medical Image Tokenizer? Rethinking Reconstruction and Generation in Medical Image Tokenization

MacTok: Robust Continuous Tokenization for Image Generation

Semantic One-Dimensional Tokenizer for Image Reconstruction and Generation