Adaptive image tokenizers compress an image into a variable number of tokens matched to its complexity, and recent work makes this allocation controllable: conditioned on a target reconstruction error, the tokenizer itself decides how many tokens an image needs. However, the fidelity measure guiding this allocation shapes which aspects of the image are preserved under compression. The pixel-wise error used in prior work is blind to the spatial structure that carries an image's meaning, and under heavy compression such tokenizers are prone to abruptly discarding tokens that encode essential content. We introduce GIST (Guided Image Semantic Tokenizer), an adaptive tokenizer that can be steered toward a requested semantic fidelity. Its token allocation is jointly controlled by pixel-level and semantic target errors, with semantic error measured in the feature space of a frozen DINOv3 encoder. This joint control enables semantic protection: with the semantic target fixed at zero, the reconstruction target can be relaxed to discard pixel detail while preserving what the image depicts. In a controlled comparison on ImageNet-100, GIST outperforms its pixel-controlled counterpart KARL at every token count, with up to 30% lower LPIPS and 37% lower DreamSim. On average, lowering its semantic target reduces LPIPS 2× faster per token than lowering the reconstruction target, and up to 5× in the low-token regime. Semantic fidelity is thus an effective control signal for adaptive tokenization: it lets a tokenizer see, and so preserve, what an image depicts.
Figures & tables
Figure 1: Semantic control unlocks semantic protection. ( Left ) GIST can be steered toward a requested semantic fidelity in DINOv3 feature space, as illustrated here with a red circle (top) and a hummingbird (bottom). Raising the requested fidelity steers the encoding from little more than the image’s color distribution to a reconstruction that preserves what the image depicts. ( Right ) Requesting full semantic fidelity turns this control into a guardrail: without it, compression into fewer tokens disintegrates the butterfly; with it, the subject remains recognizable down to 16 tokens.
Figure 2: GIST makes semantic fidelity an explicit control signal. ( Left ) KARL ( Duggal et al., 2025a ) steers the tokenizer with a single reconstruction target ϵrec for the pixel-wise L1 error between input I and its reconstruction I^ : encodings become pixel-faithful, but the error is blind to semantics, so tokens are prone to be wasted on redundant detail. ( Right ) GIST keeps ϵrec and adds a second, semantic target ϵsem for the error between DINOv3 CLS features of I and I^ : lowering ϵsem demands tokens that preserve what the image depicts, so the encoder can discard redundant pixel detail.
Figure 3: Adding semantic control Pareto-improves reconstruction quality at every token count. ( Left ) Pareto fronts on the ImageNet-100 validation set, obtained by varying ϵrec for KARL † and GIST (fixing ϵsem=0 ), and by varying ϵsem for GIST-1. The shaded regions highlight GIST’s reduction in reconstruction error relative to KARL † at matched token counts. ( Right ) Tokens used, LPIPS, and DreamSim for KARL † and GIST at equal reconstruction targets, with the best values in bold. We report the same comparison on the full set of reconstruction quality metrics in Appendix D .
Figure 4: Semantic protection keeps the subject recognizable under heavy token compression. For each sample ( Goldfish reef , top; Black swan , bottom), the upper row shows KARL † reconstructions as we vary ϵrec , and the lower row shows GIST reconstructions with ϵsem fixed at 0 and ϵrec varied over the same values. The far-left image is the original; token budget is fixed at T=256 ; per-cell strips report the active token count and LPIPS. As more tokens are halted, KARL † disintegrates the subject into artifacts that retain little beyond its color distribution, while GIST keeps it recognizable. See Appendix G for additional samples.
Figure 5: Semantic control reduces LPIPS 2× faster per token than pixel-level control. LPIPS trajectories of GIST as we vary one target with the other held fixed (shown by color). Lowering ϵrec (left) traces long, shallow curves that span nearly the full token range, whereas lowering ϵsem (right) drops LPIPS sharply over only a few added tokens.
Figure 6: Semantic conditioning makes tokens more semantically informative. (a) kNN accuracy of GIST, KARL † , and GIST-1 as we vary each method’s target values (same setup as Figure 3 ), with accuracies linearly interpolated at matched mean active token counts. GIST and GIST-1 outperform KARL † at every token count, and GIST-1 performs best despite performing on par with KARL † and worse than GIST on reconstruction quality. (b) kNN accuracy trajectories of GIST as we vary one target with the other held fixed (shown by color). Lowering ϵsem increases kNN accuracy at a much steeper rate than lowering ϵrec . Accuracies are computed by mean pooling over the active tokens; Appendix I reports the same comparison under max pooling.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
LPIPS ↓
DreamSim ↓
L1 Loss × 10 ↓
SSIM ↑
32
64
128
256
32
64
128
256
32
64
128
256
32
64
128
256
KARL (published) ( Duggal et al., 2025a )
0.30
—
0.19
0.14
0.28
—
0.16
0.11
1.13
—
0.80
0.70
0.34
—
0.44
0.49
KARL (public ckpt., our eval)
0.30
0.24
0.19
0.14
0.28
0.21
0.16
0.11
1.14
0.98
0.80
0.70
0.34
0.38
0.45
0.49
KARL † (reproduction, seed 1)
0.33
0.26
0.19
0.15
0.31
0.23
0.15
0.11
1.24
1.05
0.87
0.74
0.31
0.35
0.41
0.46
KARL † (reproduction, seed 2)
0.33
0.26
0.19
0.15
0.31
0.22
0.15
0.11
1.24
1.03
0.88
0.75
0.31
0.36
0.41
0.46
KARL † (reproduction, seed 3)
0.33
0.26
0.19
0.14
0.31
0.23
0.15
0.11
1.24
1.04
0.87
0.74
0.31
0.36
0.41
0.46
Appendix
Table 3: Reproduction of the KARL baseline. We compare KARL’s published numbers ( Duggal et al., 2025a ) against the official checkpoint and KARL † (retrained from the official, unmodified code). Our evaluation of the official checkpoint (row 2) reproduces the published numbers (row 1) to within 0.01 . The retrained models (rows 3–5) perform slightly worse than the official checkpoint, mainly on L1 and SSIM. Every std is at most 0.009 , well below this gap, so the gap is systematic rather than an artifact of seed variance. The last two rows report GIST and GIST-1; both match KARL † . Every model trained from the released code thus shares the same base capacity, below the official checkpoint by the same gap.
LPIPS ↓
DreamSim ↓
L1 Loss × 10 ↓
SSIM ↑
Method
Train
32
64
128
256
32
64
128
256
32
64
128
256
32
64
128
256
Controlled: trained by us under identical conditions
KARL †
IN-100
0.33
0.26
0.19
0.15
0.31
0.22
0.15
0.11
1.24
1.04
0.87
0.74
0.31
0.36
0.41
0.46
GIST-1
IN-100
0.33
0.26
0.19
0.15
0.30
0.22
0.15
0.11
1.24
1.04
0.87
0.75
0.31
0.36
0.41
0.46
GIST
IN-100
0.33
0.26
0.19
0.14
0.30
0.22
0.15
0.11
1.23
1.03
0.88
0.75
0.31
0.36
0.41
0.46
Reference: independently trained — not directly comparable
Appendix
Table 4: Reconstruction fidelity of adaptive tokenizers across token budgets on ImageNet-100. Top block: models we train under one identical protocol (our controlled comparison); for KARL † , we report the mean over its three retrains ( Table 3 ). Bottom block: Independently trained tokenizers, re-evaluated by us under the same protocol and reported for reference only. They differ from the controlled block in architecture, training objective, training data, and scale. In the top block, all methods perform comparably because GIST’s advantage over the baselines comes only from varying its two targets, which are both fixed at 0 here. In the bottom block, the official KARL checkpoint performs best across most metrics.
Figure 7: Tokens vs. reconstruction quality metrics on ImageNet-100. Pareto fronts on the ImageNet-100 validation set for the full set of reconstruction quality metrics ( L1 , LPIPS, SSIM, PSNR, and DreamSim), obtained by varying ϵrec for KARL † and GIST (fixing ϵsem=0 ), and by varying ϵsem for GIST-1. GIST outperforms KARL † on every metric at any token count.
Figure 8: The reconstruction gains extend to multi-object scenes. Token-fidelity Pareto fronts on the COCO 2017 validation set, using the ImageNet-100-trained checkpoints without adaptation. As in Figure 3 , we vary ϵrec for KARL † and GIST (fixing ϵsem=0 ), and vary ϵsem for GIST-1. The shaded regions highlight GIST’s reduction in reconstruction error relative to KARL † at matched token counts.
Figure 9: The reconstruction advantage persists in crowded scenes. Same evaluation as Figure 8 , with images grouped by the number of annotated object instances in the original image. Each column corresponds to an object-count range, and n denotes the number of images in that group. GIST outperforms KARL † in every group, with the largest relative gains in sparse scenes at low token counts.
Figure 10: Reconstruction quality metrics vs. input token budget on ImageNet-100. We compare GIST, GIST-1, and KARL † across all five reconstruction quality metrics ( L1 , LPIPS, SSIM, PSNR, DreamSim) on the ImageNet-100 validation set as we vary the input token budget T , with all targets fixed at zero. The three models perform on par on every metric, in contrast to the comparison under varying targets ( Figures 3 and 7 ), where GIST Pareto-improves reconstruction quality over KARL † .
Figure 11: Synthetic and abstract stimuli. KARL † (top of each pair) vs. GIST (bottom of each pair) reconstructions across the ϵrec spectrum. From top to bottom: red circle on white , checkerboard 2×2 , checkerboard 8×8 , and color wheel .
Figure 12: In-distribution samples (ImageNet-100). Same setup as Figure 11 . From top to bottom: black swan , rooster , goldfish , and peacock .
Figure 13: Out-of-distribution natural photographs. Same setup as Figure 11 . From top to bottom: blue morpho butterfly , pink peony , goldfish reef , and hummingbird .
Figure 14: Difficult cases: textures and high-frequency detail. Same setup as Figure 11 . From top to bottom: peacock feather , Mandelbrot fractal , text paragraph , and low-frequency noise .
Figure 15: Synthetic and abstract stimuli. GIST reconstructions while varying ϵsem , with ϵrec=0.24 and T=256 tokens. From top to bottom: red circle on white , checkerboard 2×2 , checkerboard 8×8 , and color wheel .
Figure 16: In-distribution samples (ImageNet-100). Same setup as Figure 15 . From top to bottom: black swan , rooster , goldfish , and peacock .
Figure 17: Out-of-distribution natural photographs. Same setup as Figure 15 . From top to bottom: blue morpho butterfly , pink peony , goldfish reef , and hummingbird .
Figure 18: Difficult cases: textures and high-frequency detail. Same setup as Figure 15 . From top to bottom: peacock feather , Mandelbrot fractal , text paragraph , and low-frequency noise .
Figure 19: The kNN comparison is robust to the choice of pooling. kNN accuracy of GIST, KARL † , and GIST-1 on the ImageNet-100 validation set, obtained by varying ϵrec for KARL † and GIST (fixing ϵsem=0 ), and by varying ϵsem for GIST-1 (same setup as Figure 6(a) ). Accuracies are computed by max pooling (solid) and mean pooling (dotted) over the active tokens. The ordering of the three methods is the same under both poolings. For GIST-1, mean pooling yields a higher accuracy, and accuracy increases with the number of active tokens. For GIST and KARL † , max pooling yields a slightly higher accuracy, and accuracy remains approximately constant.
Figure 20: The kNN comparison is robust to the choice of k . kNN accuracy of GIST, KARL † , and GIST-1 on the ImageNet-100 validation set for k∈{10,20,100,200} (same setup as Figure 6(a) ). The ordering of the three methods is the same for every k : GIST-1 performs best, followed by GIST and KARL † , at any number of active tokens. Accuracy is highest at k=20 for all three methods and decreases slightly at k=200 .
Mean kNN top-1 (%)
Advantage over KARL †
k
KARL †
GIST
GIST-1
GIST
GIST-1
10
22.0
27.4
36.3
+5.4
+14.3
20
23.4
28.6
37.3
+5.2
+13.9
100
23.3
27.9
36.3
+4.6
+13.0
200
22.1
26.6
34.7
+4.5
+12.6
Appendix
Table 5: Mean kNN accuracy of KARL † , GIST, and GIST-1 for different values of k . Mean kNN accuracy of KARL † , GIST, and GIST-1 on the ImageNet-100 validation set for k∈{10,20,100,200} (same setup as Figure 6(a) ), averaged over the range of active tokens shared by all three methods ( 47.7 – 256 tokens). The last two columns report the advantage of GIST and GIST-1 over KARL † in mean accuracy. For every k , GIST-1 performs best, followed by GIST and KARL † . The advantage of GIST-1 over KARL † lies between 12.6 and 14.3 points, and that of GIST over KARL † between 4.5 and 5.4 points. Both advantages are largest at k=10 and decrease slightly as k grows.
Figure 21: The advantage of semantic control holds under a trained attentive probe. Both panels report the best validation top-1 accuracy over the 10 training epochs for each target configuration. (a) Attentive-probe accuracy of GIST, KARL † , and GIST-1 as we vary each method’s target values (same setup as Figure 6(a) ). The ordering from the kNN evaluation is preserved, with narrower but consistent gaps: GIST-1 and GIST outperform KARL † at every token count. (b) Attentive-probe accuracy trajectories of GIST as we vary one target with the other held fixed (shown by color). Lowering ϵsem increases probe accuracy at a much steeper rate than lowering ϵrec . Together, the probe shows that conditioning on ϵsem not only exposes semantic information more readily, but also makes the tokens carry more of it.
Method
Tokens
Best epoch
Final epoch
KARL †
64
55.2
54.8
GIST
64
60.3
59.9
GIST-1
64
61.2
60.5
KARL †
192
61.4
60.6
GIST
192
63.3
63.1
GIST-1
192
65.4
65.4
Appendix
Table 6: Best-epoch vs. final-epoch attentive-probe accuracy. Top-1 accuracy (%) at 64 and 192 tokens, obtained by linear interpolation between neighboring target configurations on each method’s curve. For each configuration, Best epoch uses the highest validation accuracy over the 10 training epochs, as in Figure 21(a) , while Final epoch uses the validation accuracy after epoch 10.
Accurate and effective discrete image tokenization is crucial for long image sequence processing. However, current methods rigidly compress all content at a fixed rate, ignoring the variable information density of images and leading to either redundancy or information loss. Inspired by information entropy, we propose TaTok, a Theoretically grounded adaptive image Tokenization framework. We rigorously identify two key drawbacks in existing methods: information insufficiency when reconstructing images with patch tokens alone, and information redundancy among patch tokens. To address these, we introduce global tokens that model mutual information across patch tokens, and a Dynamic Token Filtering (DTF) algorithm based on cumulative conditional entropy to eliminate redundancy. Experiments confirm TaTok's state-of-the-art performance, delivering a 1.3x gFID improvement and 8.7x inference speedup. By allocating tokens according to information richness, TaTok enables more compressed yet accurate image tokenization, offering valuable insights for future research.
Xiusheng Huang, Xin Jiang, Jun Zhao +2
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence
Image-token compression must reduce computation without hurting the semantics. We introduce Varest, guided by semantic fungibility: tokens within a coherent semantic region can be summarized coarsely, while object boundaries and literal text require finer distinctions. Object and text localizers guide variable-resolution merging, preserving spatial coverage and finer detail at boundaries and text. Learned [REL] registers then prune visual tokens after early decoder processing. Training across budgets enables one checkpoint per backbone. Across three backbones and 13 benchmarks, Varest outperforms 15 baselines in mean performance retention at every evaluated budget, retaining 86.10-92.33% of full-token performance with only 5.56% of the original layer-averaged image-token count.
Tianhao Fu, Austin Wang, Charles Chen +5
University of Toronto, Toronto, ON, Canada · Vector Institute, Toronto, ON, Canada · McGill University, Montreal, QC, Canada +2
Visual generative models based on latent space have achieved great success, underscoring the significance of visual tokenization. Mapping images to latents boosts efficiency and enables multimodal alignment for scaling up in downstream tasks. Existing visual tokenizers primarily map images into fixed 2D spatial grids and focus on pixel-level restoration, which hinders the capture of representations with compact global semantics. To address these issues, we propose SemTok, a semantic one-dimensional tokenizer that compresses 2D images into 1D discrete tokens with high-level semantics. SemTok sets a new state-of-the-art in image reconstruction, achieving superior fidelity with a remarkably compact token representation. This is achieved via a synergistic framework with three key innovations: a 2D-to-1D tokenization scheme, a semantic alignment constraint, and a two-stage generative training strategy. Building on SemTok, we construct a masked autoregressive generation framework, which yields notable improvements in downstream image generation tasks. Experiments confirm the effectiveness of our semantic 1D tokenization. Our code will be open-sourced.
Yunpeng Qu, Kaidong Zhang, Yukang Ding +2
Tsinghua University, Beijing, China · Alibaba Group, Beijing, China