Image tokenizers play a central role in modern generative models, where the structure of the latent space critically determines the downstream generation performance. A key but underexplored property of effective latent representations is spectral organization, the ability to encode information across frequency components. In this work, we introduce structured state-space regularization, a principled approach to inducing spectral structure in latent spaces. We derive a regularization objective by revisiting state-space models (SSMs) as systems mimicking a basis function's behavior. This perspective reveals that hidden states of SSMs are induced to capture the frequency components, resulting in a novel regularizer that enforces the latent space to capture spectral structure of images. Experiments demonstrate that our regularizer improves the generative performance of image tokenizers while incurring only minimal loss in their reconstruction fidelity.
Figures & tables
Figure 1: Different choice of (c(⋅),θ(⋅)) and the resulting update rules. Existing SSMs update their hidden state based on Eq. ( 4 ), which is derived from the coefficient dynamics based on the choice (a) . One can choose a different combination of c(⋅) and θ(⋅) such as (b) , to derive new dynamics and formulate a new SSM framework.
Figure 2: State-space regularization applied to an image tokenizer. With probability α , the update Eq. ( 21 ) is applied to the latent representation of the input images to produce z^2 and I^τ2 . The network is trained to match (z^2,I^τ2) to (z2,Iτ2) .
Table 3
Figure 3: Generation results comparison using the Flux tokenizer. With only marginal loss in reconstruction quality, our method improves the generative performance of the image tokenizer.
Figure 4: Effect of spatial mean-centering function C. The one trained without C shows latents with higher latent norm, and exhibits tremendous degree of KL loss.
Figure 5: Latent channels encoding different frequency bands. The regularized tokenizer is able to reconstruct recognizable images solely from using three low-frequency channels.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Visualization of derived A matrices. We set H=W=8 . Every matrix we derived is very sparse, which enables efficient matrix-vector multiplication. To maintain training stability, we normalize the matrix to have maximum absolute value of 1.
Tokenizer
Num
batch
KL
Warmup init.
Base
Min
A,Δ
Gradient
params
size
weight
learning rate
learning rate
learning rate
learning rate
clipping norm
Flux
83.8M
32
10−6
1.0×10−7
1.0×10−5
1.0×10−6
1.0×10−6
50.0
Cosmos
81.5M
32
-
1.0×10−7
1.0×10−4
1.0×10−5
1.0×10−5
50.0
Appendix
Table 3: Implementation details of the reconstruction experiments from Sec. 4 .
Tokenizer
τmax
ΔI
Δ init.
Max A init.
dz(⋅,⋅)
dI(⋅,⋅)
λz
λI
λMAE
λLPIPS
Flux
8.0
4.0
0.1
16.0
MSE
MAE & LPIPS
0.0
1.0
1.0
0.1
Cosmos
8.0
4.0
0.1
1.0
MSE
MAE & LPIPS
5.0
1.0
1.0
0.1
Appendix
Table 4: Hyperparameters of the regularization framework
Figure 7: Image displayed with respective τ values.
Figure 8: (Left) Illustration of the channel-to-coefficient index mapping. (Right) Channels illustrated with colors, where darker color indicates that a channel encodes lower frequency.
A type
PSNR ↑
SSIM ↑
LPIPS ↓
rFID ↓
gFID ↓
sFID ↓
IS ↑
pre. ↑
re. ↑
Fourier
31.54
0.9101
0.0370
0.90
5.25
6.99
207.14
0.85
0.45
Legendre
31.57
0.9091
0.0366
0.92
5.22
7.65
206.70
0.84
0.45
Chebyshev
31.56
0.9087
0.0368
0.90
5.28
7.27
199.11
0.83
0.46
Hermite
31.60
0.9092
0.0363
0.92
5.34
7.03
196.49
0.84
0.46
random
N/A (does not converge)
-
-
-
-
-
Appendix
Table 5: Ablation for the type of A
α
PSNR ↑
LPIPS ↓
SSIM ↑
rFID ↓
gFID ↓
sFID ↓
IS ↑
pre. ↑
re. ↑
0
31.70
0.0354
0.9105
0.87
16.12
7.22
89.27
0.68
0.49
0.25
31.53
0.0373
0.9082
0.94
13.17
6.71
105.99
0.71
0.48
0.5
30.95
0.0459
0.9018
1.01
13.73
8.19
111.65
0.69
0.43
0.75
29.83
0.0649
0.8884
1.28
15.48
8.72
112.02
0.68
0.39
Appendix
Table 6: Ablation for the regularization strength α
Figure 9: Additional results from the channel revealing experiment
Figure 10: Generation results from the Cosmos tokenizer
Figure 11: Additional generation results from the Flux tokenizer
Image tokenizers convert pixels into compact latent sequences for efficient generative modeling. Most approaches arrange latents on a downsampled two-dimensional grid, tying sequence length to image resolution. We compare two variable-length strategies: computationally efficient patch-derived tokens, which inherit explicit spatial organization, and more demanding register tokens, which have no predefined spatial layout. Our reconstruction results reveal that patch-derived tokens perform substantially better under extreme compression, whereas register tokens achieve slightly better reconstruction quality at intermediate budgets. At high token budgets, both approaches achieve similar reconstruction quality. For downstream generative tasks, register tokens outperform patch-derived tokens. Adding representation alignment with external pretrained visual encoders substantially improves the generative performance of patch-derived tokens, allowing them to match or outperform register tokens depending on the token budget. We further find that both approaches learn spatially localized representations. Patch-derived tokens largely retain their explicitly imposed spatial organization, while register tokens develop an implicit two-dimensional spatial structure despite having no predefined layout. Representation alignment reduces this spatial organizational constraint, enabling individual tokens to exert a stronger non-local influence.
Philipp Schenkel, Tobias Schürmann, Omer Sahin Tas
FZI Research Center for Information Technology · Karlsruhe Institute of Technology
Visual generative models based on latent space have achieved great success, underscoring the significance of visual tokenization. Mapping images to latents boosts efficiency and enables multimodal alignment for scaling up in downstream tasks. Existing visual tokenizers primarily map images into fixed 2D spatial grids and focus on pixel-level restoration, which hinders the capture of representations with compact global semantics. To address these issues, we propose SemTok, a semantic one-dimensional tokenizer that compresses 2D images into 1D discrete tokens with high-level semantics. SemTok sets a new state-of-the-art in image reconstruction, achieving superior fidelity with a remarkably compact token representation. This is achieved via a synergistic framework with three key innovations: a 2D-to-1D tokenization scheme, a semantic alignment constraint, and a two-stage generative training strategy. Building on SemTok, we construct a masked autoregressive generation framework, which yields notable improvements in downstream image generation tasks. Experiments confirm the effectiveness of our semantic 1D tokenization. Our code will be open-sourced.
Yunpeng Qu, Kaidong Zhang, Yukang Ding +2
Tsinghua University, Beijing, China · Alibaba Group, Beijing, China
Attention-based models often develop attention sinks, where a small number of tokens repeatedly attract attention and accumulate unusually large activations. In vision transformers, these outliers are closely related to registers, which have been diagnostically linked to global, low-frequency image structure. Existing work has largely studied registers through interpretability analyses and linear probes, leaving open whether they can be operationalized as plug-and-play signals for generation without retraining. We revisit this question in tokenized image generation. Using OpenCLIP and DINOv2 on ImageNet, we find that test-time register features exhibit stronger low-frequency concentration than both [CLS] readouts and patch-mean features, and show a consistent (albeit moderate) correlation with pixel-space DCT low-frequency energy. Motivated by these diagnostics, we introduce RegToken, a training-free procedure that converts register structure into a small set of global prior tokens by (i) NFN-based layer localization, (ii) TokenRank-guided subspace extraction, and (iii) a projection-and-conservation update on the register subspace. Inserted into a frozen compact 1D token generation pipeline, RegToken improves ImageNet generation and alignment metrics (e.g., FID-5k 20.5 to 20.1, SigLIP 3.6 to 3.9) without modifying pretrained weights, and accelerates test-time optimization (Steps@τ 74 to 52). Overall, our results suggest that structures often viewed as attention artifacts can be repurposed as lightweight global priors for tokenized generation.
Cheng-Yao Hong, Yifan Wang, Yuewei Lin +1
Stony Brook University · Brookhaven National Laboratory