Mitigating Premature Discretization with Progressive Quantization for Robust Vector Tokenization

Wenhao Zhao; Qiran Zou; Zhouhan Lin; Dianbo Liu

cs.LGMar 17, 2026

Mitigating Premature Discretization with Progressive Quantization for Robust Vector Tokenization

Authors: Wenhao Zhao, Qiran Zou, Zhouhan Lin, Dianbo Liu

Organizations: National University of Singapore, Singapore · Shanghai Jiao Tong University, Shanghai, China

Abstract

Vector Quantization (VQ) has become the cornerstone of tokenization for many multimodal Large Language Models and diffusion synthesis. However, existing VQ paradigms suffer from a fundamental conflict: they enforce discretization before the encoder has captured the underlying data manifold. We term this phenomenon Premature Discretization. To resolve this, we propose Progressive Quantization (ProVQ), which incorporates the dynamics of quantization hardness as a fundamental yet previously overlooked axis in VQ training. By treating quantization as a curriculum that smoothly anneals from a continuous latent space to a discrete one, ProVQ effectively guides the codebook toward the well-expanded manifolds. Extensive experimental results demonstrate the broad effectiveness of ProVQ across diverse modalities. We report improved reconstruction and generative performance on the ImageNet-1K and ImageNet-100 benchmarks, highlighting the ProVQ's boost for generative modeling. Furthermore, ProVQ proves highly effective for modeling complex biological sequences, establishing a new performance ceiling for protein structure tokenization on the StrutTokenBench leaderboard.

Figures & tables

Figure 1: The Premature Discretization and resulting optimization deadlock. During early training stages, grid mapping forces the embedding distribution to contract and align with a sub-optimal clustered code, while uninformative guidance of embeddings causes the codebook vectors to stagnate. This mutual constraint creates a rigid optimization deadlock, which traps the model in a local minimal state and prevents it from exploring the full, well-distributed latent manifold (right).

Figure 2: Empirical Validation on Synthetic 2D datasets. (a) Synthetic dataset composed by Disk shape data plus triangle data to make gridding mapping visible by edge of triangle. (b) Comparison of reconstruction performance over different configurations, demonstrating that both the Soft Transition and the full ProVQ (Soft Transition + Manifold) strategies consistently outperform the Vanilla VQ baseline. (c) Reconstruction improvement relative to Vanilla VQ. While a Soft Transition alone yields substantial gains ( $+11.9\%$ for Disk and $+30.7\%$ for Triangle), the integration of a Manifold Warmup further boosts performance, achieving a $+33.1\%$ improvement on the triangle dataset. These results underscore that decoupling continuous and discrete learning at early stage.

Figure 3: Comparison of Embedding and Codebook Dynamics during Training. (a) Vanilla VQ: Inward-curved embedding edges signify grid mapping and an optimization deadlock, preventing full manifold coverage. (b) Soft Transition: Relaxes initial constraints to partially mitigate embedding shrinkage and improve codebook migration. (c) ProVQ (Ours): Manifold warm-up followed by soft transition achieves precise topological alignment, effectively resolving the deadlock.

Latent	Tokenizer	rFID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	Perplexity $\uparrow$	Euc dist. $\uparrow$
16 $\times$ 16	LlamaGen	2.19	20.79	0.675	8580.30	1.42
16 $\times$ 16	+ ProVQ	1.86	20.92	0.682	8591.85	6.49
24 $\times$ 24	LlamaGen	0.94	21.94	0.726	11487.83	1.42
24 $\times$ 24	+ ProVQ	0.81	21.99	0.729	11551.56	6.49

Table 1: Tokenizer performance on ImageNet-1K. We compare the ProVQ with LlamaGen tokenizer across latent resolutions (

16\times 16

and

24\times 24

Type	Model	#Para.	gFID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
Diffusion	ADM ( Dhariwal and Nichol, 2021 )	554M	10.94	101.0	0.69	0.63
	CDM ( Ho et al., 2022 )	–	4.88	158.7	–	–
	LDM-4 ( Rombach et al., 2022 )	400M	3.60	247.7	–	–
	DiT-XL/2 ( Peebles and Xie, 2023 )	675M	2.27	278.2	0.83	0.57
VAR	VAR-d16 ( Tian et al., 2024 )	310M	3.30	274.4	0.84	0.51
	VAR-d20 ( Tian et al., 2024 )	600M	2.57	302.6	0.83	0.56

Table 2: Generative results on Imagenet1K(

256\times 256

). Notably, integrating the ProVQ tokenizer into LlamaGen-B and LlamaGen-L architectures leads to consistent improvements in gFID and Recall. These results demonstrate that the enhanced reconstruction fidelity and discrete bottleneck by ProVQ translate to improved generative quality and more robust distribution coverage of ground truth.

Task	Split	Baselines					Ours
		FoldSeek	ProTokens	ESM3	Van.VQ	AminoA.	Van.VQ	AminoA.
							+ ProVQ	+ ProVQ
Functional Site Prediction (AUROC%)
BindInt	Fold	53.18	44.66	44.30	47.25	47.11	48.95	48.28
	SupFam	46.20	86.05	90.77	86.71	90.53	91.04	91.55
BindBio	Fold	52.37	58.47	62.84	62.02	65.73	65.36	63.90

Table 3: Evaluation of ProVQ on StructTokenBench ( Yuan et al., 2025 ) . We compare the ProVQ progressive quantization strategy against several baselines: Van. VQ (Vanilla VQ based on ESM3) and AminoA. (AminoAseed) etc. Results demonstrate that the integration of ProVQ consistently improves both Vanilla VQ and AminoAseed. Notably, AminoA. + ProVQ achieves the highest average performance across all tasks, highlighted by a

9.69\%

improvement(from

46.01\%

55.70\%

) in structure property prediction.

Model	CASP14		CAMEO		Euc. Distance $\uparrow$
Model	UR% $\uparrow$	Perplexity $\uparrow$	UR% $\uparrow$	Perplexity $\uparrow$	Euc. Distance $\uparrow$
VanillaVQ	5.55	0.0339	5.60	0.0337	46.80
+ ProVQ	41.40	0.2985	43.19	0.3006	69.68
AminoAseed	64.45	0.4946	68.87	0.5119	42.71
+ ProVQ	78.36	0.6021	85.36	0.6276	43.67

Table 4: Codebook analysis across StructTokenBench (CASP14 and CAMEO datasets). Metrics include codebook utilization rate (UR%), normalized perplexity, and average pairwise Euclidean distance (Euc. Distance). Results indicate that ProVQ consistently improves both the efficiency and diversity of the discrete latent space across different Vanilla VQ and AminoAseed.

Base Model	Enhancement	rFID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	Perplexity $\uparrow$
SimVQ	—	4.08	20.33	0.614	8,157.35
SimVQ	+ Soft Transition	3.39	20.53	0.628	8,171.83
Vanilla	—	3.81	20.64	0.629	7,123.79
Vanilla	+ Soft Transition	3.49	20.45	0.624	8,530.51
Vanilla	+ Manifold Warmup (best)	3.66	20.62	0.636	8,442.90
Vanilla	+ Manifold Warmup (overfit)	3.64	20.47	0.628	8,460.61

Table 5: Ablation of Soft Transition and Manifold warmup on ImageNet-100 (

256\times 256

). The best indicates a manifold warmup phase where the autoencoder (AE) reaches its peak validation rFID. The overfit label denotes an extended warmup duration where the AE exhibits overfitting on the rFID.

Scheduler	rFID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	Perplexity $\uparrow$
cosine	3.23	20.48	0.627	8518.33
hard	3.40	20.49	0.633	8532.74

Table 6: Ablation for scheduler setting of soft transition. The cosine means cosine-annealing scheduler while hard scheduler maintains the

\alpha=1

. Results show that gradual reduction of

alpha

provides a soft landing and consequently improves reconstruction performance.

Vector Quantization

Vector quantization compresses vectors into discrete codebook entries for efficient storage, transmission, retrieval, and learned representations.

See all 122 papers

VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizers
Jul 21, 2026Xianghong Fang, Yuan Yuan, Dehan Kong +1Visual Tokenization Image Reconstruction
University of Toronto · 2Boston College · 3Vijil
Transporting Quantiles to Codebook: Scalable Vector Quantization without Codebook Collapse
Oct 26, 2026Yuwei Zeng, Zekun ShiVector Quantization Vector-Quantized Variational Autoencoder
National University of Singapore · SEA AI Lab
Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey
Jul 21, 2025Jindong Li, Yali Fu, Jiahong Liu +5Multimodal Large Language Models Vector Quantization
Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Jilin University, Changchun, China · The Chinese University of Hong Kong, Hong Kong, China +2

Jul 21, 2026J/K move · Enter open · S save

VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizers

Xianghong Fang, Yuan Yuan, Dehan Kong, Tim G. J. Rudner

University of Toronto · 2Boston College · 3Vijil

Visual Tokenization Image Reconstruction Vector Quantization

Open paper AskPDF arXiv

Vector Quantization (VQ) underpins modern discrete visual tokenization. However, training quantization modules for state-of-the-art VQ-based models requires significant computational resources which, in practice, all but prevents the development of novel, cutting-edge VQ techniques under resource constraints. To address this limitation, we propose {\bf VQ-Transplant}, a simple framework that enables plug-and-play integration of new VQ modules into frozen, pre-trained tokenizers by replacing their native VQ modules. Crucially, the proposed transplantation process preserves all encoder-decoder parameters, obviating the need for costly end-to-end retraining when modifying the quantization method. To mitigate decoder-quantization mismatch, we introduce a lightweight decoder adaptation strategy (trained for only 5 epochs on ImageNet-1k) to align feature priors with the new quantization space. In our empirical evaluation, we find that VQ-Transplant allows obtaining near state-of-the-art reconstruction fidelity for industry-level models like VAR while reducing the training cost by 95%. VQ-Transplant democratizes quantization research by enabling resource-efficient integration of novel VQ techniques while matching industry-level reconstruction performance.

Abstract

Vector Quantization

Explore similar work

VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizers

Transporting Quantiles to Codebook: Scalable Vector Quantization without Codebook Collapse

Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey