Organizations: Huazhong University of Science and Technology · Meituan LongCat · Shanghai Jiao Tong University · Peking University · Mohamed bin Zayed University of Artificial Intelligence · Tsinghua University · The Chinese University of Hong Kong
Discrete visual tokens represent vision in a discrete form analogous to language, offering a native interface for multimodal modeling under a shared next-token prediction objective. However, quantization error is widely assumed to cause unavoidable semantic loss, fundamentally limiting the semantic fidelity of discrete visual tokens and positioning continuous visual representations as an intrinsic upper bound. Using multimodal understanding as a proxy for semantic fidelity, we revisit this assumption and find quantization error unreliable, as discrete representations with substantially larger errors can achieve better understanding. Instead, the effective bit budget of discrete indices emerges as a key determinant of downstream performance. However, existing quantizers eventually saturate as additional codebooks contribute progressively less semantic information. As one approach to alleviating this saturation, we introduce Factorized Vector Quantization, which quantizes complementary, fixed-dimensional views of the visual feature with symmetric parallel codebooks. It continues to improve at larger bit budgets, ultimately matching continuous visual representations in multimodal understanding. Trained on industrial-scale multimodal data, our model achieves state-of-the-art performance among discrete-representation models, outperforming UniAR and Emu3-Chat by 22.8% and 59.2%, respectively, on a nine-metric average. To our knowledge, it is also the first discrete-representation model competitive with strong continuous-representation models such as Qwen3-VL, including on fine-grained OCR tasks. We further identify the key principles and a practical recipe enabling this competitiveness. These results and insights support the potential of discrete visual representations to underpin large-scale native multimodal foundation models. Code and models are available at https://github.com/hustvl/FVQ
Figures & tables
Figure 1: Scaling the bit budget of discrete visual representations. (a) Average multimodal understanding performance improves with the per-patch bit budget. FVQ continues to improve in the high-budget regime and reaches the continuous QwenViT baseline, whereas existing multi-codebook quantizers saturate. (b) Qualitative examples of text recognition at increasing bit budgets. Higher budgets progressively recover the fine-grained textual information required for correct predictions; green and red denote correct and incorrect answers, respectively.
Figure 2: Overview of the analyses in Section 2 : reconstruction error versus semantic fidelity (left) and bit-budget scaling across representative quantizers (right). Braw , Bcont , and Bdisc denote per-patch budgets for raw pixels, continuous features, and discrete indices. f , n , V , b , D , L , and C denote patch side length, channels, values per channel, bits per feature dimension, feature dimension, number of codebooks, and codebook size, respectively.
Quantization
Multimodal Understanding
Quantizer
C×L
MSE ↓
MathVista
RealWorldQA
MMStar
OCRBench
TextVQA
AI2D
ChartQA
DocVQA
Avg. ↑
VQ
16,384×1
0.027
39.4
52.4
40.5
367
51.3
67.1
54.2
68.2
51.2
RVQ (Odd)
16,384×4
0.246
41.4
56.2
41.7
615
66.9
75.1
63.5
79.5
60.7
RVQ (Even)
16,384×4
0.028
40.3
56.2
41.7
564
66.1
73.6
61.7
78.4
59.3
RVQ (All)
16,384×8
0.016
43.7
59.3
43.1
652
71.6
78.8
67.3
82.8
64.0
Table 1: Quantization error and multimodal understanding of discrete QwenViT representations. C and L denote the codebook size and number of selected codebooks, respectively. MSE measures feature-reconstruction error. Avg. is the unweighted mean over the eight benchmarks, with OCRBench divided by 10.
Figure 3: Overview of FVQ. During quantizer training, multiple dense projections produce complementary views of the same visual feature, which are quantized in parallel by independent codebooks. The selected codewords are combined through tight-frame synthesis for feature reconstruction. In the subsequent VLM training stage, the quantizer is frozen, and only the resulting discrete indices are mapped to newly initialized token embeddings for multimodal understanding. The nine-metric average is computed over MMMU, MathVista, MMStar, OCRBench/10, ChartQA, CharXiv-DQ, CharXiv-RQ, DocVQA, and InfoVQA.
Quantization
Multimodal Understanding
Quantizer
L
B
MathVista
RealWorldQA
MMStar
OCRBench
TextVQA
AI2D
ChartQA
DocVQA
Avg. ↑
VQ ( van den Oord et al., 2017 )
1
14
39.4
52.4
40.5
367
51.3
67.1
54.2
68.2
51.2
SimVQ ( Zhu et al., 2025b )
1
14
37.9
49.7
39.1
204
32.2
63.9
41.9
49.9
41.9
RVQ ( Lee et al., 2022 )
2
28
41.4
54.8
42.7
563
65.8
74.3
64.4
79.0
59.8
4
56
43.2
58.7
43.1
625
69.7
78.2
66.8
81.7
63.0
8
112
43.7
59.3
43.1
652
71.6
78.8
67.3
82.8
64.0
Table 2: Bit-budget scaling of discrete QwenViT representations under the shared experimental setting. All discrete variants quantize the same features with codebook size C=16,384 . L is the number of codebooks and B=Llog2C is the nominal per-patch bit budget. Bold indicates the best discrete result. Continuous denotes the original QwenViT representation.
Method
MMMU
MMMU-Pro
MathVista
RealWorldQA
MMStar
Continuous
LLaVA-OV-1.5-8B ( An et al., 2025 )
55.4
31.3
69.6
68.1
67.7
LLaVA-OV-2-8B ( An et al., 2026 )
–
–
–
69.7
64.8
InternVL3-8B ( Zhu et al., 2025a )
62.7
–
71.6
70.8
68.2
InternVL3.5-8B ( Wang et al., 2025 )
73.4
–
78.4
67.5
69.3
Qwen2.5-VL-7B ( Bai et al., 2025b )
58.6
38.3
68.2
–
63.9
Table 3: Comparison on STEM and general multimodal understanding benchmarks. For a fair comparison, results marked with * are obtained by us under the same evaluation setting as our model.
Method
OCRBench
OCRBenchV2 (en/zh)
ChartQA
CharXiv (DQ/RQ)
DocVQA
InfoVQA
OmniDocBench (en/zh) ↓
Continuous
LLaVA-OV-1.5-8B ( An et al., 2025 )
829
–
86.5
70.9/–
95.0
78.4
–
LLaVA-OV-2-8B ( An et al., 2026 )
782
–
85.9
–
95.2
74.4
–
InternVL3-8B ( Zhu et al., 2025a )
880
–
86.6
73.6/37.6
92.7
76.8
–
InternVL3.5-8B ( Wang et al., 2025 )
832
–
86.7
72.2/44.4
92.3
76.2
–
Qwen2.5-VL-7B ( Bai et al., 2025b )
864
56.3/57.2
87.3
73.9/42.5
95.7
82.6
0.308/0.398
Table 4: Comparison of OCR, chart, and document understanding performance. For a fair comparison, results marked with * are obtained by us under the same evaluation setting as our model.
Figure 4: Challenging cases for fine-grained visual understanding: (a) reasoning over an extremely high-resolution long document, (b) extracting irregular handwritten answers, and (c) long-form transcription of a dense full page.
Encoder
B
MathVista
RealWorldQA
MMStar
OCRBench
TextVQA
AI2D
ChartQA
DocVQA
Avg. ↑
MoonViT-v2
14
34.9
47.5
36.5
198
36.7
63.3
27.9
43.0
38.7
112
44.6
54.1
39.9
608
62.1
78.4
69.5
79.7
61.1
SigLIP2
14
38.6
52.7
43.5
287
47.4
68.3
44.6
33.7
44.7
112
40.0
56.3
43.9
499
64.7
75.6
57.1
43.3
53.9
Table 5: Comparison of VQ ( van den Oord et al., 2017 ) and FVQ across vision encoders. B denotes the per-patch bit budget; B=14 and B=112 correspond to single-codebook VQ and eight-codebook FVQ, respectively.
Method
MMMU
MathVista
RealWorldQA
MMStar
OCRBench
ChartQA
DocVQA
Continuous
LLaVA-OneVision ( Li et al., 2024 )
48.8
63.2
66.3
61.7
–
80.0
87.5
SigLIP2-g ( Tschannen et al., 2025 )
48.3
–
–
57.2
671
72.8
75.0
AIMv2-H ( Fini et al., 2025 )
48.2
–
–
55.2
687
72.5
80.1
InternViT-2.5-6B ( Chen et al., 2024b )
48.1
–
–
55.3
757
77.4
85.7
Discrete
Table 6: Comparison under LLaVA-OneVision SFT. LLaVA-OneVision uses Qwen2-7B ( Li et al., 2024 ) ; all other models use Qwen2.5-7B. LLaVA-OneVision results are from its paper, and the other baselines are reported by ViQ ( Yu et al., 2026b ) . Bold indicates the best discrete result.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Tokenizer
Alignment
PT
SFT
LLM LR
–
Frozen
1×10−5
1×10−5
Visual LR
1×10−4
2×10−4
1×10−5
2×10−6
LR schedule
Cosine
Cosine
Cosine
Constant
Weight decay
0
0.05
0.05
0.05
Gradient norm clip
1.0
1.0
1.0
1.0
Optimizer
AdamW ( β1=0.9 , β2=0.96 )
Appendix
Table 7: Training hyperparameter configurations for tokenizer training, alignment, pre-training (PT), and supervised fine-tuning (SFT).
Configuration
MathVista
RealWorldQA
MMStar
OCRBench
TextVQA
AI2D
ChartQA
DocVQA
Avg.
16× + Merger
44.5
57.6
44.2
658
69.7
79.3
68.2
81.7
63.9
32× + Pre-Buffer
46.0
59.2
43.3
634
71.5
75.9
66.8
81.9
63.5
Appendix
Table 8: Effect of quantization position and post-embedding adaptation. 16× and 32× denote spatial downsampling factors relative to the input image.
Figure 5: Dense Chinese full-page OCR. The source page and beginning of the transcription appear on the left, with the complete output continuing on the right. The model receives an OCRBenchV2 similarity score of 0.9690.
Figure 6: Dense English full-page OCR. The source page and beginning of the transcription appear on the left, with the complete output continuing on the right. The model receives an OCRBenchV2 similarity score of 0.9828.
Figure 7: Qualitative examples of extreme-resolution document retrieval and dense plot comparison. The complete model responses are shown with formatting adapted for typesetting.
Figure 8: Geometry reasoning with the complete model response, reformatted for typesetting.
Figure 9: Spatial reasoning with the complete model response, reformatted for typesetting.
Visual representation learning develops image and video features that capture useful structure for recognition, retrieval, spatial reasoning, and action.
A unified representation for text and vision is a natural pursuit, as it enables simpler multimodal modeling and more efficient training. However, representing images as discrete signals in the same way as text inevitably introduces severe information loss. Existing work struggles to balance low-level details and high-level semantics in discrete representations: reconstruction-oriented representations often lack semantic information, whereas semantically stronger features typically suffer from severe loss of detail. We present ViQ, a Visual Quantized Representations framework, which is designed to balance semantics and details in discrete representations while supporting inputs at native resolutions, thereby enabling it to serve as a unified and general discrete representation for arbitrary visual inputs. Our approach structures quantization learning into two stages: text-aligned pre-training and feature discretization. With text-aligned pre-training, we enhance the visual encoder semantic-rich supervision from the pretrained language model and enable it to process native-resolution visual inputs. During discretization, we propose a proximal representation learning strategy to progressively compact the feature space, along with a position-aware head-wise quantization mechanism that enables flexible processing of arbitrary resolutions. Extensive experiments on multimodal tasks demonstrate that ViQ achieves competitive performance compared to state-of-the-art multimodal vision encoders with continuous and high-dimensional visual features, while maintaining high precision in low-level reconstruction. We also show that multimodal training with visual quantized representations largely improves efficiency, yielding up to 20%-70% acceleration with different base LLMs and training recipes.
Xumin Yu, Zuyan Liu, Zhenyu Yang +5
Tencent HY Vision Team · Tsinghua University · Institute of Automation, CAS +1
Visual generation with discrete tokens has gained significant attention as it enables a unified token prediction paradigm shared with language models, promising seamless multimodal architectures. However, current discrete generation methods remain limited to low-dimensional latent tokens (typically 8-32 dims), sacrificing the semantic richness essential for understanding. While high-dimensional pretrained representations (768-1024 dims) could bridge this gap, their discrete generation poses fundamental challenges. In this paper, we present Cubic Discrete Diffusion (CubiD), the first discrete generation model for high-dimensional representations. CubiD performs fine-grained masking throughout the high-dimensional discrete representation -- any dimension at any position can be masked and predicted from partial observations. This enables the model to learn rich correlations both within and across spatial positions, with the number of generation steps fixed at T regardless of feature dimensionality, where T≪hwd. On ImageNet-256, CubiD achieves state-of-the-art discrete generation with strong scaling behavior from 900M to 3.7B parameters. Crucially, we validate that these discretized tokens preserve original representation capabilities, demonstrating that the same discrete tokens can effectively serve both understanding and generation tasks. We hope this work will inspire future research toward unified multimodal architectures. Code is available at: https://github.com/YuqingWang1029/CubiD.
Yuqing Wang, Chuofan Ma, Zhijie Lin +7
University of Hong Kong · ByteDance Seed · Carnegie Mellon University +2
Vision-language models improve perception by feeding increasingly long visual token sequences into language backbones, but the resulting inference cost raises a basic scaling question: as multimodal models grow, how many visual tokens are actually needed, and how should they be allocated under a fixed visual token budget? Existing training-free pruning methods typically answer this with one-shot proxies such as decoder attention, visual similarity, or conditional diversity. We argue that visual token pruning is better viewed as task-conditioned evidence search, especially under aggressive compression and across model scales. We propose F^3A, a training-free router for visual token pruning that operates before the language model consumes image tokens. F^3A builds lightweight question-conditioned cues, matches them to visual-grid tokens through frozen sparse sensing heads, and allocates a fixed vision token budget via coarse evidence localization, local refinement, coverage-preserving competition, and recovery of under-covered regions. It requires no model training, no extra LLM forward pass and preserves the original multimodal prompting and decoding pipeline.
YiJie Huang, Yiqun Zhang, Zhuoyue Jia +7
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · Shanghai Artificial Intelligence Laboratory · School of Computer and Communication Engineering, Northeastern University, Qinhuangdao 066004, China