Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?

Jialv Zou; Xiaotong Li; Zhengrong Yue; Xu Huang; Yaxin Luo; Wenhao Yan; Haodong Li; Wenyu Liu

cs.CVOct 27, 2026

Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?

Authors: Jialv Zou, Xiaotong Li, Zhengrong Yue, Xu Huang, Yaxin Luo, Wenhao Yan, Haodong Li, Wenyu Liu,

Organizations: Huazhong University of Science and Technology · Meituan LongCat · Shanghai Jiao Tong University · Peking University ·

Abstract

Discrete visual tokens represent vision in a discrete form analogous to language, offering a native interface for multimodal modeling under a shared next-token prediction objective. However, quantization error is widely assumed to cause unavoidable semantic loss, fundamentally limiting the semantic fidelity of discrete visual tokens and positioning continuous visual representations as an intrinsic upper bound. Using multimodal understanding as a proxy for semantic fidelity, we revisit this assumption and find quantization error unreliable, as discrete representations with substantially larger errors can achieve better understanding. Instead, the effective bit budget of discrete indices emerges as a key determinant of downstream performance. However, existing quantizers eventually saturate as additional codebooks contribute progressively less semantic information. As one approach to alleviating this saturation, we introduce Factorized Vector Quantization, which quantizes complementary, fixed-dimensional views of the visual feature with symmetric parallel codebooks. It continues to improve at larger bit budgets, ultimately matching continuous visual representations in multimodal understanding. Trained on industrial-scale multimodal data, our model achieves state-of-the-art performance among discrete-representation models, outperforming UniAR and Emu3-Chat by 22.8% and 59.2%, respectively, on a nine-metric average. To our knowledge, it is also the first discrete-representation model competitive with strong continuous-representation models such as Qwen3-VL, including on fine-grained OCR tasks. We further identify the key principles and a practical recipe enabling this competitiveness. These results and insights support the potential of discrete visual representations to underpin large-scale native multimodal foundation models. Code and models are available at https://github.com/hustvl/FVQ

Figures & tables

Figure 1: Scaling the bit budget of discrete visual representations. (a) Average multimodal understanding performance improves with the per-patch bit budget. FVQ continues to improve in the high-budget regime and reaches the continuous QwenViT baseline, whereas existing multi-codebook quantizers saturate. (b) Qualitative examples of text recognition at increasing bit budgets. Higher budgets progressively recover the fine-grained textual information required for correct predictions; green and red denote correct and incorrect answers, respectively.

Figure 2: Overview of the analyses in Section 2 : reconstruction error versus semantic fidelity (left) and bit-budget scaling across representative quantizers (right). $B_{\mathrm{raw}}$ , $B_{\mathrm{cont}}$ , and $B_{\mathrm{disc}}$ denote per-patch budgets for raw pixels, continuous features, and discrete indices. $f$ , $n$ , $V$ , $b$ , $D$ , $L$ , and $C$ denote patch side length, channels, values per channel, bits per feature dimension, feature dimension, number of codebooks, and codebook size, respectively.

Quantization			Multimodal Understanding
Quantizer	$C\times L$	MSE $\downarrow$	MathVista	RealWorldQA	MMStar	OCRBench	TextVQA	AI2D	ChartQA	DocVQA	Avg. $\uparrow$
VQ	$16{,}384\times 1$	0.027	39.4	52.4	40.5	367	51.3	67.1	54.2	68.2	51.2
RVQ (Odd)	$16{,}384\times 4$	0.246	41.4	56.2	41.7	615	66.9	75.1	63.5	79.5	60.7
RVQ (Even)	$16{,}384\times 4$	0.028	40.3	56.2	41.7	564	66.1	73.6	61.7	78.4	59.3
RVQ (All)	$16{,}384\times 8$	0.016	43.7	59.3	43.1	652	71.6	78.8	67.3	82.8	64.0

Table 1: Quantization error and multimodal understanding of discrete QwenViT representations.

C

and

L

denote the codebook size and number of selected codebooks, respectively. MSE measures feature-reconstruction error. Avg. is the unweighted mean over the eight benchmarks, with OCRBench divided by 10.

Figure 3: Overview of FVQ. During quantizer training, multiple dense projections produce complementary views of the same visual feature, which are quantized in parallel by independent codebooks. The selected codewords are combined through tight-frame synthesis for feature reconstruction. In the subsequent VLM training stage, the quantizer is frozen, and only the resulting discrete indices are mapped to newly initialized token embeddings for multimodal understanding. The nine-metric average is computed over MMMU, MathVista, MMStar, OCRBench/10, ChartQA, CharXiv-DQ, CharXiv-RQ, DocVQA, and InfoVQA.

Quantization			Multimodal Understanding
Quantizer	$L$	$B$	MathVista	RealWorldQA	MMStar	OCRBench	TextVQA	AI2D	ChartQA	DocVQA	Avg. $\uparrow$
VQ ( van den Oord et al., 2017 )	1	14	39.4	52.4	40.5	367	51.3	67.1	54.2	68.2	51.2
SimVQ ( Zhu et al., 2025b )	1	14	37.9	49.7	39.1	204	32.2	63.9	41.9	49.9	41.9
RVQ ( Lee et al., 2022 )	2	28	41.4	54.8	42.7	563	65.8	74.3	64.4	79.0	59.8
	4	56	43.2	58.7	43.1	625	69.7	78.2	66.8	81.7	63.0
	8	112	43.7	59.3	43.1	652	71.6	78.8	67.3	82.8	64.0

Table 2: Bit-budget scaling of discrete QwenViT representations under the shared experimental setting. All discrete variants quantize the same features with codebook size

C=16{,}384

L

is the number of codebooks and

B=L\log_{2}C

is the nominal per-patch bit budget. Bold indicates the best discrete result. Continuous denotes the original QwenViT representation.

Continuous
Method	MMMU	MMMU-Pro	MathVista	RealWorldQA	MMStar
LLaVA-OV-1.5-8B ( An et al., 2025 )	55.4	31.3	69.6	68.1	67.7
LLaVA-OV-2-8B ( An et al., 2026 )	–	–	–	69.7	64.8
InternVL3-8B ( Zhu et al., 2025a )	62.7	–	71.6	70.8	68.2
InternVL3.5-8B ( Wang et al., 2025 )	73.4	–	78.4	67.5	69.3
Qwen2.5-VL-7B ( Bai et al., 2025b )	58.6	38.3	68.2	–	63.9

Table 3: Comparison on STEM and general multimodal understanding benchmarks. For a fair comparison, results marked with * are obtained by us under the same evaluation setting as our model.

Continuous
Method	OCRBench	OCRBenchV2 (en/zh)	ChartQA	CharXiv (DQ/RQ)	DocVQA	InfoVQA	OmniDocBench (en/zh) $\downarrow$
LLaVA-OV-1.5-8B ( An et al., 2025 )	829	–	86.5	70.9/–	95.0	78.4	–
LLaVA-OV-2-8B ( An et al., 2026 )	782	–	85.9	–	95.2	74.4	–
InternVL3-8B ( Zhu et al., 2025a )	880	–	86.6	73.6/37.6	92.7	76.8	–
InternVL3.5-8B ( Wang et al., 2025 )	832	–	86.7	72.2/44.4	92.3	76.2	–
Qwen2.5-VL-7B ( Bai et al., 2025b )	864	56.3/57.2	87.3	73.9/42.5	95.7	82.6	0.308/0.398

Table 4: Comparison of OCR, chart, and document understanding performance. For a fair comparison, results marked with * are obtained by us under the same evaluation setting as our model.

Figure 4: Challenging cases for fine-grained visual understanding: (a) reasoning over an extremely high-resolution long document, (b) extracting irregular handwritten answers, and (c) long-form transcription of a dense full page.

Encoder	$B$	MathVista	RealWorldQA	MMStar	OCRBench	TextVQA	AI2D	ChartQA	DocVQA	Avg. $\uparrow$
MoonViT-v2	14	34.9	47.5	36.5	198	36.7	63.3	27.9	43.0	38.7
MoonViT-v2	112	44.6	54.1	39.9	608	62.1	78.4	69.5	79.7	61.1
SigLIP2	14	38.6	52.7	43.5	287	47.4	68.3	44.6	33.7	44.7
SigLIP2	112	40.0	56.3	43.9	499	64.7	75.6	57.1	43.3	53.9

Table 5: Comparison of VQ ( van den Oord et al., 2017 ) and FVQ across vision encoders.

B

denotes the per-patch bit budget;

B=14

and

B=112

correspond to single-codebook VQ and eight-codebook FVQ, respectively.

Method	MMMU	MathVista	RealWorldQA	MMStar	OCRBench	ChartQA	DocVQA
Continuous
LLaVA-OneVision ( Li et al., 2024 )	48.8	63.2	66.3	61.7	–	80.0	87.5
SigLIP2-g ( Tschannen et al., 2025 )	48.3	–	–	57.2	671	72.8	75.0
AIMv2-H ( Fini et al., 2025 )	48.2	–	–	55.2	687	72.5	80.1
InternViT-2.5-6B ( Chen et al., 2024b )	48.1	–	–	55.3	757	77.4	85.7
Discrete

Table 6: Comparison under LLaVA-OneVision SFT. LLaVA-OneVision uses Qwen2-7B ( Li et al., 2024 ) ; all other models use Qwen2.5-7B. LLaVA-OneVision results are from its paper, and the other baselines are reported by ViQ ( Yu et al., 2026b ) . Bold indicates the best discrete result.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Tokenizer	Alignment	PT	SFT
LLM LR	–	Frozen	$1\times 10^{-5}$	$1\times 10^{-5}$
Visual LR	$1\times 10^{-4}$	$2\times 10^{-4}$	$1\times 10^{-5}$	$2\times 10^{-6}$
LR schedule	Cosine	Cosine	Cosine	Constant
Weight decay	0	0.05	0.05	0.05
Gradient norm clip	1.0	1.0	1.0	1.0
Optimizer	AdamW ( $\beta_{1}=0.9$ , $\beta_{2}=0.96$ )

Appendix

Table 7: Training hyperparameter configurations for tokenizer training, alignment, pre-training (PT), and supervised fine-tuning (SFT).

Configuration	MathVista	RealWorldQA	MMStar	OCRBench	TextVQA	AI2D	ChartQA	DocVQA	Avg.
$16\times$ + Merger	44.5	57.6	44.2	658	69.7	79.3	68.2	81.7	63.9
$32\times$ + Pre-Buffer	46.0	59.2	43.3	634	71.5	75.9	66.8	81.9	63.5

Appendix

Table 8: Effect of quantization position and post-embedding adaptation.

16\times

and

32\times

denote spatial downsampling factors relative to the input image.

Panel of Figure 5 — Figure 5: Dense Chinese full-page OCR. The source page and beginning of the transcription appear on the left, with the complete output continuing on the right. The model receives an OCRBenchV2 similarity score of 0.9690.

Panel of Figure 6 — Figure 6: Dense English full-page OCR. The source page and beginning of the transcription appear on the left, with the complete output continuing on the right. The model receives an OCRBenchV2 similarity score of 0.9828.

Panel of Figure 7 — Figure 7: Qualitative examples of extreme-resolution document retrieval and dense plot comparison. The complete model responses are shown with formatting adapted for typesetting.

Figure 8: Geometry reasoning with the complete model response, reformatted for typesetting.

Figure 9: Spatial reasoning with the complete model response, reformatted for typesetting.

Visual Representation Learning

Visual representation learning develops image and video features that capture useful structure for recognition, retrieval, spatial reasoning, and action.

See all 224 papers

ViQ: Text-Aligned Visual Quantized Representations at Any Resolution
Jun 25, 2026Xumin Yu, Zuyan Liu, Zhenyu Yang +5Visual Representation Learning Multimodal Pretraining
Tencent HY Vision Team · Tsinghua University · Institute of Automation, CAS +1
Cubic Discrete Diffusion: Discrete Visual Generation on High-Dimensional Representation Tokens
Mar 19, 2026Yuqing Wang, Chuofan Ma, Zhijie Lin +7Image Generation Diffusion-Based Image Generation
University of Hong Kong · ByteDance Seed · Carnegie Mellon University +2
How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A
May 9, 2026YiJie Huang, Yiqun Zhang, Zhuoyue Jia +7Efficient VLM Inference Large Vision-Language Models
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · Shanghai Artificial Intelligence Laboratory · School of Computer and Communication Engineering, Northeastern University, Qinhuangdao 066004, China

Jun 25, 2026J/K move · Enter open · S save

ViQ: Text-Aligned Visual Quantized Representations at Any Resolution

Xumin Yu, Zuyan Liu, Zhenyu Yang, Yuhao Dong+4

Tencent HY Vision Team · Tsinghua University · Institute of Automation, CAS+1

Visual Representation Learning Multimodal Pretraining Vision-Language Alignment Visual Token Compression Multimodal Understanding

Open paper AskPDF arXiv

A unified representation for text and vision is a natural pursuit, as it enables simpler multimodal modeling and more efficient training. However, representing images as discrete signals in the same way as text inevitably introduces severe information loss. Existing work struggles to balance low-level details and high-level semantics in discrete representations: reconstruction-oriented representations often lack semantic information, whereas semantically stronger features typically suffer from severe loss of detail. We present ViQ, a Visual Quantized Representations framework, which is designed to balance semantics and details in discrete representations while supporting inputs at native resolutions, thereby enabling it to serve as a unified and general discrete representation for arbitrary visual inputs. Our approach structures quantization learning into two stages: text-aligned pre-training and feature discretization. With text-aligned pre-training, we enhance the visual encoder semantic-rich supervision from the pretrained language model and enable it to process native-resolution visual inputs. During discretization, we propose a proximal representation learning strategy to progressively compact the feature space, along with a position-aware head-wise quantization mechanism that enables flexible processing of arbitrary resolutions. Extensive experiments on multimodal tasks demonstrate that ViQ achieves competitive performance compared to state-of-the-art multimodal vision encoders with continuous and high-dimensional visual features, while maintaining high precision in low-level reconstruction. We also show that multimodal training with visual quantized representations largely improves efficiency, yielding up to 20%-70% acceleration with different base LLMs and training recipes.

Abstract

Visual Representation Learning

Explore similar work

ViQ: Text-Aligned Visual Quantized Representations at Any Resolution

Cubic Discrete Diffusion: Discrete Visual Generation on High-Dimensional Representation Tokens

How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A