Autoregressive large language models (LLMs) scale well by expressing diverse tasks as sequences of discrete natural-language tokens and training with next-token prediction, which unifies comprehension and generation under self-supervision. Extending this paradigm to multimodal data requires a shared, discrete representation across modalities. However, most vision-language models (VLMs) still rely on a hybrid interface: discrete text tokens paired with continuous Vision Transformer (ViT) features. Because supervision is largely text-driven, these models are often biased toward understanding and cannot fully leverage large-scale self-supervised learning on non-text data. Recent work has explored discrete visual tokenization to enable fully autoregressive multimodal modeling, showing promising progress toward unified understanding and generation. Yet existing discrete vision tokens frequently lose information due to limited code capacity, resulting in noticeably weaker understanding than continuous-feature VLMs. We present Kelix, a fully discrete autoregressive unified model that closes the understanding gap between discrete and continuous visual representations.
Figures & tables
Figure 1: Benchmark performance of Kelix. Compared with prior discrete unified models, Kelix significantly improves understanding, even matching continuous understanding-only VLMs, while achieving competitive image generation results with current SOTA.
Figure 2: The auto-regressive inference workflow of our method, including the Kelix-Tok, and unified semantic understanding and image generation.
Figure 3: The auto-regressive training workflow of our method, including the Kelix Tokenizer, and Unified LLM model and Image DiT.
Notation
Description
Basic concept
I
An arbitrary image I .
P
The patch number of a Image P .
Z/z
The embedding matrix-of-all-patchs/vector-of-specific-patch.
W
The learnable parameter matrix.
VQ codebook concept
Table 1: A list of commonly used notation.
Model
Date
General Benchmarks
Knowledge Benchmarks
Text-rich Benchmarks
SEED-Bench
RealWorldQA
MMBench EN
AI2D test
MMMU
MathVista
ChartQA
TextVQA
OCRBench
Understanding-only Models
LLaVA-OV (7B)
2024.08
76.7
69.9
–
81.4
48.8
–
80.9
–
62.2
BLIP-3 (4B)
2024.08
72.2
60.5
–
–
41.1
39.6
–
71.0
–
InternVL3.5 (4B)
2025.08
–
66.3
–
82.6
66.6
77.1
86.0
77.9
82.2
Qwen2.5VL (3B)
2025.02
–
65.4
76.4
81.6
53.1
62.3
84.0
79.7
79.7
Table 2: Comprehensive comparison on a curated set of multimodal benchmarks. The release dates denote the time of arXiv submission or official blog release.
Model
Date
GenEval Benchmark
WISE Benchmark
Single
Two
Count
Color
Pos.
C-Attr
Overall
Cult.
Time
Space
Bio.
Phys.
Chem.
Overall
Dedicated T2I Models
SDXL (3.5B)
2023.07
98.0
74.0
39.0
85.0
15.0
23.0
55.0
43.0
48.0
47.0
44.0
45.0
27.0
43.0
DALL-E 3
2023.10
96.0
87.0
47.0
83.0
43.0
45.0
67.0
–
–
–
–
–
–
–
SD3 Medium (2B)
2024.06
99.0
94.0
72.0
89.0
33.0
60.0
74.0
43.0
50.0
52.0
41.0
53.0
33.0
45.0
FLUX.1-dev (12B)
2024.08
98.0
93.0
75.0
93.0
68.0
65.0
82.0
48.0
58.0
62.0
42.0
51.0
35.0
50.0
Table 3: Comprehensive comparison of image generation performance on GenEval and WISE benchmarks. The release dates denote the time of arXiv submission or official release. For Kelix, we report performance using the default resolution of 1024×1024 .
Method
Date
Global
Entity
Attribute
Relation
Other
Overall ↑
Dedicated T2I Models
SDXL (2.6B)
2023.07
83.3
82.4
80.9
86.8
80.4
74.7
DALL-E 3
2023.10
91.0
89.6
88.4
90.6
89.8
83.5
SD3 Medium (2B)
2024.06
87.9
91.0
88.8
80.7
88.7
84.1
FLUX.1-dev (12B)
2024.08
–
–
–
–
–
–
LLM & Diffusion Conjunction
Table 4: Comparison of image generation performance on the DPG-Bench benchmark. All scores are reported on a 100-point scale. The release dates denote the time of arXiv submission or official release.
Figure 4: Overview of the high-quality text-to-image training data synthesis pipeline, comprising prompt construction, frontier model distillation, VLM-based multi-dimensional quality verification, and final selection.
Figure 5: Semantic tokenizer training recipe. In Stage 1, we freeze the ViT and LLM backbones. To ensure that the commitment loss can be optimized, we insert the MLP and SimVQ up-projector before VQ codebook. In Stage2 and Stage3, we enable the gradients of ViT and Qwen3.
Figure 6: The LLM training recipe of Kelix, we first pre-train the Block Decoder and the image token embedding for a better warm start, and then unifying the image token and text token to a same embedding table for full-parameter tuning.
Stage
Samples
Tokens
Data Type
Stage 1: Cross-Modal Alignment
191M
108B
Pure text corpora; image-caption pairs
Stage 2.1: Interleaved Scaling
173M
191B
Large-scale interleaved image-text data
Stage 2.2: Multi-Task Pre-Training
593M
679B
Large-scale pure text corpora; descriptive captions; simple visual-language QA; text-to-image pairs (reversed from captions)
Stage 3: Annealing Training
40M
70B
Pure text Chain-of-Thought (CoT) data; multimodal CoT data; high-quality filtered multimodal QA; filtered pure text corpora; general image generation data (filtered via rejection sampling)
Stage 4: SFT
4M
8B
Replayed Stage 3 high-quality data; task-specific CoT data; task-specific image generation data (e.g., attribute/subject/scene/style/perspective control); task-specific enhancement data ( e.g. , complex OCR, math reasoning, STEM data)
Table 5: Data statistics of each LLM training stage, including the number of samples, total token volume, and data type.
Configuration
General Benchmarks
Knowledge Benchmarks
Text-rich Benchmarks
RWQA
MMB CN
MMB EN
Avg.
AI2D
MMMU
MathV
Avg.
Chart
Text
OCRB
Avg.
(a) Baseline Comparison
Discrete Baseline
55.3
53.8
57.8
55.6
61.6
31.3
40.3
44.4
68.7
64.8
66.0
66.5
Continuous Baseline
59.7
56.6
63.1
59.8
65.4
34.2
50.3
50.0
78.9
75.8
79.3
78.0
Kelix-Tok (Default)
58.8
55.9
59.3
58.0
62.9
33.9
50.0
48.9
76.5
74.7
79.2
76.8
(b) Independent vs. Shared Codebooks (fixed N=8 , Mean Pool)
Table 6: Ablation studies on Kelix-Tok .
Figure 7: Ablation study on parameter N : LM Loss and OCRBench curves.
Configuration
General Benchmarks
Knowledge Benchmarks
Text-rich Benchmarks
RWQA
MMB CN
MMB EN
Avg.
AI2D
MMMU
MathV
Avg.
Chart
Text
OCRB
Avg.
exp1: Unified Baseline
61.3
67.7
66.9
65.3
70.8
42.0
55.8
56.2
76.9
72.3
79.9
76.4
exp2: Random Init Input Emb.
56.7
60.7
59.3
58.9
67.5
34.7
53.8
52.0
77.8
70.4
77.0
75.1
exp3: 9-Head Predictor (Output)
64.8
60.4
61.6
62.3
69.3
41.7
52.9
54.6
75.2
71.4
76.5
74.4
Table 7: Ablation studies on our Unified Multimodal LLM Architecture . We evaluate the design of multimodal interfaces, representation paradigms, and training synergies across Stage 1 and Stage 3.
Figure 8: The visualization comparison of different unified models in image generation.
Figure 9: The visualization of Kelix in image generation.
Figure 10: The visualization of Kelix in image generation.
Figure 11: The visualization of Kelix in image generation.
The prevailing Next-Token Prediction (NTP) paradigm has driven the success of large language models through discrete autoregressive modeling. However, contemporary multimodal systems remain language-centric, often treating non-linguistic modalities as external attachments, leading to fragmented architectures and suboptimal integration. To transcend this limitation, we introduce Discrete Native Autoregressive (DiNA), a unified framework that represents multimodal information within a shared discrete space, enabling a consistent and principled autoregressive modeling across modalities. A key innovation is the Discrete Native Any-resolution Visual Transformer (dNaViT), which performs tokenization and de-tokenization at arbitrary resolutions, transforming continuous visual signals into hierarchical discrete tokens. Building on this foundation, we develop LongCat-Next, a native multimodal model that processes text, vision, and audio under a single autoregressive objective with minimal modality-specific design. As an industrial-strength foundation model, it excels at seeing, painting, and talking within a single framework, achieving strong performance across a wide range of multimodal benchmarks. In particular, LongCat-Next addresses the long-standing performance ceiling of discrete vision modeling on understanding tasks and provides a unified approach to effectively reconcile the conflict between understanding and generation. As an attempt toward native multimodality, we open-source the LongCat-Next and its tokenizers, hoping to foster further research and development in the community. GitHub: https://github.com/meituan-longcat/LongCat-Next
While Vision-Language Models (VLMs) demonstrate strong capabilities, they continue to suffer from a critical limitation: insufficient fine-grained visual perception, which fundamentally limits their multimodal understanding. We attribute this bottleneck to text-dominant optimization biases during pre-training, which encourage the model to overlook fine-grained visual details, thereby limiting the capability of multimodal understanding. We investigate that overcoming this bottleneck requires two key elements: (1) a unified token space paradigm that ensures stable training dynamics, and (2) a modality-aligned dense visual supervision signal enriched with both structural granularity and semantic information to capture critical visual representations. Based on these insights, we propose VIVAS, a framework built upon the unified token space paradigm, which introduces a dense-structural-semantic vision tokenizer, which expands the textual vocabulary into a unified vision-language vocabulary by incorporating a visual vocabulary. During pretraining, VIVAS performs vision-language unified autoregressive supervision over both visual details and linguistic content, thereby enhancing visual perception to improve multimodal understanding. Trained end-to-end on 12.4T tokens, VIVAS achieves state-of-the-art performance across 7 tasks and 39 multimodal benchmarks.
Zhehan Kan, Yubo Zhu, Xinghua Jiang +10
Tsinghua University · Tencent Youtu Lab · Nanjing University
Discrete autoregressive modeling, notably through next-token prediction, has become a cornerstone of modern language models. Extending this paradigm to vision allows visual and language models to converge on a unified framework, thereby fostering shared methodology and infrastructure. This work investigates a central question: how far can a finite discrete vocabulary go in representing the richness of the visual world? We present dNaViT, a Discrete Native-resolution Vision Transformer that produces discrete visual primitives analogous to language tokens. Built upon a frozen pretrained encoder, dNaViT applies lightweight residual modulation with a dual-branch objective to inject reconstruction-aware cues into semantic features. By leveraging grouped product quantization, it scales discrete capacity without inflating spatial sequence lengths, while a balanced training recipe maintains both semantic and pixel-level information. Consequently, dNaViT functions as a versatile discrete visual tokenizer, enabling unified encoding across downstream tasks alongside high-fidelity image decoding. Extensive evaluations show that dNaViT matches or exceeds task-specific baselines across 12 visual understanding benchmarks, 4 generation benchmarks, and an image editing benchmark. These findings demonstrate that a compact discrete vocabulary can serve as a universal visual currency and unlock the full potential of fully unified, native multimodal methodology.