LongCat-Next: Lexicalizing Modalities as Discrete Tokens

Meituan LongCat Team; Bin Xiao; Chao Wang; Chengjiang Li; Chi Zhang; Chong Peng; Hang Yu; Hao Yang

cs.CVMar 29, 2026

LongCat-Next: Lexicalizing Modalities as Discrete Tokens

Authors: Meituan LongCat Team, Bin Xiao, Chao Wang, Chengjiang Li, Chi Zhang, Chong Peng, Hang Yu, Hao Yang, Haonan Yan, Haoze Sun, Haozhe Zhao, Hong Liu, Hui Su, Jiaqi Zhang, Jiawei Wang, Jing Li, Kefeng Zhang, Manyuan Zhang, Minhao Jing, Peng Pei, Quan Chen, Taofeng Xue, Tongxin Pan, Xiaotong Li, Xiaoyang Li, Xiaoyu Zhao, Xing Hu, Xinyang Lin, Xunliang Cai, Yan Bai, Yan Feng, Yanjie Li, Yao Qiu, Yerui Sun, Yifan Lu, Ying Luo, Yipeng Mei, Yitian Chen, Yuchen Xie, Yufang Liu, Yufei Chen, Yulei Qian, Yuqi Peng, Zhihang Yu, Zhixiong Han, Changran Wang, Chen Chen, Dian Zheng, Fengjiao Chen, Ge Yang, Haowei Guo, Haozhe Wang, Hongyu Li, Huicheng Jiang, Jiale Hong, Jialv Zou, Jiamu Li, Jianping Lin, Jiaxing Liu, Jie Yang, Jing Jin, Jun Kuang, Juncheng She, Kunming Luo, Kuofeng Gao, Lin Qiu, Linsen Guo, Mianqiu Huang, Qi Li, Qian Wang, Rumei Li, Siyu Ren, Wei Wang, Wenlong He, Xi Chen, Xiao Liu, Xiaoyu Li, Xu Huang, Xuanyu Zhu, Xuezhi Cao, Yaoming Zhu, Yifei Cao, Yimeng Jia, Yizhen Jiang, Yufei Gao, Zeyang Hu, Zhenlong Yuan, Zijian Zhang, Ziwen Wang

Abstract

The prevailing Next-Token Prediction (NTP) paradigm has driven the success of large language models through discrete autoregressive modeling. However, contemporary multimodal systems remain language-centric, often treating non-linguistic modalities as external attachments, leading to fragmented architectures and suboptimal integration. To transcend this limitation, we introduce Discrete Native Autoregressive (DiNA), a unified framework that represents multimodal information within a shared discrete space, enabling a consistent and principled autoregressive modeling across modalities. A key innovation is the Discrete Native Any-resolution Visual Transformer (dNaViT), which performs tokenization and de-tokenization at arbitrary resolutions, transforming continuous visual signals into hierarchical discrete tokens. Building on this foundation, we develop LongCat-Next, a native multimodal model that processes text, vision, and audio under a single autoregressive objective with minimal modality-specific design. As an industrial-strength foundation model, it excels at seeing, painting, and talking within a single framework, achieving strong performance across a wide range of multimodal benchmarks. In particular, LongCat-Next addresses the long-standing performance ceiling of discrete vision modeling on understanding tasks and provides a unified approach to effectively reconcile the conflict between understanding and generation. As an attempt toward native multimodality, we open-source the LongCat-Next and its tokenizers, hoping to foster further research and development in the community. GitHub: https://github.com/meituan-longcat/LongCat-Next

Figures & tables

Figure 1: Benchmark performance of LongCat-Next.

Figure 2: Overview of the LongCat-Next architecture, designed under a Discrete Native Autoregression (DiNA) paradigm that extends multimodality into a native language-style modeling framework via paired tokenizers.

Figure 3: Overview of the discrete Native-Resolution Vision Transformer (dNaViT) design.

Figure 4: The tokenizer and de-tokenizer training pipeline of the proposed dNaViT, which encodes images into discrete tokens via RVQ and decodes them back into the image space through a pixel decoder at arbitrary resolutions.

Figure 5: Visual reconstruction from different frozen vision encoders, trained with a lightweight pixel decoder. The results suggest that residual connections inherently preserve a latent pathway for low-level signal propagation.

Figure 6: Illustration of the audio tokenizer framework.

Figure 7: Two speech generation strategies with internal linguistic guidance. The user’s input audio is treated as a pure audio modality, whereas the output generated by Longcat-Next (or the assistant) is regarded as a text-guided audio modality. To indicate modality conversion and facilitate segment alignment, we introduce three special tokens: AS (Audio Start), AE (Audio End), and TE (Text End), which respectively denote the beginning and end of an audio segment, and the end of a text segment.

Figure 8: The understanding and generation cases of LongCat-Next at arbitrary resolution.

Benchmark	LongCat-Next	Qwen3-Omni-A3B-Instruct	GPT5-minimal	Gemini2.5-Flash-Lite	Specialist MLLMs
Benchmark					InternVL3.5-A3B-Flash	Qwen3-VL-A3B-Instruct
STEM & Reasoning
MMMU-Pro	60.3	57.0 ∗	62.7 ∗	64.1	–	60.4 ∗
MMMU ${}_{\text{val}}$	70.6	69.1 ∗	74.4 ∗	74.9	75.6 ∗	74.2 ∗
MathVista ${}_{\text{mini}}$	83.1	75.9 ∗	50.9 ∗	78.2	80.9 ∗	80.1 ∗
MathVision	64.7	56.3 ∗	45.8 ∗	61.9	55.7 ∗	60.2 ∗

Table 1: Comparison of LongCat-Next and top-tier models on vision benchmarks. Values marked with * are sourced from public reports.

Model	GenEval	DPG	LongText-EN	LongText-ZH	WISE	TIFF	CVTG
Emu-3.5 [ 15 ]	72.67	${89.42}^{*}$	${97.60}^{*}$	${92.80}^{*}$	57.64	${89.48/88.18}^{*}$	${91.23}^{*}$
Qwen-Image 2507 [ 123 ]	${87.00}^{*}$	${88.32}^{*}$	${94.30}^{*}$	${94.60}^{*}$	${63.00}^{*}$	${86.10}^{}$ / ${86.80}^{}$	${82.88}^{*}$
Gemini 2.5 Flash Image [ 13 ]	79.67	85.82	86.04	-	76.27	90.53 / 90.80	${73.64}^{*}$
FLUX.1-dev [ 55 ]	${66.00}^{*}$	${84.00}^{*}$	${60.70}^{*}$	${0.50}^{*}$	${50.00}^{*}$	${71.10/71.80}^{*}$	${49.65}^{*}$
Seeddream 3.0 [ 35 ]	-	${94.31}^{*}$	${89.60}^{*}$	${87.80}^{*}$	-	${86.02/84.31}^{*}$	${59.24}^{*}$
LongCat-Next	84.44	84.66	93.15	89.08	57.00	82.85 / 84.38	76.36

Table 2: Comparison with specialized T2I models. Despite significantly smaller model, our model achieves competitive performance across multiple benchmarks, demonstrating strong industrial-level generation capability. Values marked with * are sourced from public reports.

Model	Understanding					Generation
Model	MMMU	MathVista	OCRBench	DocVQA	MMStar	GenEval	DPG	LT-EN/ZH	WISE	TIFF	CVTG
Janus-Pro [ 12 ]	41.0	–	–	–	–	80	84.19	1.90 / 0.60	35	65.50 / 68.80	–
Show-o2 [ 131 ]	48.9	–	–	–	56.6	76	86.14	– / –	–	65.80 / 65.00	–
OneCAT [ 57 ]	41.9	61.7	–	91.2	–	90	84.53	– / –	–	–	–
Mogao [ 61 ]	44.2	–	–	–	–	89	84.33	– / –	–	–	–
BAGEL [ 21 ]	55.3	73.1	80.9	92.2	69.3	82	85.07	43.70	4.70 / 49	71.50 / 71.70	35.60

Table 3: Comparison with unified multimodal models. We evaluate across both understanding and generation benchmarks. Our model (LongCat-Next) consistently outperforms prior unified approaches across most generation benchmarks while maintaining highly competitive understanding capabilities, establishing a new state-of-the-art for unified generation-understanding models. All results are from public reports.

Benchmark	LongCat-Next	Gemini-3.1-Flash-Lite-preview	Gemini-2.5-Flash-Lite-preview	Qwen3-Omni-A3B-Instruct	MiMo-Audio	Kimi-Audio	Step-Audio-2-mini
ASR
LibriSpeech ${}_{\text{test-clean}}$ $\downarrow$	1.63	2.38	3.14	1.22 ∗	2.47	1.28 ∗	1.33 ∗
LibriSpeech ${}_{\text{test-other}}$ $\downarrow$	3.42	5.68	7.19	2.48 ∗	6.13	2.42 ∗	2.86 ∗
AISHELL-1 $\downarrow$	1.47	6.00	11.64	0.84 ∗	1.69	0.60 ∗	0.78 ∗
AISHELL-2 $\downarrow$	2.82	8.00	15.33	2.34 ∗	3.02	2.56 ∗	2.16 ∗
Fleurs ${}_{\text{zh}}$ $\downarrow$	3.24	3.88	7.77	2.20 ∗	9.29	2.69 ∗	2.53 ∗

Table 4: Comparison of LongCat-Next and top-tier models on audio benchmarks.Values marked with * are sourced from public reports.

Benchmark	LongCat-Next	Kimi-Linear-48B-A3B	Qwen3-Next-80B-A3B-Instruct	Qwen3-Omni-A3B-Instruct
Agentic Tool Use
Tau2-Airline(avg@8)	56.50	44.00	45.5*	27.00
Tau2-Retail(avg@8)	73.68	18.86	57.3*	40.80
Tau2-Telecom(avg@8)	62.06	15.68	13.2*	4.39
VitaBench(avg@4)	5.80	-	5.80	-
Agentic Coding

Table 5: Comparison between LongCat-Next and other models. Values marked with * are sourced from public reports.

Figure 9: L: Loss comparison on pre-align stage. R: Performance comparison of discrete and continuous versions.

Metric	ResNet50	ViT-B/16 (Pretrained)	ViT-B/16 (Random)	QwenViT (w/o merger)
PSNR ( $\uparrow$ )	$20.88\pm 3.44$	$21.86\pm 3.14$	$30.52\pm 3.42$	$18.16\pm 2.61$
SSIM ( $\uparrow$ )	$0.509\pm 0.174$	$0.581\pm 0.139$	$0.887\pm 0.051$	$0.46\pm 0.14$
rFID ( $\downarrow$ )	0.4619	0.8850	0.5847	0.987

Table 6: Quantitative reconstruction performance across various visual encoder architectures. PSNR and SSIM represent reconstruction fidelity (

\uparrow

), while rFID measures perceptual discrepancy (

\downarrow

Figure 10: Visual understanding and generation interaction under DiNA framework.

Figure 11: Training dynamics of modality-agnostic MoE before and after multimodal training. Left: token distribution across experts. Right: the emergence of modality-specialized roles within an initially modality-agnostic MoE.

Figure 12: Modality-aware feature distribution divergence: a comparison of traditional Qwen2.5-VL, data-native-training Qwen3.5, and our architectural native modality with LongCat-Next.

Figure 14: Left: The data curation and training process consists of three stages: (I) Pre-training with filtering and recaptioning for basic alignment; (II) Mid-training with semantic clustering and rebalancing to address data imbalance; (III) Supervised Fine-Tuning on high-quality, instruction-following data for improved aesthetics and text rendering. Right: The distribution of image sources.

Task	Data Format	Prop. (%)
Automatic Speech Recognition (ASR)	<prompt, audio, transcript>	11
Contiguous TextAudio (TTS)	<textaudio_1, textaudio_2, textaudio_3, …>	40
Interleaved Audio and Text (INTLV)	<audio_1, text_2, audio_3, text_4, …>	22
Interleaved Audio and TextAudio (INTLV-TA)	<audio_1, textaudio_2, audio_3, textaudio_4, …>	22
Pure Audio	<audio>	2
others (AQA, S2TT, etc.)	<prompt, audio, response>	3

Table 8: Detailed Statistics of Audio Pre-training Data. Text-Guided Audio is abbreviated as TextAudio here.

Figure 16: Infra training pipeline designed for LongCat-Next.

Panel of Figure 17 — Figure 17: Reward Scores in RL.

Panel of Figure 18 — Figure 18: Training metrics comparison in RL.

	STEM				General
Dataset	MMMU	MMMU-Pro	MathVista	MathVision	MMStar	RealWorldQA	MMVP
baseline	64.22	51.27	80.30	49.28	66.33	66.01	73.33
RL	66.45	53.58	81.90	53.52	71.13	72.54	74.66
OCR & Doc
Dataset	CharXiv ${}_{\text{DQ}}$	CharXiv ${}_{\text{RQ}}$	InfoVQA	ChartQA	AI2D	OmniDocBench(zh / en) $\downarrow$
baseline	85.45	51.8	81.69	88.56	81.57	0.187	0.256

Table 9: Image Understanding and image generation performance based on Qwen-7B.

Figure 20: A unified tokenizer and detokenizer for both understanding and generation within the DiNA paradigm.

Figure 22: Our model is capable of producing formatted outputs and performing numerical comparisons in the OCR recognition phase, thereby enhancing the accuracy and interpretability of the recognition results.

Figure 23: Two examples of LongCat-Next solving complex mathematical logic puzzles through multi-step reasoning. The top panel showcases the model’s ability to identify arithmetic patterns within a cross-shaped visual structure to deduce a missing number. The bottom panel illustrates advanced spatial reasoning and constraint satisfaction, where the model systematically tests different hypotheses to deduce the correct placement of digits in a 3x3 grid and calculate the required sum.

Figure 24: Two examples of LongCat-Next solving geometric problems. The top panel demonstrates the model’s ability to deduce square areas through nested geometric relationships, while the bottom panel highlights its capacity to decompose a complex quadrilateral into triangles and verify right-angle properties to calculate the total area.

Figure 25: The effect of visual de-tokenizer about the pixel decoder and refiner module.

Unified Multimodal Models

Unified multimodal models combine multiple data types in shared systems for cross-modal understanding, generation, reasoning, and interaction.

See all 283 papers

Mar 6, 2026

Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion

While recent multimodal large language models (MLLMs) have made impressive strides, they predominantly employ a conventional autoregressive architecture as their backbone, leaving significant room to explore effective and efficient alternatives in architectural design. Concurrently, recent studies have successfully applied discrete diffusion models to various domains, such as visual understanding and image generation, revealing their considerable potential as a promising backbone for multimodal systems. Drawing inspiration from these pioneering studies, we introduce Omni-Diffusion, the first any-to-any multimodal language model built entirely on mask-based discrete diffusion models, which unifies understanding and generation across text, speech, and images. Omni-Diffusion employs a unified mask-based discrete diffusion model to directly capture the joint distribution over discrete multimodal tokens. This approach supports not only bimodal tasks but also more complex scenarios involving multiple modalities. On a diverse set of benchmarks, our method outperforms or performs on par with existing multimodal systems that process two or more modalities, highlighting the significant promise of diffusion models in powering the next generation of multimodal foundation models. Project webpage: https://omni-diffusion.github.io.

Lijiang Li, Zuwei Long, Yunhang Shen +6

Nanjing University · Tencent Youtu Lab · CASIA

Masked Diffusion Models Unified Multimodal Models Multimodal Generation Multimodal Foundation Models Multimodal Learning Diffusion Models+2

Ask PDF

May 18, 2026

Lance: Unified Multimodal Modeling by Multi-Task Synergy

We present Lance, a lightweight native unified model supporting multimodal understanding, generation, and editing for both images and videos. Rather than relying on model capacity scaling or text-image-dominant designs, Lance explores a practical paradigm for unified multimodal modeling via collaborative multi-task training. It is grounded in two core principles: unified context modeling and decoupled capability pathways. Specifically, Lance is trained from scratch and employs a dual-stream mixture-of-experts architecture on shared interleaved multimodal sequences, enabling joint context learning while decoupling the pathways for understanding and generation. We further introduce modality-aware rotary positional encoding to mitigate interference among heterogeneous visual tokens and boost cross-task alignment. During training, Lance adopts a staged multi-task training paradigm with capability-oriented objectives and adaptive data scheduling to strengthen both semantic comprehension and visual generation performance. Experimental results demonstrate that Lance substantially outperforms existing open-source unified models in image and video generation, while retaining strong multimodal understanding capabilities. The homepage is available at https://lance-project.github.io.

Fengyi Fu, Mengqi Huang, Shaojin Wu +10

Intelligent Creation Lab, ByteDance

Unified Multimodal Models Multi-Task Learning Multimodal Generation Multimodal Foundation Models Multimodal Learning Multimodal Understanding

Ask PDF

May 25, 2026

Toward Native Multimodal Modeling: A Roadmap

Multimodal modeling represents a vital step from modality-agnostic reasoning toward world modeling. While early approaches predominantly rely on late-fusion that assembles encoders and frozen language backbones with output heads, recent efforts have shifted the paradigm toward native multimodal modeling (NMM) with the intrinsic integration of modalities for superior multimodal performance. Despite its potential, the design space of native architectures remains insufficiently defined. In this paper, we present the community with a formalized roadmap for this transition. Specifically, we formally define the architectural nativity, distinguishing mid-fusion and early-fusion from non-native paradigms. We further organize the existing native models through the lens of input-output duality into three categories: (i) Multi-to-Text for cross-modal comprehension with text-only output; (ii) Multi-to-Target for scenario-oriented generation, e.g., image, audio and video generation, and (iii) Multi-to-Multi for unified modeling with symmetric input-output. We deliver a comprehensive and industrial-grade investigation into the transition toward the definitive NMM framework, where understanding and generation seamlessly coexist within a unified transformer paradigm. We systematically unpack the end-to-end pipeline from industrial perspectives from architectural coordination, massive data curation, to full-stack training recipes, inference & deployment, and the comprehensive evaluation for truly native modeling.

Siyu An, Junru Lu, Junnan Dong +18

Tencent Youtu Lab

Unified Multimodal Models Multimodal Generation Multimodal Foundation Models Multimodal Fusion Multimodal Understanding Multimodal Transformer

Ask PDF

Abstract

Unified Multimodal Models

Explore similar work

Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion

Lance: Unified Multimodal Modeling by Multi-Task Synergy

Toward Native Multimodal Modeling: A Roadmap