Unified multimodal generation covers discrete, continuous, and hybrid paradigms. Discrete approaches with single-codebook visual tokenizers align naturally with text modeling but have limited representational capacity. Continuous and hybrid approaches retain richer visual representations, but they model text and images in heterogeneous spaces with different training objectives, potentially introducing optimization conflicts between modalities with misaligned representation spaces. These choices expose a central question: how to efficiently model high-capacity visual representations alongside textual content in a shared optimization space within a single backbone. We present Intern Lumina U2, to our knowledge the first diffusion language model to represent multi-codebook visual tokens in a fully discrete space. Each visual position is represented by a group of codes from complementary codebooks, increasing its representational capacity while preserving compatibility with discrete text modeling. At the input, the embeddings are concatenated at each position; at the output, a lightweight multi-codebook autoregressive head predicts the codes within each position sequentially, while the diffusion backbone processes all positions in parallel. Built on the unified 4D representation of AToken, Intern Lumina U2 supports image, video, and 3D inputs within a single backbone and achieves leading performance among unified models in visual understanding, image generation, and image editing. We further build a full-stack training, inference, and evaluation system on both Ascend and NVIDIA platforms to verify the cross-platform effectiveness of our model. We also improve the inference efficiency through a stability-aware sampling strategy, which combines confidence-plateau-guided unmasking approach with drift-triggered cache refreshing mechanisms, accelerating image generation by 4.45x.
Figures & tables
Supported modalities
Representation and decoding
Model
Image und.
Image gen.
Video und.
3D und.
Discrete tokens
Masked-diff. decoding
Multi- codebook
Diffusion text
No continuous generator
Autoregressive
Emu3
✓
✓
✓
✗
✓
✗
✗
✗
✓
Emu3.5
✓
✓
✗
✗
✓
✗
✗
✗
✓
Janus-Pro
✓
✓
✗
✗
✓
✗
✗
✗
✓
VILA-U
✓
✓
✓
✗
✓
✗
✓
✗
✓
Table 1: Capability and design checklist for representative unified multimodal models, grouped by backbone paradigm. Discrete tokens refers to the visual state inside the backbone; Masked-diff. decoding marks models that generate visual tokens by iterative masked denoising rather than left-to-right prediction; Multi-codebook denotes multiple codes per visual position with their dependence modeled explicitly; Diffusion text marks models that also generate text through the masked-diffusion process rather than autoregressively; No continuous generator marks systems that decode images through backbone tokens, without a separate continuous (diffusion or flow) generator. Systems: Emu3 ( Wang et al., 2026a ) , Emu3.5 ( Cui and others, 2025 ) , Janus-Pro ( Chen et al., 2025 ) , VILA-U ( Wu et al., 2025d ) , UniTok ( Ma et al., 2025a ) , BAGEL ( Deng et al., 2025 ) , InternVL-U ( Tian and others, 2026 ) , SenseNova-U1 ( SenseNova Team, 2026 ) , Show-o2 ( Xie et al., 2025b ) , Show-o ( Xie et al., 2025a ) , MMaDA ( Yang et al., 2025 ) , Lavida-O ( Li et al., 2026b ) , Lumina-DiMOO ( Xin et al., 2025 ) , LLaDA2.0-Uni ( Bie and others, 2026 ) .
Figure 3: Overall architecture of Intern Lumina U2. Visual inputs are encoded into visual tokens by AToken and text into text tokens; the MoE diffusion LLM recovers masked text tokens through the text head, with text answers generated by block diffusion, and masked visual tokens through the multi-codebook AR head, which predicts the K codes of each visual token sequentially.
Figure 4: Three candidate input interfaces for multi-codebook visual tokens. We adopt the third design: independent embeddings for the eight codebooks, concatenation at each spatial position, and projection into one backbone token.
Figure 5: Training and inference of the multi-codebook AR head: one teacher-forced causal pass yields all K logits at training; at inference the prefix grows for K steps, each sampled code c^k is embedded into ek and appended. Positions are processed in parallel, codebooks autoregressively.
Figure 6: Task-aware attention mask at block level. (a) Image answer: one noisy visual answer V~ after the clean context C , bidirectional over all positions. (b) Text answer: block diffusion over the noisy blocks b~1,b~2,b~3 and their clean copies b1,b2,b3 , with MBD , MOBC , and MBC . (c) A text answer followed by an image answer, as in thinking before generation: the image reads the clean copy of the preceding text blocks. (d) Multi-turn text with conditions C1,C2 , noisy answers x~1,x~2 , and clean copies x1,x2 : a later turn reads the clean copies of earlier answers.
Figure 7: Data construction pipeline. Six data categories are built with category-specific rules, converted to one record format, encoded offline, and re-weighted at each training stage. Bars show backbone-position shares (Table 2 ).
Stage
Position share (%)
Und.
T2I
Text
Edit
Video
3D
Vision–Language alignment
30
50
20
–
–
–
Perception pretraining
30
50
20
–
–
–
Omni-visual mid-training, Stage 1
30
60
10
–
–
–
Omni-visual mid-training, Stage 2
40
40
10
5
4
1
Instruction SFT
25
50
5
10
8
2
Table 2: Data mixture at each training stage. Shares are fractions of estimated backbone positions per pass over the mixture; the corresponding resolution and optimization settings are given in Table 3 .
Configuration
Vision- Language alignment
Perception pretraining
Omni-visual mid-training
Instruction SFT
Stage 1
Stage 2
Training setup
Generation resolution
256
512
1024
1024
1024
Sequence length
8,192
8,192
16,384
16,384
16,384
Text length limit
2,048
2,048
2,048
2,048
2,048
Text block length
32
32
32
32
32
Table 3: Training configurations for the primary Ascend run and the NVIDIA reference line.
Method
Steps
BB fwd
Lat. (s)
Speedup
GenEval
Cumulative system optimizations (fixed 96-step schedule)
Production path
96.0
192
159.9
1.00 ×
86.6
+ CFG batching
96.0
192
117.2
1.36 ×
89.0
+ AR-head KV cache
96.0
192
84.2
1.90 ×
88.6
+ hidden-state cache
96.0
192 (65%)
64.1
2.49 ×
89.2
+ V-verify refresh
96.0
192 (63%)
64.5
2.48 ×
89.3
Table 4: Inference acceleration under a matched 1024×1024 text-to-image protocol with the same checkpoint, 553 GenEval prompts, one image per prompt, CFG scale 4.0 , temperature 0.9 , and seed 42 . “Steps” is the realized number of outer diffusion steps, prompt-dependent for the plateau rows and fixed elsewhere. “BB fwd” counts conditional plus unconditional backbone forwards, with the cache-hit ratio in parentheses where a cache is active. Latency is wall-clock per image after a discarded kernel preheat and includes cache warm-up. The selected PHV setting is δp=0.18 , fp=0.005 , with the remaining cache and V-verify controls fixed across rows. GenEval uses one image per prompt and is therefore not comparable to the four-seed protocol of Table 7 ; no configuration scores below the production path.
Figure 8: Confidence calibration for masked visual positions under the fixed 96 -step, 1024×1024 protocol. The visual confidence remains below the text-scale commit threshold, and its trajectory provides a more useful signal for adaptive sampling than its absolute level.
Figure 9: Offline temporal-stability test for PHV on 200 prompts. Each plateau-eligible position is paired with a still-masked matched-control token from the same image that does not satisfy the plateau criterion while exhibiting comparable confidence and EMA values. Both are held masked for 2 , 4 , or 8 additional refinement steps. (a) The difference in top-1 flip rate between the plateau token and its matched-control token grows with the hold duration. (b) Per-prompt flip rates at H=8 ; 157 of 200 prompts fall below the diagonal. (c) Posterior max-probability trajectories for three representative plateau–matched-control pairs.
Understanding-only
Unified
Intern Lumina U2
Benchmark
Qwen2.5- VL
LLaDA-V
BAGEL
Intern VL-U
Lumina- DiMOO
LLaDA-o
LLaDA 2.0-Uni
Ascend
NVIDIA
#Params
7B
8B
7B+7B
2B+1.7B
8B
8B
16B-A1.4B
16.9B-A1.4B
Documents and charts
ChartQA
84.1
78.3
74.3
76.6
8.3
87.9
80.1
86.2
81.2
CharXiv (DQ)
73.9
47.0
70.6
53.3
27.8
69.8
68.4
85.9
84.0
Perception and hallucination
Table 5: Image understanding and reasoning, grouped into document and chart understanding, perception and hallucination, and knowledge and mathematical reasoning. Benchmarks: ChartQA ( Masry et al., 2022 ) , CharXiv ( Wang et al., 2024b ) , HallusionBench ( Guan et al., 2024 ) , MME ( Fu et al., 2025a ) , RealWorldQA ( xAI, 2024 ) , MMMU ( Yue et al., 2024 ) , MMMU-Pro ( Yue et al., 2025 ) , MathVista ( Lu et al., 2024 ) , MathVision ( Wang et al., 2024a ) , and DynaMath ( Zou et al., 2025 ) . Baseline numbers are taken from Bie and others (2026) . Bold marks the best score in each row.
Model
Video-MME
MLVU
MVBench
Video-LLaVA-7B
39.9
–
43.1
LLaVA-Mini
–
42.8
44.5
VideoLLaMA2-7B
46.6
–
54.6
LLaDA-V
56.1
59.5
–
Ours (Ascend)
52.6
62.1
61.8
Ours (NVIDIA)
56.7
64.9
57.2
Table 6: Video and 3D understanding from the same checkpoint. Left: video benchmarks (Video-MME ( Fu et al., 2025b ) without subtitles, MLVU ( Zhou et al., 2025b ) m-avg, MVBench ( Li et al., 2024 ) ); baseline numbers are taken from Lin et al. (2024) ; Zhang et al. (2025) ; Cheng et al. (2024) ; You et al. (2026a) ; Fu et al. (2025b) . Right: 3D MM-Vet ( Qi et al., 2024 ) , with baseline numbers for ShapeLLM and PointLLM ( Xu et al., 2024 ) taken from Qi et al. (2024) . Bold marks the best score in each column.
Model
#Params
GenEval
DPG-Bench
TIIF
WISE
Generation-only models
FLUX.1 [Dev]
12B
66
83.84
71.8
0.50
Qwen-Image
20B
87
88.32
86.8
0.62
Unified understanding-and-generation models
Janus-Pro
7B
80
84.19
65.0
0.35
Show-o2
7B
76
86.14
–
0.39
Table 7: Text-to-image generation. GenEval ( Ghosh et al., 2023 ) , DPG-Bench ( Hu et al., 2024 ) , TIIF-Bench ( Wei et al., 2026 ) , and WISE ( Niu et al., 2026 ) overall scores (GenEval on a 0–100 scale); higher is better. Baseline numbers are taken from Bie and others (2026) , Tian and others (2026) , and Xie et al. (2025b) ; TIIF reports the overall score on long prompts. UniGenBench results are reported per dimension in Table 8 . Bold marks the best unified-model score in each column
Model
Sty
Wor
Att
Act
Rel
Log
Gra
Com
Lay
Txt
Overall
Generation-only models
FLUX.1 [Dev]
83.90
88.92
67.84
62.17
67.26
30.91
60.96
47.04
71.83
32.18
61.30
Qwen-Image
94.70
94.15
87.93
82.60
80.08
51.59
60.96
72.94
86.57
72.13
78.36
Unified understanding-and-generation models
Janus-Pro
90.80
86.71
67.74
64.26
68.40
37.05
64.44
62.11
72.01
2.59
61.61
BAGEL
90.20
85.60
67.74
61.98
70.69
30.23
66.44
58.12
76.49
7.76
61.53
Table 8: UniGenBench ( Wang et al., 2025b ) (EN) results on the ten first-level dimensions; Overall is their mean. Baseline numbers are taken from Bie and others (2026) ; bold marks the best unified-model score in each column. Dimensions: Sty(le), Wor(ld knowledge), Att(ribute), Act(ion), Rel(ationship), Log(ical reasoning), Gra(mmar), Com(pound), Lay(out), Txt (text generation).
Model
Short prompts
Long prompts
Basic
Adv.
Real
Overall
Basic
Adv.
Real
Overall
Generation-only models
FLUX.1 [Dev]
83.1
65.8
70.7
71.1
78.7
68.5
71.5
71.8
Qwen-Image
86.2
79.3
90.3
86.1
87.2
80.9
91.4
86.8
Unified understanding-and-generation models
Janus-Pro
79.3
59.7
65.8
65.5
78.3
58.8
60.3
65.0
Table 9: TIIF-Bench ( Wei et al., 2026 ) results by instruction category for short and long prompts. Basic and Advanced are the average of their sub-categories; Real World is the Designer real-world score. Baseline numbers are taken from Tian and others (2026) ; bold marks the best unified-model score in each column.
Model
Alignment
Text
Reasoning
Style
Diversity
Overall
Generation-only models
FLUX.1 [Dev]
0.786
0.523
0.253
0.368
0.238
0.434
Qwen-Image
0.882
0.891
0.306
0.418
0.197
0.539
Unified understanding-and-generation models
Janus-Pro
0.553
0.001
0.139
0.276
0.365
0.267
BAGEL
0.769
0.244
0.173
0.367
0.251
0.361
Table 10: OneIG-Bench ( Chang et al., 2025 ) (EN). Baseline numbers are taken from Bie and others (2026) ; Overall is the mean of the five dimensions. Bold marks the best unified-model score in each column.
Model
GenExam Physics
VeriphyT2IBench
Strict
Relaxed
Local
Global
Reference and generation-only models
Seedream 4.0
3.5
49.0
64.80
65.07
FLUX.1 Kontext max
0.0
25.6
–
–
Gemini 2.5 Flash Image
–
–
69.98
66.14
GPT-Image-1
–
–
68.32
68.56
Table 11: Physics-faithful scientific-diagram generation on the physics subset of GenExam and on VeriphyT2IBench. GenExam reports aggregate strict and relaxed MLLM-judge scores; VeriphyT2IBench reports question-level accuracy on its local attribute bank and global whole-diagram bank. Baseline values and protocol definitions follow Zhang et al. (2026) ; Princigram uses physics-specific structured supervision and prompting, whereas the other rows are general-purpose generators. Bold marks the best score in each column.
Local scoring
Seedream
Gemini
GPT-Image
SenseNova
Princigram-D
Princigram-B
Intern Lumina U2
E&M
65.55
67.75
69.58
53.46
76.00
78.59
83.30
Mechanics
56.19
61.63
60.25
52.16
68.24
71.59
71.34
Optics
77.81
82.33
79.54
80.96
75.64
78.92
79.23
Quantum mechanics
69.67
73.27
71.35
68.93
73.41
77.59
76.32
Thermodynamics
42.91
58.90
49.24
46.24
60.40
66.95
63.73
Acoustics
62.42
68.44
66.88
65.26
68.39
73.01
72.17
Table 12: VeriphyT2IBench domain-level breakdown for representative models. Each panel reports question-level accuracy (percent; higher is better) for the local attribute bank or the global whole-diagram bank. The overall rows are omitted because the corresponding aggregate scores are already reported in Table 11 .
Easy prompts
Seedream
Gemini
GPT-Image
SenseNova
Princigram-D
Princigram-B
Intern Lumina U2
E&M
75.32
80.56
80.41
75.45
82.11
84.74
83.78
Mechanics
66.08
67.84
68.19
64.96
74.41
76.77
75.40
Optics
80.81
83.21
81.79
82.98
78.05
79.01
81.37
Quantum mechanics
70.97
74.13
75.41
73.39
75.58
81.34
81.47
Thermodynamics
64.75
66.77
67.66
64.61
68.93
72.13
69.30
Acoustics
64.88
69.86
69.19
69.72
67.84
71.13
71.94
Table 13: VeriphyT2IBench breakdown by prompt difficulty. Scores are percentages and higher is better. The panels report the six physics domains at easy, medium, and hard difficulty for the same representative models as Table 12 .
Model
#Params
ImgEdit
GEdit-EN
GEdit-CN
Editing-specialist models
Step1X-Edit
12B
3.06
6.97
6.86
Qwen-Image-Edit
20B
4.35
7.56
7.52
Unified understanding-and-generation models
BAGEL
7B+7B
3.20
6.52
6.50
OmniGen2
3B+4B
3.44
6.41
–
Table 14: Instruction-based image editing. ImgEdit ( Ye et al., 2025b ) overall score (1–5) and GEdit-Bench ( Liu et al., 2025b ) overall score GO (0–10) for English and Chinese prompts. Baseline numbers are taken from Bie and others (2026) and Tian and others (2026) . Bold marks the best unified-model score in each column.
Design
Table organization
Softmax width
Params
Peak memory (GB)
Step time (s)
Next-code accuracy
D1 universal
1×189,968
189,968
778.11M
43.79
7.1
0.0426±0.0002
D2 offset-shared
1×32,784
32,784
778.11M
18.68
6.2
0.0415±0.0002
D3 independent (ours)
8×4,096
4,096
793.02M
18.62
6.6
0.0459±0.0002
Table 15: Ablation of the visual input interface. Each design determines both the lookup-table organization and the corresponding classifier width. Accuracy is next-code top-1 on held-out supervised codes at the final checkpoint ( ≈5 M codes per arm; ± one standard error). Memory and step time are measured at sequence length 4608 on identical hardware. Bold marks the best value in each column.
Figure 10: Input-interface ablation. (a) Final next-code top-1 accuracy for each of the eight codebooks, normalized by D3; the blue annotations give D3’s absolute accuracy at the first and last codebooks. (b) Image loss over the two training epochs, shown as a rolling median with an inset for the converged tail. Higher is better in (a), whereas lower is better in (b).
Output head
Codebook dependence
GenEval
DPG-Bench
TIIF
UniGenBench
Independent parallel
no
68.94
83.22
70.1
59.6
Q-Former
no
69.67
78.92
63.9
56.7
Multi-codebook AR (ours)
yes
82.78
83.27
73.2
62.1
Table 16: Ablation of the visual output head: three training runs at 512-pixel generation resolution with comparable token budgets (about 197B–220B tokens each). All scores are on a 0–100 scale; TIIF is the mean accuracy and UniGenBench the EN ten-dimension average. Bold marks the best score in each column.
Figure 11: Inference-sampling ablation over the plateau change threshold δp and confidence floor fp , 553 prompts per cell. (a) GenEval over the 6×6 grid; the black box marks the selected quality-first setting δp=0.18 , fp=0.005 . (b) Speed–quality frontier over all 36 cells, i.e. the configurations not improved on in both latency and quality by any other cell; the selected setting (red) reaches GenEval 89.3 at 35.9 s per image ( 4.45× ).
Table 18: Special tokens. Ids in the lower block index the per-codebook visual embedding tables; all other ids index the text vocabulary.
idx
token
type
turn
is_clean
block
position id
Turn 0: clean condition (media + question)
0
<BOI>
1
0
1
−1
0
1–2
v00v01
2
0
1
−1
1–2
3
┘
2
0
1
−1
3
4–5
v10v11
2
0
1
−1
4–5
6
┘
2
0
1
−1
6
Appendix
Table 19: A packed image-understanding sample with per-position metadata. In this example, type 1 denotes text or structural tokens, type 2 a media-condition token, type 3 a text-answer token, and type 4 a visual-answer token. The clean user-provided condition and the assistant answer form one turn, and the clean copy of the answer belongs to the same turn. The is_clean flag is 1 for clean tokens and 0 for noisy tokens, the block id is −1 for conditions and 0 for this answer block, and a clean copy reuses the position ids of its noisy counterpart. Here vij is the visual token ( K codes) at grid row i , column j ; ┘ is the image-row delimiter (code 5001); qi and ai are question and answer tokens; and M is <|mask|> . Labels are −100 at all positions that do not participate in supervision; only masked answer positions contribute to the loss.
Modality
Grid
Flattening order
Delimiter tokens
Image H×W
(H/16)×(W/16)
row-major (y,x)
5001 per row
Video T×H×W
⌈T/4⌉×(H/16)×(W/16)
frame-major (t,y,x)
5004 per row, 5005 per frame
3D asset
sparse voxels, coords (x,y,z)
(z,y,x)
5002 per row, 5003 per layer
Appendix
Table 20: Visual token layouts. Patch size is (t,h,w)=(4,16,16) ; every grid position carries K=8 codes.
Category
Type
Sources
Encoding resolution
Stages
Image understanding
Public
LLaVA-OneVision ( Li et al., 2025 ) and LLaVA-OneVision-1.5 ( An et al., 2025 ) instruction sets
256 / 512 / native
1–5
Private
Image subsets of the InternVL ( Wang et al., 2025a ) instruction data, covering captioning; document, chart, table, and equation data; OCR data built from web image–text pairs; grounding, counting, and GUI or web screenshot data; and structured-reasoning data and samples with <think> reasoning traces
Video understanding
Public
Caption and QA sets over WebVid ( Bain et al., 2021 ) , Kinetics-400/600/700 ( Kay et al., 2017 ; Carreira et al., 2018 ; Carreira et al., 2019 ) , ActivityNet ( Caba Heilbron et al., 2015 ) , YouCook2 ( Zhou et al., 2018 ) , LSMDC ( Rohrbach et al., 2017 ) , TextVR ( Wu et al., 2025c ) , NTU RGB+D ( Shahroudy et al., 2016 ) , NExT-QA ( Xiao et al., 2021 ) , CLEVRER ( Yi et al., 2020 ) , STAR ( Wu et al., 2021 ) , Perception Test ( Pătrăucean et al., 2023 ) , EgoSchema ( Mangalam et al., 2023 ) , and MovieChat ( Song et al., 2024 )
64 frames, 4482 area
4–5
Private
Video subsets of the InternVL instruction data; temporal-reasoning and counting sets
3D understanding
Public
Objaverse ( Deitke et al., 2023 ) assets with Cap3D-style captions ( Luo et al., 2023 ) ; GAPartNet ( Geng et al., 2023 ) part-level annotations; Sketchfab ( Sketchfab, 2026 ) assets
sparse voxels
4–5
Private
–
Appendix
Table 21: Sources by data category, with the public and private sources of each category listed separately. Encoding resolution is the long side (images and editing) or the frame area budget (video); “native” means the original resolution clamped to a [256,1024] long side. Stages are numbered 1–5 as in Section 5 .
Source
Scored
Text rule
Following <60
Aesthetic top 70%
Kept
English
22.69
− 4.33
− 4.75
− 3.09
10.51 (46%)
Chinese
31.35
− 6.92
− 6.26
− 4.15
14.02 (45%)
Appendix
Table 22: Quality filtering of the curated text-to-image sources (counts in millions).
Figure 12: Qualitative examples for text-to-image generation and omni-visual understanding.
Figure 13: Qualitative examples for image editing.
We present LLaDA2.0-Uni, a unified discrete diffusion large language model (dLLM) that supports multimodal understanding and generation within a natively integrated framework. Its architecture combines a fully semantic discrete tokenizer, a MoE-based dLLM backbone, and a diffusion decoder. By discretizing continuous visual inputs via SigLIP-VQ, the model enables block-level masked diffusion for both text and vision inputs within the backbone, while the decoder reconstructs visual tokens into high-fidelity images. Inference efficiency is enhanced beyond parallel decoding through prefix-aware optimizations in the backbone and few-step distillation in the decoder. Supported by carefully curated large-scale data and a tailored multi-stage training pipeline, LLaDA2.0-Uni matches specialized VLMs in multimodal understanding while delivering strong performance in image generation and editing. Its native support for interleaved generation and reasoning establishes a promising and scalable paradigm for next-generation unified foundation models. Codes and models are available at https://github.com/inclusionAI/LLaDA2.0-Uni.
Diffusion Transformers (DiT) trained with flow matching in a VAE latent space have unified visual generation across images and videos. A natural next step toward a single architecture for both generation (visual synthesis) and understanding (text generation) is to apply this framework to language modeling. We propose TextLDM, which transfers the visual latent diffusion recipe to text generation with minimal architectural modification. A Transformer-based VAE maps discrete tokens to continuous latents, enhanced by Representation Alignment (REPA) with a frozen pretrained language model to produce representations effective for conditional denoising. A standard DiT then performs flow matching in this latent space, identical in architecture to its visual counterpart. The central challenge we address is obtaining high-quality continuous text representations: we find that reconstruction fidelity alone is insufficient, and that aligning latent features with a pretrained language model via REPA is critical for downstream generation quality. Trained from scratch on OpenWebText2, TextLDM substantially outperforms prior diffusion language models and matches GPT-2 under the same settings. Our results establish that the visual DiT recipe transfers effectively to language, taking a concrete step toward unified diffusion architectures for multimodal generation and understanding.
Visual generation with discrete tokens has gained significant attention as it enables a unified token prediction paradigm shared with language models, promising seamless multimodal architectures. However, current discrete generation methods remain limited to low-dimensional latent tokens (typically 8-32 dims), sacrificing the semantic richness essential for understanding. While high-dimensional pretrained representations (768-1024 dims) could bridge this gap, their discrete generation poses fundamental challenges. In this paper, we present Cubic Discrete Diffusion (CubiD), the first discrete generation model for high-dimensional representations. CubiD performs fine-grained masking throughout the high-dimensional discrete representation -- any dimension at any position can be masked and predicted from partial observations. This enables the model to learn rich correlations both within and across spatial positions, with the number of generation steps fixed at T regardless of feature dimensionality, where T≪hwd. On ImageNet-256, CubiD achieves state-of-the-art discrete generation with strong scaling behavior from 900M to 3.7B parameters. Crucially, we validate that these discretized tokens preserve original representation capabilities, demonstrating that the same discrete tokens can effectively serve both understanding and generation tasks. We hope this work will inspire future research toward unified multimodal architectures. Code is available at: https://github.com/YuqingWang1029/CubiD.
Yuqing Wang, Chuofan Ma, Zhijie Lin +7
University of Hong Kong · ByteDance Seed · Carnegie Mellon University +2