Visual generation with discrete tokens has gained significant attention as it enables a unified token prediction paradigm shared with language models, promising seamless multimodal architectures. However, current discrete generation methods remain limited to low-dimensional latent tokens (typically 8-32 dims), sacrificing the semantic richness essential for understanding. While high-dimensional pretrained representations (768-1024 dims) could bridge this gap, their discrete generation poses fundamental challenges. In this paper, we present Cubic Discrete Diffusion (CubiD), the first discrete generation model for high-dimensional representations. CubiD performs fine-grained masking throughout the high-dimensional discrete representation -- any dimension at any position can be masked and predicted from partial observations. This enables the model to learn rich correlations both within and across spatial positions, with the number of generation steps fixed at T regardless of feature dimensionality, where T≪hwd. On ImageNet-256, CubiD achieves state-of-the-art discrete generation with strong scaling behavior from 900M to 3.7B parameters. Crucially, we validate that these discretized tokens preserve original representation capabilities, demonstrating that the same discrete tokens can effectively serve both understanding and generation tasks. We hope this work will inspire future research toward unified multimodal architectures. Code is available at: https://github.com/YuqingWang1029/CubiD.
Figures & tables
Figure 1: Comparison of discrete visual generation approaches. (a) Low-dimensional token generation: Both methods operate at the spatial level—autoregressive requires h×w sequential steps, while discrete diffusion achieves parallel generation in T<h×w iterations. (b) High-dimensional token generation: Autoregressive becomes intractable ( h×w×d steps), and standard discrete diffusion cannot model intra-position dependencies. Our Cubic Discrete Diffusion performs fine-grained masking across the entire 3D tensor—any dimension at any position can be masked and predicted independently—enabling efficient generation in T≪h×w×d iterations while capturing both spatial and dimensional correlations.
Figure 2: Generated samples from CubiD. Class-conditional generation results on ImageNet 256×256 using high-dimensional representation tokens from DINOv2-B encoder, demonstrating fine details and textures across diverse categories.
Figure 3: Overview of Cubic Discrete Diffusion. (a) High-dimensional Token Discretization. Given an input image, a frozen representation encoder extracts continuous tokens, which are then discretized through dimension-wise quantization into h×w×d discrete tokens. (b) Training via Dimension-wise Mask Modeling. During training, we randomly mask tokens across both spatial and dimensional axes of the tensor (white: masked tokens, pink: visible ground truth tokens, other colors: predicted tokens). The transformer learns to predict these masked tokens from the unmasked context, capturing the complex dependencies across both spatial and dimensional axes.
Figure 4: Inference process of CubiD. Top row shows the latent token state (white: masked, pink: unmasked), bottom row shows corresponding decoded images. During generation, CubiD starts from a fully masked tensor (0%) and progressively unmasks tokens until reaching a complete image (100%). At each iteration, the model predicts all masked tokens in parallel and randomly unmasks a subset. The percentages show the progress through generation steps. Generation takes hundreds of iterations regardless of feature dimensionality, making high-dimensional discrete generation computationally feasible. The visualization demonstrates a coarse-to-fine generation process, where early iterations establish overall structure and later iterations refine details.
Model
Hidden Dim
Blocks
Parameters
CubiD-L
1536
32
946M
CubiD-XL
1920
32
1.4B
CubiD-XXL
3072
32
3.7B
Table 1: Model sizes and architecture configurations of CubiD.
Table 2: Effect of quantization levels on reconstruction quality. Both encoders achieve continuous-level performance with appropriate quantization levels (L=8 for DINOv2, L=16 for SigLIP2).
Tokenizer
Type
GQA
TextVQA
POPE
MME
SigLIP2
Continuous
63.2
59.6
85.4
1484
SigLIP2-VQ
Discrete
54.9
45.6
81.2
1189
SigLIP2-DQ
Discrete
63.1
59.8
85.0
1480
Table 3: Understanding performance on LLaVA benchmarks with different quantization methods. Evaluation using SigLIP2 features. VQ: vector quantization, DQ: dimension-wise quantization. DQ maintains continuous-level performance while VQ shows significant degradation.
Table 4: Ablation studies on CubiD design choices. Gray rows indicate best results.
Figure 5: Qualitative comparison of different masking strategies. Top row: Per-dim masking completely fails, producing severe texture-like artifacts. Middle row: Per-spatial masking generates images with significant local inconsistencies and blurry details. Bottom row: Our per-element masking produces clear, coherent images with fine details. The dramatic quality difference validates that high-dimensional tokens require fine-grained masking across both spatial and dimensional axes.
Method
Latent Dim
#Params
Generation@256 w/o guidance
Generation@256 w/ cfg or re
gFID ↓
IS ↑
Prec. ↑
Rec. ↑
gFID ↓
IS ↑
Prec. ↑
Rec. ↑
Discrete Diffusion Models with Low-dimensional Tokens
MaskGIT [ 3 ]
16
227M
6.18
182.1
0.80
0.51
4.02 re
355.6 re
-
-
VQ-Diffusion [ 12 ]
16
370M
11.89
-
-
-
-
-
-
-
Token-Critic [ 20 ]
16
368M
4.69
174.5
0.76
0.51
-
-
-
-
DPC [ 21 ]
16
454M
4.45
244.8
-
-
-
-
-
-
Table 5: Discrete generation methods on ImageNet [ 8 ] 256×256. Latent Dim denotes the original dimensionality of the latent space (features before vector quantization for low-dimensional methods, before and after dimension-wise quantization for CubiD). Results with superscript ”re” denote rejection sampling. CubiD is the first and only discrete method to directly generate with native high-dimensional representation tokens (768d), while all other methods use compressed or low-dimensional tokens (mostly below 32).
Method
Params (B)
gFID↓
IS↑
SiT-XL [ 27 ]
0.67
2.41
131.37
USiT-H [ 5 ]
0.50
1.89
174.58
USiT-2B [ 5 ]
1.58
1.72
187.68
CubiD
0.95
1.58
188.70
Table 6: CubiD on low-dimensional tokens on ImageNet 512×512. Results using DC-AE-f32c32 tokenizer producing 32-dimensional tokens.
Method
Token Dim
gFID↓
IS↑
CubiD
32
1.55
296.5
Table 7: CubiD with compressed representation tokens on ImageNet 256×256. Features compressed from 768d to 32d.
Figure 6: Uncurated samples on ImageNet 256×256 using CubiD-XXL conditioned on the specified classes.
Figure 7: Uncurated samples on ImageNet 256×256 using CubiD-XXL conditioned on the specified classes.
Unified multimodal generation covers discrete, continuous, and hybrid paradigms. Discrete approaches with single-codebook visual tokenizers align naturally with text modeling but have limited representational capacity. Continuous and hybrid approaches retain richer visual representations, but they model text and images in heterogeneous spaces with different training objectives, potentially introducing optimization conflicts between modalities with misaligned representation spaces. These choices expose a central question: how to efficiently model high-capacity visual representations alongside textual content in a shared optimization space within a single backbone. We present Intern Lumina U2, to our knowledge the first diffusion language model to represent multi-codebook visual tokens in a fully discrete space. Each visual position is represented by a group of codes from complementary codebooks, increasing its representational capacity while preserving compatibility with discrete text modeling. At the input, the embeddings are concatenated at each position; at the output, a lightweight multi-codebook autoregressive head predicts the codes within each position sequentially, while the diffusion backbone processes all positions in parallel. Built on the unified 4D representation of AToken, Intern Lumina U2 supports image, video, and 3D inputs within a single backbone and achieves leading performance among unified models in visual understanding, image generation, and image editing. We further build a full-stack training, inference, and evaluation system on both Ascend and NVIDIA platforms to verify the cross-platform effectiveness of our model. We also improve the inference efficiency through a stability-aware sampling strategy, which combines confidence-plateau-guided unmasking approach with drift-triggered cache refreshing mechanisms, accelerating image generation by 4.45x.
We propose Nemotron-Labs-Diffusion-Image, a state-of-the-art masked discrete diffusion model (MDM) for high-resolution text-to-image synthesis. Compared with prior work on masked image generation, Nemotron-Labs-Diffusion-Image addresses two key challenges. First, unlike continuous diffusion models which progressively refine latent representations across the entire image, standard MDMs lack self-correcting capability because discrete tokens cannot be modified once they are unmasked. Second, although increasing the vocabulary size of discrete image tokenizers improves reconstruction fidelity, it introduces optimization difficulties for generative modeling as the per-token training signal becomes increasingly sparse. To address the first challenge, Nemotron-Labs-Diffusion-Image incorporates a token-editing mechanism that enables the model to dynamically revise already-unmasked tokens during inference, similar to how a sculptor iteratively refines their work. To tackle the second challenge, we propose a Grouped Cross-Entropy (GCE) objective that assigns positive learning signals to tokens neighboring the ground truth in embedding space, thereby alleviating signal sparsity. To further improve training efficiency, we implement a custom fused operator for GCE that significantly reduces VRAM usage in large-vocabulary settings. Experimental results demonstrate that these innovations substantially improve both training efficiency and image fidelity of masked discrete image generators, achieving a score of 0.90 on GenEval, 86.9 on DPG and 10.76 of HPSv3.
Most visual tokenizers for image generation are bifurcated into two families with complementary limitations: continuous VAEs offer high-fidelity reconstruction but suffer from dense, entangled latents that are poorly suited for semantic control, whereas discrete VQ-based models enable autoregressive generation yet struggle with gradient sparsity, unstable training, and codebook collapse. In this work, we introduce MergeTok, a unified tokenizer that jointly optimizes continuous (VAE) and discrete (VQ) tokenizers within a encoder-decoder architecture, leveraging token merging techniques as a semantic bridge. By clustering similar tokens during encoding, MergeTok establishes a structural prior that provides dual supervision signals: (i) it imposes merged-token semantic alignment in the VAE branch, regularizing its latent space toward disentangled, semantic-aware representations; (ii) it derives group-wise constraints, promoting intra-group diversity and inter-group exclusivity that stabilize VQ training. MergeTok shows competitive reconstruction and generation performance on ImageNet-256, with substantially lower rFID than strong VAE and VQ models under matched token budgets, while producing semantically-organized token representations compatible with both autoregressive and diffusion generators. This shows that a single architecture can endow visual tokenizers with robust semantic organization and generator-friendly discreteness.
Luyuan Zhang, Siyuan Li, Zedong Wang +7
1Tsinghua University · 2Westlake University · 3Zhejiang University +3