Discrete diffusion models with parallel decoding are often distilled using token-wise KL objectives. Although token-wise KL exactly matches the corresponding product-form block distributions, its probability-space optimization geometry and teacher-only candidate truncation may provide insufficient supervision under aggressive few-step distillation. Concrete Score Matching (CSM) provides an alternative within-position objective by aligning relative logit geometry, but its complete pairwise form incurs \mathcal{O}(|\mathcal{V}|^2) complexity. We propose Bidirectional Centered Score Matching (BCSM), an efficient within-position distillation objective that constructs an \mathcal{O}(K) bidirectional active set, reduces pairwise comparisons to \mathcal{O}(K^2), and factorizes them into an \mathcal{O}(K) objective. We further introduce cross-timestep transition alignment. Experiments demonstrate strong quality--efficiency trade-offs, including up to 6\times NFE reduction with a 16.9% relative HumanEval gain and a 15.1% relative MMMU gain at single-step inference.
Figures & tables
Figure 1: Conceptual comparison of discrete distillation objectives. (a) Vanilla KL Divergence aligns position-wise categorical probabilities with linear O(∣V∣) complexity. (b) Concrete Score Matching aligns pairwise relative probability ratios but incurs O(∣V∣2) complexity. (c) Bidirectional Centered Score Matching restricts evaluation to a dynamic subspace ( K≪∣V∣ ) and applies mean-centering factorization, reducing the pairwise matching complexity to O(K) while preserving relative logit differences within the active set.
Figure 2: Kneedle-based extraction of the dynamic active subspace. (a) The sorted logits are normalized to form the logit profile yi over normalized rank xi ; the prefix up to kt defines the active subspace Ωt . (b) The distance di=(1−xi)−yi reaches its maximum at kt=argmaxidi , which determines the adaptive boundary (red dotted line). The same procedure is applied independently to the student logits to obtain ks and Ωs .
Benchmark
Method
LLaDA-8B
Dream-7B
NFE ↓
Score ↑
NFE ↓
Score ↑
GSM8K (4-shot)
Base Model
256.0
77.4
256.0
75.0
Fast-dLLM
107.5
78.9
136.4
75.0
D2F
87.7
77.3
128.3
77.7
CDLM
89.7
73.9
125.2
78.5
Ours
86.9
77.6
126.7
77.8
Table 1: Performance comparison on diffusion language models. Results for LLaDA-8B-Instruct and Dream-7B-Base on math and coding benchmarks. Undistilled base models (NFE=256.0) serve as references. Per CDLM ( Kim et al. 2025 ) , fractional NFEs denote average evaluations per block (BS=256).
Method
Bard-VL
MMMU
RWQA
AI2D
Math
MMVU
Avg.
Teacher (NFE=4)
35.26
72.68
82.32
67.33
43.90
60.30
Student (NFE=32)
32.14
68.10
79.79
61.93
37.50
55.89
NFE = 1
Student
30.98
68.24
77.66
40.40
35.30
50.52
CSM
OOM
Table 2: Performance comparison on Bard-4B-VL under varying NFEs. Teacher (BS=4, NFE=4) and undistilled student (BS=32, NFE=32) serve as references; others use BS=32. “OOM” denotes out of memory.
Table 3: Comprehensive ablation studies of BCSM on the OCR benchmark.
Figure 3: Efficiency and performance comparison on diffusion language models. The top row reports LLaDA-8B results on MBPP and MATH, while the bottom row reports Dream-7B results on GSM8K and HumanEval. Each marker denotes a distillation method, with the number of function evaluations (NFE) on the x-axis and task performance on the y-axis. Lower NFE and higher scores are preferred. Across the four benchmarks, BCSM lies on the efficiency–performance Pareto frontier, attaining higher scores at similar NFE and comparable scores with substantially fewer evaluations.
Figure 4: Comparison of non-reference n-gram rates (lower is better). BCSM yields lower reference mismatch across all n-gram orders than the baselines, providing empirical evidence of improved local consistency.
Figure 5: Qualitative comparison of generated responses ( NFE=16 ). Red text indicates incorrect or fragmented spans.
Figure 6: Qualitative comparison of generated responses across distillation methods. In these examples, the KL-based baselines exhibit spelling errors, token repetitions, or malformed compositions, whereas BCSM produces outputs with higher reference agreement. These examples illustrate empirical differences in output quality and do not imply explicit recovery of cross-position dependencies.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Setting
Model and distillation
Training variant
BARD-VL mask diffusion
Teacher
Frozen BARD-VL-4B with block size BT=4
Student
BARD-VL-4B with block size BS=32
Distillation route
Direct single-stage cross-block distillation: 4B(BT=4)→4B(BS=32)
Trainable parameters
All student parameters
Appendix
Table 4: Explicit hyperparameters for BCSM distillation under the BARD-VL mask-diffusion setting. Both teacher and student use the BARD-VL-4B backbone. A frozen teacher with block size 4 directly supervises a student with block size 32 , without progressive block merging or multi-stage distillation.
Configuration
Setting
Model and distillation
Backbones
LLaDA-Instruct-8B and Dream-Base-7B
Teacher
Frozen pretrained diffusion language model with bidirectional full attention
Student initialization
Initialized from the corresponding pretrained teacher backbone
Trainable parameters
Student LoRA parameters
Optimization
Appendix
Table 5: Explicit hyperparameters for BCSM distillation under the D2F setting. The original D2F backbone initialization, LoRA optimization, sequence partitioning, and block-wise masking range are retained, while its asymmetric KL objective is replaced by BCSM.
Figure 7: Qualitative comparison of cases where all methods fail to recover the exact reference. In these examples, KL-based methods produce fragmented tokens or repetitions, whereas BCSM retains longer recognizable spans and more complete local structures. The comparison reports an empirical property of the generated outputs rather than explicit cross-position dependency modeling.
It is currently difficult to distill discrete diffusion models. In contrast, continuous diffusion literature has many distillation approaches methods that can reduce sampling steps to a handful. Our method, Discrete Moment Matching Distillation (D-MMD), leverages ideas that have been highly successful in the continuous domain. Whereas previous discrete distillation methods collapse, D-MMD maintains high quality and diversity (given sufficient sampling steps). This is demonstrated on both text and image datasets. Moreover, the newly distilled generators can outperform their teachers.
Continuous diffusion language models generate all tokens in parallel, yet high-quality generation can still require hundreds of network evaluations (NFEs). We study how distributional distillation can reduce this cost by exploiting the student's probabilistic token outputs. Our unified formulation connects the student's output parameterization to the resulting gradient estimators and yields two methods with the same student architecture and reverse-KL matching objective: Simplex-DMD uses continuous token relaxations and pathwise gradients, while Reinforce-DMD uses categorical sampling and REINFORCE with a learned density ratio. We develop both methods for multi-step generation and investigate the training and sampling choices associated with each parameterization. On OpenWebText, for sequences of 1,024 tokens, Simplex-DMD achieves a generative perplexity of 45.6 at a unigram entropy of 5.44 nats in just 4 NFEs, a 49% reduction relative to the strongest evaluated diffusion baseline at matched entropy and sampling budget. Reinforce-DMD improves the frontier at larger budgets, reaching a generative perplexity of 14.9 at an entropy of 5.00 nats with 256 NFEs, a 20% reduction under the same comparison protocol.
Paul Le Van Kiem, Dario Shariatian, Umut Simsekli +1
Inria, PSL Research University · Cohere · CMAP, Ecole Polytechnique
Numerous distillation methods have been developed to accelerate diffusion models. Recent research indicates that hybrid strategies---blending different paradigms---yield superior results compared to single-method approaches. However, current works predominantly rely on a simple combination of objectives, leaving the question of how to effectively synergize them largely underexplored. In this paper, we specifically investigate the synergy between Score Distillation and Adversarial Distillation. We reveal that their operating zones are distinct and complementary: Score Distillation is effective in high-noise regimes for capturing the global distribution yet lacks fine-grained supervision in low-noise regions. Conversely, Adversarial Distillation is effective in low-noise regions but is prone to instability and suffers from discriminator failure when applied to high-noise intervals. Leveraging this insight, we propose Efficient Adversarial Score Distillation (EASD), a framework that synergizes these objectives via a regime-adaptive sampling strategy, directing each objective to focus on its effective interval. We further apply this paradigm to DiT-based Flow Matching models. Empirically, our method achieves a promising one-step FID of 1.39 on ImageNet 256×256 using the SiT-XL/2+REPA model. This work provides a concrete guideline for maximizing the potential of hybrid distillation. Our code and model are available at https://github.com/pf0607/EASD .
Fei Peng, Junqiang Wu, Haoxian Tan +5
Tsinghua University · Peng Cheng Laboratory · Meituan Inc.