Bidirectional Centered Score Matching for Discrete Diffusion Distillation

Peng Tu; Baoyou Chen; Haoyuan Xia; Zilong Dong; Weihao Yuan; Siyu Zhu

cs.AINov 3, 2026

Bidirectional Centered Score Matching for Discrete Diffusion Distillation

Authors: Peng Tu, Baoyou Chen, Haoyuan Xia, Zilong Dong, Weihao Yuan, Siyu Zhu

Organizations: Shanghai Academy of AI for Science · Fudan University · Shanghai Innovation Institute · Alibaba Group ·

Abstract

Discrete diffusion models with parallel decoding are often distilled using token-wise KL objectives. Although token-wise KL exactly matches the corresponding product-form block distributions, its probability-space optimization geometry and teacher-only candidate truncation may provide insufficient supervision under aggressive few-step distillation. Concrete Score Matching (CSM) provides an alternative within-position objective by aligning relative logit geometry, but its complete pairwise form incurs \mathcal{O}(|\mathcal{V}|^2) complexity. We propose Bidirectional Centered Score Matching (BCSM), an efficient within-position distillation objective that constructs an \mathcal{O}(K) bidirectional active set, reduces pairwise comparisons to \mathcal{O}(K^2), and factorizes them into an \mathcal{O}(K) objective. We further introduce cross-timestep transition alignment. Experiments demonstrate strong quality--efficiency trade-offs, including up to 6\times NFE reduction with a 16.9% relative HumanEval gain and a 15.1% relative MMMU gain at single-step inference.

Figures & tables

Panel of Figure 1 — Figure 1: Conceptual comparison of discrete distillation objectives. (a) Vanilla KL Divergence aligns position-wise categorical probabilities with linear $\mathcal{O}(|\mathcal{V}|)$ complexity. (b) Concrete Score Matching aligns pairwise relative probability ratios but incurs $\mathcal{O}(|\mathcal{V}|^{2})$ complexity. (c) Bidirectional Centered Score Matching restricts evaluation to a dynamic subspace ( $K\ll|\mathcal{V}|$ ) and applies mean-centering factorization, reducing the pairwise matching complexity to $\mathcal{O}(K)$ while preserving relative logit differences within the active set.

Figure 2: Kneedle-based extraction of the dynamic active subspace. (a) The sorted logits are normalized to form the logit profile

y_{i}

over normalized rank

x_{i}

; the prefix up to

k^{t}

defines the active subspace

\Omega_{t}

. (b) The distance

d_{i}=(1-x_{i})-y_{i}

reaches its maximum at

k^{t}=\arg\max_{i}d_{i}

, which determines the adaptive boundary (red dotted line). The same procedure is applied independently to the student logits to obtain

k^{s}

and

\Omega_{s}

Benchmark	Method	LLaDA-8B		Dream-7B
Benchmark	Method	NFE $\downarrow$	Score $\uparrow$	NFE $\downarrow$	Score $\uparrow$
GSM8K (4-shot)	Base Model	256.0	77.4	256.0	75.0
	Fast-dLLM	107.5	78.9	136.4	75.0
	D2F	87.7	77.3	128.3	77.7
	CDLM	89.7	73.9	125.2	78.5
	Ours	86.9	77.6	126.7	77.8

Table 1: Performance comparison on diffusion language models. Results for LLaDA-8B-Instruct and Dream-7B-Base on math and coding benchmarks. Undistilled base models (NFE=256.0) serve as references. Per CDLM ( Kim et al. 2025 ) , fractional NFEs denote average evaluations per block (BS=256).

Method	Bard-VL
Method	MMMU	RWQA	AI2D	Math	MMVU	Avg.
Teacher (NFE=4)	35.26	72.68	82.32	67.33	43.90	60.30
Student (NFE=32)	32.14	68.10	79.79	61.93	37.50	55.89
NFE = 1
Student	30.98	68.24	77.66	40.40	35.30	50.52
CSM	OOM

Table 2: Performance comparison on Bard-4B-VL under varying NFEs. Teacher (BS=4, NFE=4) and undistilled student (BS=32, NFE=32) serve as references; others use BS=32. “OOM” denotes out of memory.

Table 3: Comprehensive ablation studies of BCSM on the OCR benchmark.

Figure 3: Efficiency and performance comparison on diffusion language models. The top row reports LLaDA-8B results on MBPP and MATH, while the bottom row reports Dream-7B results on GSM8K and HumanEval. Each marker denotes a distillation method, with the number of function evaluations (NFE) on the x-axis and task performance on the y-axis. Lower NFE and higher scores are preferred. Across the four benchmarks, BCSM lies on the efficiency–performance Pareto frontier, attaining higher scores at similar NFE and comparable scores with substantially fewer evaluations.

Figure 4: Comparison of non-reference n-gram rates (lower is better). BCSM yields lower reference mismatch across all n-gram orders than the baselines, providing empirical evidence of improved local consistency.

Figure 5: Qualitative comparison of generated responses ( $\mathrm{NFE}=16$ ). Red text indicates incorrect or fragmented spans.

Figure 6: Qualitative comparison of generated responses across distillation methods. In these examples, the KL-based baselines exhibit spelling errors, token repetitions, or malformed compositions, whereas BCSM produces outputs with higher reference agreement. These examples illustrate empirical differences in output quality and do not imply explicit recovery of cross-position dependencies.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Setting
Model and distillation
Training variant	BARD-VL mask diffusion
Teacher	Frozen BARD-VL-4B with block size $B_{\mathrm{T}}=4$
Student	BARD-VL-4B with block size $B_{\mathrm{S}}=32$
Distillation route	Direct single-stage cross-block distillation: $4\mathrm{B}\,(B_{\mathrm{T}}=4)\rightarrow 4\mathrm{B}\,(B_{\mathrm{S}}=32)$
Trainable parameters	All student parameters

Appendix

Table 4: Explicit hyperparameters for BCSM distillation under the BARD-VL mask-diffusion setting. Both teacher and student use the BARD-VL-4B backbone. A frozen teacher with block size

4

directly supervises a student with block size

32

, without progressive block merging or multi-stage distillation.

Configuration	Setting
Model and distillation
Backbones	LLaDA-Instruct-8B and Dream-Base-7B
Teacher	Frozen pretrained diffusion language model with bidirectional full attention
Student initialization	Initialized from the corresponding pretrained teacher backbone
Trainable parameters	Student LoRA parameters
Optimization

Appendix

Table 5: Explicit hyperparameters for BCSM distillation under the D2F setting. The original D2F backbone initialization, LoRA optimization, sequence partitioning, and block-wise masking range are retained, while its asymmetric KL objective is replaced by BCSM.

Figure 7: Qualitative comparison of cases where all methods fail to recover the exact reference. In these examples, KL-based methods produce fragmented tokens or repetitions, whereas BCSM retains longer recognizable spans and more complete local structures. The comparison reports an empirical property of the generated outputs rather than explicit cross-position dependency modeling.

Diffusion Distillation

Diffusion distillation compresses iterative generative models into faster samplers while aiming to preserve sample quality, diversity, and control.

See all 24 papers

Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD
Mar 20, 2026Emiel Hoogeboom, David Ruhe, Jonathan Heek +2Diffusion Model Distillation Diffusion Models
Google DeepMind Amsterdam
Distribution Matching Distillation for Continuous Diffusion Language Models
Sep 30, 2026Paul Le Van Kiem, Dario Shariatian, Umut Simsekli +1Continuous Diffusion Language Models Diffusion Language Model Distillation
Inria, PSL Research University · Cohere · CMAP, Ecole Polytechnique
Efficient Hybrid Distillation: Synergizing Score and Adversarial Objectives for One-Step Diffusion
Nov 2, 2026Fei Peng, Junqiang Wu, Haoxian Tan +5Diffusion Model Distillation Score Distillation Sampling
Tsinghua University · Peng Cheng Laboratory · Meituan Inc.

Mar 20, 2026J/K move · Enter open · S save

Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD

Emiel Hoogeboom, David Ruhe, Jonathan Heek, Thomas Mensink+1

Google DeepMind Amsterdam

Diffusion Model Distillation Diffusion Models Diffusion Language Model Distillation Discrete Diffusion Sampling

Open paper AskPDF arXiv

It is currently difficult to distill discrete diffusion models. In contrast, continuous diffusion literature has many distillation approaches methods that can reduce sampling steps to a handful. Our method, Discrete Moment Matching Distillation (D-MMD), leverages ideas that have been highly successful in the continuous domain. Whereas previous discrete distillation methods collapse, D-MMD maintains high quality and diversity (given sufficient sampling steps). This is demonstrated on both text and image datasets. Moreover, the newly distilled generators can outperform their teachers.

Abstract

Diffusion Distillation

Explore similar work

Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD

Distribution Matching Distillation for Continuous Diffusion Language Models

Efficient Hybrid Distillation: Synergizing Score and Adversarial Objectives for One-Step Diffusion