ForkLeft: Entropy-First Rollouts for Prefix-Aligned Autoregressive-to-Diffusion Distillation

Junming Liu; Jicheng Wang; Yifeng He; Hao Chen; Jianzhong Qi

cs.AISep 26, 2026

ForkLeft: Entropy-First Rollouts for Prefix-Aligned Autoregressive-to-Diffusion Distillation

Authors: Junming Liu, Jicheng Wang, Yifeng He, Hao Chen, Jianzhong Qi

Organizations: University of Melbourne · University of California, Davis · University of Hong Kong

Abstract

Autoregressive Next-Token Prediction (NTP) has enabled strong reasoning capabilities in language models, while Diffusion Language Models (DLMs) offer flexible token orders and parallel generation. We ask whether DLMs can acquire NTP-style reasoning through distillation without giving up their native generation process. Direct distillation, however, faces a fundamental mismatch: an autoregressive teacher predicts from a left prefix, whereas a DLM can condition on tokens on both sides. We introduce ForkLeft, a distillation framework that resolves this mismatch by separating the student's rollout from teacher supervision. During training, the student first performs entropy-first rollouts that commit uncertain positions and expose potential forks. We then fix the resulting student prefix and distill an NTP teacher under the same context, with answer correctness determining the supervision source. At inference, the student returns to its native confidence-first parallel decoding. With Qwen3-30B-A3B-Base, ForkLeft improves Efficient-DLM-4B on all ten benchmarks, raising MATH500 from 72.60% to 79.60% and consistently outperforming three alternative designs. The gains scale with teacher strength and generalize to SDAR-4B with only $500$ updates. At matched scale, the distilled 4B and 8B students exceed the published SDAR-Chat and OPDLM models on seven benchmarks, showing that DLMs can learn NTP-style reasoning without sacrificing native parallel generation. Code and datasets will be released upon acceptance.

Figures & tables

Panel of Figure 1 — Figure 1: A common target does not imply a common prediction problem. (a) To predict C , the teacher sees AB ; the student can also see its own $\hat{\texttt{D}},\hat{\texttt{E}}$ . Gray tokens are unavailable to the teacher; letters are schematic. (b) We teach with the same completed student prefix AB , hiding C and its suffix. Native generation remains bidirectional.

Panel of Figure 2 — Figure 2: ForkLeft exposes the fork, aligns the view, and keeps native inference. One update on the running example A–E: an entropy-first rollout, with the frozen teacher answering only for verification (A); two verdicts select one of four routes, with loss positions sampled uniformly (B); both models predict the target from the same student prefix (C); prefix sensitivity ranks confidences and decoding stays confidence-first (D). Dashed orange arrows carry fixed targets; teal arrows carry gradients.

Benchmark	Items	Base	ForkLeft	Gain (pp)
GSM8K	$1319$	82.71	85.22	+2.50
MATH500	$500$	72.60	79.60	+7.00
AIME24	$30$	10.00	16.67	+6.67
AIME25	$30$	6.67	13.33	+6.67
LMB-H	$45$	8.89	11.11	+2.22

Table 1: ForkLeft improves EDLM-4B on all ten benchmarks. The same student before and after

2000

updates with Qwen3-30B-A3B-Base; accuracies in percent, gains from unrounded counts; AR reference models in Table 5 .

Panel of Figure 3 — Figure 3: The student continues to improve as training proceeds. Accuracy gains over the EDLM-4B base at $100$ , $250$ , $500$ , $1000$ , and $2000$ updates, grouped by evaluation protocol on shared scales. Every trajectory is nondecreasing; AIME25 and LMB-H plateau from $1000$ to $2000$ updates.

Panel of Figure 4 — Figure 4: Entropy-first rollouts and prefix alignment reinforce each other. (a) Mean gain over the EDLM-4B base for the four order–view configurations, with the number of benchmarks that improve. (b) The same means as an interaction plot; faint lines are the prefix-view benchmarks.

Panel of Figure 5 — Figure 5: Teacher distributions add to hard targets on the same questions. All three objectives train on the teacher-correct, student-wrong questions of the main run; rows are ordered by benchmark size, right labels give the net change in correct answers, and grey marks changes of at most two items. (a) Adding OPD to SFT raises six benchmarks; GPQA-D is the only larger decrease. (b) Adding SFT to OPD lowers none.

Panel of Figure 6 — Figure 6: Gains rise with teacher strength for both student sizes. (a) Mean gain over each student’s own base for every trained pair, with the number of benchmarks that improve. (b) The same means against teacher strength; faint lines are the EDLM-4B benchmarks, the hollow marker is the Qwen2.5-7B teacher, and no curve is fitted.

Panel of Figure 7 — Figure 7: ForkLeft generalizes to SDAR-4B. Accuracy change per benchmark after $500$ updates with Qwen3-30B-A3B-Base (pp and net correct answers); grey: within two items.

Model	MATH500	AIME24	AIME25	MLogiQA	CEval	MMLU	MT-AIME
SDAR-Chat-4B	72.80	10.00	7.50	46.50	62.90	74.90	3.00
OPDLM-4B	72.80	14.40	12.60	46.50	66.90	65.50	5.30
EDLM-4B $+$ ForkLeft	79.60	16.67	13.33	53.13	73.70	76.05	15.82
SDAR-Chat-8B	78.60	10.00	10.00	46.30	70.20	78.60	4.00
OPDLM-8B	71.20	14.70	12.40	42.00	73.30	70.90	7.90
EDLM-8B $+$ ForkLeft	81.20	20.00	16.67	55.38	78.08	79.11	18.00

Table 2: Distilled students exceed published diffusion models of the same size on seven benchmarks. Accuracy (%); SDAR-Chat and OPDLM scores from Su et al. (2026) under our prompt template and scorer.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	$n$	Base	100	250	500	$1000$	$2000$
GSM8K	$1319$	$1091$	$1097$	$1105$	$1113$	$1120$	$1124$
MATH500	500	363	368	375	384	392	398
AIME24	30	3	3	4	4	4	5
AIME25	30	2	2	3	3	4	4
LMB-H	45	4	4	4	5	5	5
GPQA-D	198	61	61	62	62	63	64

Appendix

Table 3: Correct counts for the EDLM-4B base and every measured training checkpoint.

Benchmark	$n$	B-low	B-high	L-low	L-high	OPD	SFT	Both
GSM8K	$1319$	$1087$	$1094$	$1105$	$1124$	$1112$	$1118$	$1123$
MATH500	500	361	364	381	398	389	392	397
AIME24	30	2	3	4	5	4	5	4
AIME25	30	2	2	3	4	3	4	3
LMB-H	45	4	4	4	5	4	5	4
GPQA-D	198	60	61	62	64	63	66	63

Appendix

Table 4: Counts underlying the order–view and shared-subset comparisons. B/L: bidirectional/left view; low/high: training entropy priority. The three rightmost columns use the same selected subset.

Benchmark	$n$	Q3-30B	Q3-8B	Q3-4B	Q2.5-7B	EDLM-8B	Q3-8B-I	Math7B-I
GSM8K	$1319$	$1192$	$1123$	$1160$	$1085$	$1118$	$1194$	$1178$
MATH500	500	399	391	357	350	391	374	420
AIME24	30	11	2	4	5	4	3	3
AIME25	30	0	6	6	1	3	4	3
LMB-H	45	8	9	3	8	8	5	8
GPQA-D	198	71	64	74	64	67	66	62

Appendix

Table 5: Reference checkpoint counts. Q3-30B, Q3-8B and Q2.5-7B are Base checkpoints; Q3-8B-I denotes the separately reported Qwen3-8B, and Math7B-I denotes Qwen2.5-Math-7B-Instruct. EDLM-8B is the base. These are reference scores, not matched post-training baselines.

Benchmark	$n$	4/4	4/7	4/8	4/30	8/8	8/30
GSM8K	$1319$	$1109$	$1113$	$1118$	$1124$	$1140$	$1149$
MATH500	500	382	385	389	398	398	406
AIME24	30	4	4	4	5	5	6
AIME25	30	3	3	3	4	4	5
LMB-H	45	5	5	5	5	6	7
GPQA-D	198	62	62	63	64	68	70

Appendix

Table 6: Teacher–student extension counts. Columns give student/teacher sizes. Teachers 4, 8, and 30 denote Qwen3-4B, Qwen3-8B-Base, and Qwen3-30B-A3B-Base; 7 denotes Qwen2.5-7B-Base.

Benchmark	$n$	Base	500 updates	$\Delta$ correct
GSM8K	$1319$	$1095$	$1120$	+25
MATH500	500	359	364	+5
AIME24	30	4	2	-2
AIME25	30	2	2	+0
LMB-H	45	3	4	+1
GPQA-D	198	62	71	+9

Appendix

Table 7: SDAR-4B with Qwen3-30B-A3B-Base: base and 500-update correct counts.

	4B				8B
Benchmark	SDAR-Chat	OPDLM	EDLM	$+$ ForkLeft	SDAR-Chat	OPDLM	EDLM	$+$ ForkLeft
GSM8K	89.90	87.60	82.71	85.22	91.30	87.10	84.76	87.11
MATH500	72.80	72.80	72.60	79.60	78.60	71.20	78.20	81.20
AIME24	10.00	14.40	10.00	16.67	10.00	14.70	13.33	20.00
AIME25	7.50	12.60	6.67	13.33	10.00	12.40	10.00	16.67
LMB-H	6.90	11.10	8.89	11.11	8.90	20.00	17.78	15.56

Appendix

Table 8: All ten benchmarks for the published models and the distilled students. SDAR-Chat and OPDLM scores as reported by Su et al. (2026) ; EDLM is each student’s base and

+

ForkLeft the distilled student; bold marks the best of the three published or distilled systems per benchmark and scale.

Diffusion Language Model Distillation

Diffusion language model distillation transfers knowledge or decoding behavior to make diffusion-based text generation faster and more capable.

See all 37 papers

JustOPD: On-Policy Distillation from Autoregressive to Diffusion Language Models
Oct 12, 2026Zikun Zhang, Jiayuan Sheng, David D. Yao +1Diffusion Language Model Distillation On-Policy Distillation
Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027.
Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation
Jun 4, 2026Xingyu Su, Jacob Helwig, Shubham Parashar +6Language Model Distillation Diffusion Language Models
Department of Computer Science and Engineering, Texas A&M University · Department of Bioinformatics and Systems Medicine,2026 University of Texas Health Science Center at Houston · Department of Electrical and Computer Engineering, Texas A&M UniversityJun 4
Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding?
Feb 26, 2026Pengxiang Li, Dilxat Muhtar, Tianlong Chen +2Parallel Decoding for Diffusion Language Models Diffusion Language Model Decoding
The Hong Kong Polytechnic University, Hong Kong, China · ELLIS Institute Tübingen, Tübingen, Germany · Max Planck Institute for Intelligent Systems, Tübingen, Germany +3

Oct 12, 2026J/K move · Enter open · S save

JustOPD: On-Policy Distillation from Autoregressive to Diffusion Language Models

Zikun Zhang, Jiayuan Sheng, David D. Yao, Wenpin Tang

Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027.

Diffusion Language Model Distillation On-Policy Distillation Cross-Architecture Knowledge Distillation Diffusion Language Models

Open paper AskPDF arXiv

On-policy distillation (OPD) is a post-training approach for large language models (LLMs) that enables a student model to learn a teacher model's distribution from sequences sampled from the student's own policy. Despite its promising capability, OPD for diffusion language models (DLMs) remains underexplored and existing studies primarily focuses on DLM-to-DLM distillation. The limited availability of strong teacher DLMs and the flexible generation order of DLMs impose additional technical challenges as well. However, it remains unclear how to effectively distill knowledge from a strong autoregressive language model (ARM) into a weaker DLM, given the mismatch between the left-to-right decoding of ARMs and the flexible decoding order of DLMs. Inspired by Ni et al. (2026), we propose JustOPD, an ARM-to-DLM on-policy distillation method that uses autoregressive rollouts during training to align the student's generation process with the ARM teacher's left-to-right decoding. This alignment enables effective distillation from ARM teachers while preserving the DLM's flexible generation-order capability at inference time. Experiments using Qwen3-14B as the teacher and Dream-7B as the student demonstrate that JustOPD consistently improves mathematical reasoning performance over the base DLM across multiple divergence choices.

Abstract

Diffusion Language Model Distillation

Explore similar work

JustOPD: On-Policy Distillation from Autoregressive to Diffusion Language Models

Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation

Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding?