Turing LLM Post-Training: Calibrated Distillation and Efficient Dynamic-MoE Execution

Yue He; Jiahui Hu; Shaman Tang; Hua Zhou; Qiman Wu; Menglin Li; Yizhao Wang; Yuheng Zhang

cs.LGOct 29, 2026

Turing LLM Post-Training: Calibrated Distillation and Efficient Dynamic-MoE Execution

Authors: Yue He, Jiahui Hu, Shaman Tang, Hua Zhou, Qiman Wu, Menglin Li, Yizhao Wang, Yuheng Zhang,

Organizations: Foundation Model Team, XPeng Inc.

Abstract

We study post-training for dynamically routed mixture-of-experts (MoE) models, focusing on integrating independently optimized specialists while keeping consolidation efficient. Our approach combines calibrated multi-teacher on-policy distillation (C-MOPD), which matches each selected teacher on a local Top- $K$ support with teacher-wise loss normalization and clipping, with an execution path that uses post-capacity bounded-width dispatch, final-route replay, and asynchronous rollout--training overlap. On Turing-20B-A2B, the selected model achieves a four-domain normalized integration score of 0.9988 and retains at least 90.16% of the measured specialist gain in every domain. At matched $K=32$ , the full calibration objective scores 0.9962, compared with 0.9724 without calibration and 0.9834 without clipping. At a matched 8K response cap, the model exceeds both shared SFT and Qwen3-8B on all four reasoning-enabled mathematics and code benchmarks (unweighted mean: 85.92, versus 80.33 for Qwen3-8B and 74.21 for SFT), and improves over SFT on ten of twelve broader evaluation metrics. In a PPU-based profile, the combined execution configuration reduces consolidation step time from 398.7 to 139.3 seconds, an observed 2.86 $\times$ speedup over serial execution. These results demonstrate strong specialist integration alongside more efficient consolidation, while reflecting the complete training and execution framework rather than any single component.

Figures & tables

Figure 1: The Turing LLM post-training framework. Shared supervised initialization provides the starting point for domain-specialist RL. C-MOPD consolidates the frozen specialists into one student through teacher-local Top-

K

matching and loss-scale calibration on student-generated prefixes. Bounded-width dispatch, route replay following R3, and asynchronous overlap support efficient dynamic-MoE execution in the consolidation workflow.

Task	Think	Non-think	Verification and reward
Code	80%	20%	Extraction, signature validity, compilation, and unit-test pass rate; auxiliary terms are gated by executable validity.
Math	80%	20%	Final-answer correctness and normal completion, with penalties for truncation and repeated reasoning.
IF	20%	80%	Constraint satisfaction or option correctness, with a preference for concise valid outputs.
Agent	100%	0%	Tool-call format, argument correctness, task completion, and process feedback from a 20B process reward model.

Table 1: Prompt-form mixtures and reward components for specialist training. Think and Non-think denote reasoning-enabled and direct-response prompts; percentages give their sampling proportions within each domain.

Figure 2: Dynamic-MoE execution during consolidation. (a) RW32 bounds post-capacity dispatch width at 32, preserving non-overflow expert assignments and retaining the highest-scoring 32 assignments on overflow. (b) R3-style final-route replay reuses the executed expert assignments and recomputes routing weights under the current actor parameters. AsyncOverlap overlaps rollout with actor updates using a one-step policy lag. (c) Base-model Quantile Routing selects a variable number of experts per token, followed by per-expert capacity filtering.

Reference / model	Math	Code	IF	Agent	Macro
SFT reference	0.0000	0.0000	0.0000	0.0000	0.0000
Specialist references	1.0000	1.0000	1.0000	1.0000	1.0000
C-MOPD ( $K=16$ , $c_{\max}=5$ )	0.9407	1.0093	0.9016	1.1435	0.9988

Table 2: Normalized specialist-gain recovery. The SFT and specialist rows define the normalization references; Macro is the unweighted mean of the four domain ratios.

Metric	SFT	Specialist	C-MOPD	$\Delta$ vs. SFT	$\Delta$ vs. specialist
GSM8K	90.07	91.96	91.58	+1.51	$-0.38$
MATH	65.86	74.50	74.10	+8.24	$-0.40$
MBPP	69.31	72.75	73.28	+3.97	+0.53
HumanEval	79.88	86.59	84.15	+4.27	$-2.44$
IFEval	58.41	80.96	78.74	+20.33	$-2.22$
BFCL v4 (Think)	20.54	32.04	33.69	+13.15	+1.65

Table 3: Stage-wise scores on the benchmarks directly associated with each specialist. All rows use direct-response evaluation except BFCL v4, which uses reasoning-enabled evaluation.

Objective	Variant	Math	Code	IF	Agent	Macro
Sampled-token MOPD		0.7747	0.7830	0.9836	1.1417	0.9208
Top- $K$ ( $K=32$ )	No calibration	0.8938	0.9142	0.9756	1.1061	0.9724
	No clipping	0.8736	0.9356	0.9592	1.1652	0.9834
	Full C-MOPD	0.8298	0.9514	0.9836	1.2200	0.9962

Table 4: Calibration ablations at

K=32

, with sampled-token MOPD as a separate method reference. No calibration removes normalization and clipping; No clipping retains normalization.

Sweep	Value	Math	Code	IF	Agent	Macro
Support size	$K=8$	0.7846	0.9661	0.9264	1.1870	0.9660
	$K=16$	0.9407	1.0093	0.9016	1.1435	0.9988
	$K=32$	0.8298	0.9514	0.9836	1.2200	0.9962
	$K=64$	0.8250	0.8226	0.9016	1.1809	0.9325
Clipping threshold	$c_{\max}=2$	0.7949	0.9403	0.8772	1.0600	0.9181
	$c_{\max}=5$	0.8298	0.9514	0.9836	1.2200	0.9962

Table 5: Normalized integration scores for two sweeps: support size

K

with

c_{\max}=5

fixed, and clipping threshold

c_{\max}

with

K=32

fixed.

Metric	Qwen3-8B	Turing SFT	C-MOPD	$\Delta$ vs. SFT
General Knowledge and Reasoning Tasks
MMLU	74.23	76.59	78.10	+1.51
CMMLU	76.81	83.49	83.48	$-0.01$
C-Eval	67.74	80.48	81.07	+0.59
MMLU-Pro	25.71	45.75	48.53	+2.78
BBH	80.47	72.10	74.25	+2.15

Table 6: Broader benchmark results. The first eleven rows use direct-response evaluation; BFCL v4 uses reasoning-enabled evaluation.

\Delta

denotes the change from Turing SFT.

Metric	Qwen3-8B	Turing SFT	C-MOPD	$\Delta$ vs. SFT	$\Delta$ vs. Qwen3
GSM8K	92.72	92.95	95.68	+2.73	+2.96
MATH-500	80.80	73.00	82.40	+9.40	+1.60
MBPP	62.43	55.29	77.78	+22.49	+15.35
HumanEval	85.37	75.61	87.80	+12.19	+2.43

Table 7: Reasoning-enabled mathematics and code evaluation with an 8K response cap for all models.

	Cap hits (%)			Accuracy (%)
Metric	C-MOPD 8K	Qwen3 8K	Qwen3 32K	Qwen3 32K
GSM8K	0.08	1.74	0.30	92.95
MATH-500	3.60	19.00	2.60	95.00
HumanEval	0.00	12.20	10.37	86.59
MBPP	0.75	14.79	9.52	64.02

Table 8: Output-cap hit rates under reasoning-enabled prompts and Qwen3-8B accuracy at 32K. A cap hit is a response that reaches the configured token limit. Corresponding 8K accuracies are reported in Table 7 . Bold marks the lowest cap-hit rate for each task.

Metric	Consolidate	AsyncOverlap	AsyncOverlapRW32
Step time (s)	398.7	330.1	139.3
Trainer wait (s)	265.0	195.8	18.3
Trainer compute (s)	133.7	134.3	121.0
Rollout time (s)	242.9	312.1	93.9
Rollout throughput (tok/GPU/s)	183.9	143.1	478.4
Actor throughput (tok/s)	26,966	26,451	31,165

Table 9: Average consolidation-system measurements over the common profiling interval.

Figure 3: Consolidation-system profile. (a) Mean step time decomposed into trainer compute and waiting. (b) Rollout throughput in tokens per GPU per second. Summary values compare AsyncOverlapRW32 with serial Consolidate. Rollout generation may overlap trainer phases.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Steps	Batch / step	Training pool	Epochs
SFT	10,000	512 packed rows	5.27M samples	$\sim$ 1.0
Code RL	240	128 prompts $\times$ 16 rollouts	16K prompts	$\sim$ 2.0
IF RL	480	128 prompts $\times$ 16 rollouts	16.2K prompts	$\sim$ 4.0
Math RL	550	128 prompts $\times$ 16 rollouts	35.5K prompts	$\sim$ 2.0
Agent RL	512	64 prompts $\times$ 16 rollouts	8.5K prompts	$\sim$ 4.0
C-MOPD	400	128 prompts $\times$ 1 rollout	51.5K prompts	$\sim$ 1.0

Appendix

Table 10: Post-training stages and data budgets. Pool sizes are rounded. RL and C-MOPD batch sizes distinguish sampled prompts from rollouts per prompt; epochs denote approximate passes through the corresponding data pool.

Data category	Examples	Share
Mathematics and code distillation	2.69M	51.0%
Agent tool use	1.51M	28.7%
General and instruction following (including ICL)	0.67M	12.7%
Long context and dialogue	0.40M	7.6%
Total	5.27M	100%

Appendix

Table 11: Composition of the SFT mixture. Counts are rounded to two decimal places in millions; shares use unrounded counts.

Figure 4: Specialist RL diagnostics over the displayed training intervals. Normalized verifier reward tracks task-specific rollout quality; the effective-prompt count records groups with non-degenerate relative-advantage signals; response length is measured in generated tokens; KL is measured against each specialist’s RL reference policy rather than the C-MOPD student. Light lines show per-step values and dark lines show TensorBoard EMA curves ( $s=0.9$ ).

Benchmark	Split	Prompting	Metric
MMLU / CMMLU	Test	5-shot	Acc.
C-Eval	Validation	5-shot	Acc.
MMLU-Pro	Test	5-shot CoT	Acc.
GPQA Diamond	Diamond	5-shot CoT	Acc.
BBH	All 27 tasks	3-shot CoT	EM
GSM8K	Test	4-shot CoT	EM

Appendix

Table 12: Benchmark splits, prompting settings, and scoring metrics used in the reported evaluations.

Metric	Qwen3-8B Base	Turing-20B-A2B Pretrain
MMLU	78.92	79.83
CMMLU	81.28	84.17
C-Eval	83.06	81.76
MMLU-Pro	55.35	55.17
BBH	73.67	73.61
GPQA Main	34.82	37.05

Appendix

Table 13: Base-model results under the base-model prompting setup. GPQA uses the Main subset.

	References		Sampled	Objective controls at $K=32$			Selected
Metric	SFT	Teacher	MOPD	No calibration	No clipping	Full	$K=16$ Full
Mathematics Tasks
GSM8K	90.07	91.96	90.45	91.81	91.13	91.66	91.58
MATH	65.86	74.50	70.32	73.28	73.14	73.34	74.10
GSM8K (Think)	92.95	95.68	95.53	95.38	95.53	95.38	95.68
MATH-500 (Think)	73.00	83.00	83.60	82.20	82.40	80.80	82.40

Appendix

Table 14: Raw benchmark scores (%) for the reference checkpoints, sampled-token MOPD, three calibration variants at

K=32

, and the selected C-MOPD model. Teacher denotes the corresponding domain specialist. Bold marks the highest score in each row.

	Support size, $c_{\max}=5$				Clipping, $K=32$
Metric	$K=8$	$K=16$	$K=32$	$K=64$	$c_{\max}=2$	$c_{\max}=5$	$c_{\max}=10$
Mathematics Tasks
GSM8K	90.75	91.58	91.66	91.21	90.30	91.66	90.90
MATH	73.90	74.10	73.34	73.22	73.22	73.34	73.30
GSM8K (Think)	95.68	95.68	95.38	94.84	95.45	95.38	95.30
MATH-500 (Think)	79.80	82.40	80.80	81.80	81.40	80.80	80.60

Appendix

Table 15: Raw benchmark scores (%) for the support-size and clipping-threshold sweeps. Bold marks the highest score within each row and sweep block.

Configuration	Step time (s)	Wait (%)	Speedup	HumanEval	MBPP	MBPP+
Consolidate	381.4	65.4	1.0 $\times$	86.59	78.04	66.67
AsyncOverlap	335.3	59.8	1.1 $\times$	84.76	76.46	65.34
AsyncOverlapRW32	137.2	12.4	2.8 $\times$	87.20	76.19	65.87

Appendix

Table 16: Timing and code benchmark scores for the PPU execution configurations.

LLM Post-Training

Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.

See all 249 papers

Post-Trained MoE Can Skip Half Experts via Self-Distillation
May 18, 2026Xingtai Lv, Li Sheng, Kaiyan Zhang +12Mixture of Experts Communication-Efficient Mixture-of-Experts
Tsinghua University · Shanghai AI Lab · WeChat AI +1
UMoE:Unlocking Every Expert in Domain-Specific Training
Jul 13, 2026Xuefeng Li, Pengfei LiuMixture-of-Experts Pruning Mixture-of-Experts Language Models
1SII · 2SJTU · 3GAIR
Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models
Oct 19, 2026Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin +7Mixture-of-Experts Pruning Mixture-of-Experts Language Models
LinkedIn · MIT · Google

May 18, 2026J/K move · Enter open · S save

Post-Trained MoE Can Skip Half Experts via Self-Distillation

Xingtai Lv, Li Sheng, Kaiyan Zhang, Yichen You+11

Tsinghua University · Shanghai AI Lab · WeChat AI+1

Mixture of Experts Communication-Efficient Mixture-of-Experts Mixture-of-Experts Inference LLM Inference Acceleration Knowledge Distillation

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) scales language models efficiently through sparse expert activation, and its dynamic variant further reduces computation by adjusting the activated experts in an input-dependent manner. Existing dynamic MoE methods usually rely on pre-training from scratch or task-specific adaptation, leaving the practical conversion of fully trained MoE underexplored. Enabling such adaptation would directly alleviate the inference costs by allowing easy tokens to bypass unnecessary expert during serving. This paper introduces Zero-Expert Self-Distillation Adaptation (ZEDA), a low-cost framework that transforms post-trained static MoE models into efficient dynamic ones. To stabilize this architectural conversion, ZEDA injects parameter-free zero-output experts into each MoE layer and adapts the augmented model through two-stage self-distillation, utilizing the original MoE as a frozen teacher and applying a group-level balancing loss. On Qwen3-30B-A3B and GLM-4.7-Flash across 11 benchmarks spanning math, code, and instruction following, ZEDA eliminates over 50% of expert FLOPs at marginal accuracy loss. It outperforms the strongest dynamic MoE baseline by 6.1 and 4.0 points on the two models, and delivers ~1.20

\times

end-to-end inference speedup.

Abstract

LLM Post-Training

Explore similar work

Post-Trained MoE Can Skip Half Experts via Self-Distillation

UMoE:Unlocking Every Expert in Domain-Specific Training

Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models