CataOPD: Catalytic On-Policy Distillation for Large Language Model Reasoning

Wenjin Liu; Chenxi Wang; Jiapu Wang; Zhe Cui; Anh Tuan Luu; Haoran Luo

cs.LGAug 25, 2026

CataOPD: Catalytic On-Policy Distillation for Large Language Model Reasoning

Authors: Wenjin Liu, Chenxi Wang, Jiapu Wang, Zhe Cui, Anh Tuan Luu, Haoran Luo

Organizations: Nanyang Technological University, College of Computing and Data Science, Singapore · Hithink Research, Code Generation Department, Hangzhou, China · Nanjing University of Science and Technology, School of Computer Science and Technology, Nanjing, China

Abstract

Reinforcement learning (RL) and on-policy distillation (OPD) are two representative paradigms for improving large language model reasoning. However, when no correct trajectory is sampled, RL lacks a positive correctness signal, while OPD remains constrained by the reasoning trajectories reachable under the student's on-policy distribution. Therefore, we propose CataOPD, where the teacher acts as a catalyst rather than a target, expanding reachability while internalizing verified student-produced trajectories into a catalyst-free policy. Self-Rescue Routing uses empirically all-failed groups as routing signals rather than teacher-intervention triggers, first seeking correct trajectories through additional on-policy self-sampling. For problems unresolved after self-rescue, Catalytic-Guided Self-Resolution uses catalytic guidance to elicit a verified student-produced trajectory in the guided student distribution. Barrier-Weighted Internalization weights tokens by guided-to-unguided log-probability gaps, focusing updates on decisive tokens difficult without guidance. Experimental results show that CataOPD outperforms current baselines, extends independent student reasoning to still-unrecovered problems, and improves out-of-distribution generalization under catalyst-free inference. Our project is available at https://github.com/QwenQKing/CataOPD.

Figures & tables

Figure 1: Comparison of CataOPD and baselines.

Figure 2: Comparison results of CataOPD and the base LLM, SFT, RL, and OPD baselines across twelve benchmarks.

Figure 3: Overview of CataOPD, a catalytic OPD framework where the teacher acts as a catalyst rather than a target.

Method	MetaMath	GSM+	BigMath	OmniMath	Numina	MATH	DeepMath	OpenMath	Average
GPT-5.4-mini	89.84	88.28	78.12	42.19	69.53	85.16	43.75	41.41	67.29
Qwen2.5-3B
Base LLM	70.31	75.00	59.38	34.38	59.38	64.06	52.34	39.06	56.74
SFT	64.84	32.81	55.47	24.22	50.00	53.91	57.03	35.94	46.78
CoT-SFT	84.38	86.72	71.09	29.69	71.88	74.22	58.59	29.69	63.28
GRPO	87.50	86.72	73.44	31.25	72.66	75.00	61.72	39.06	65.92

Table 1: Main results of CataOPD and baselines on eight benchmarks. Accuracy are in %. Best values are bold.

Panel of Figure 4 — Figure 4: Comparison of CataOPD and baselines on four OOD benchmarks. Panels (a)–(d) show per-benchmark accuracy; orange dashed lines mark GPT-5.4-mini, and blue markers/lines mark CataOPD. Panels (e)–(f) show OOD averages and margins over the strongest baseline, while (g) shows CataOPD’s accuracy across benchmarks.

Figure 5: Hard-problem analysis of CataOPD for Qwen2.5-3B and Qwen2.5-7B across all twelve evaluated benchmarks: (a) accuracy gains with 95% confidence intervals; (b) Pass@

8

transitions between questions solved and unsolved by the base model.

Method	MetaMath	GSM+	BigMath	OmniMath	Numina	MATH	DeepMath	OpenMath
w/o Self-Rescue	62.50	37.50	53.12	16.41	44.53	49.22	45.31	25.00
w/o Guidance	87.50	86.72	71.88	32.81	72.66	75.78	64.84	40.62
w/o Barrier	81.25	85.94	72.66	30.47	67.97	73.44	63.28	37.50
[0pt][0pt] CataOPD	88.28	87.50	74.22	35.94	73.44	76.56	65.62	41.41

Table 2: Component ablations for CataOPD with Qwen2.5-3B across eight benchmarks. The w/o variants remove Self-Rescue Routing, Catalytic-Guided Self-Resolution, and Barrier-Weighted Internalization, respectively.

Panel of Figure 6 — Figure 6: Training-time verified-trajectory coverage before and after recovery; annotations show final-step gaps.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Prompt templates used in CataOPD. The student template specifies the required reasoning and final-answer format, together with the training-time guidance format. The teacher-side template constrains catalytic guidance to provide a solution method while withholding the reference answer and answer-copyable information.

Method	Backbone	Teacher	Alg.	LR	BS	G	m	H	Int.	Epochs
Base LLM	All	–	–	–	–	–	–	–	–	–
GPT-5.4-mini	–	–	API	–	–	–	–	–	–	–
SFT	All	–	SFT	$10^{-4}$	32	–	–	–	–	3
GRPO	All	–	GRPO	$10^{-6}$	128	5	–	–	–	1
CoT-SFT	All	GPT-5.4-mini	SFT	$10^{-4}$	32	–	–	–	–	3
OPSD	All	Frozen self	Token JSD	$5\!\times\!10^{-6}$	128	1	–	–	–	1

Appendix

Table 3: Hyperparameter settings for baselines and CataOPD. All denotes Qwen2.5-3B, Qwen2.5-7B, and Qwen3-1.7B. Here Alg. denotes the algorithm, LR denotes the learning rate, BS denotes the effective batch size, G denotes trajectories per question, m denotes the self-rescue sampling budget, H denotes the maximum catalytic rounds, and Int. reports

\lambda/\beta/b_{\max}

. BWI denotes Barrier-Weighted Internalization. CataOPD uses

\epsilon=0.2

\kappa=3.0

, and

c_{\mathrm{KL}}=0.01

Backbone	All-failed (%)	Self-rescue (%)	Teacher success (%)	Total recovery (%)
Qwen3-1.7B	30.00 [28.96, 31.05]	16.54 [15.06, 18.13]	8.85 [7.63, 10.11]	23.93 [22.21, 25.64]
Qwen2.5-3B	52.19 [50.65, 53.68]	17.87 [16.79, 18.92]	22.03 [20.73, 23.31]	35.97 [34.66, 37.26]
Qwen2.5-7B	43.34 [42.06, 44.63]	19.02 [18.00, 20.06]	21.48 [19.94, 23.04]	36.41 [34.87, 37.91]
Backbone	Teacher attempts	Teacher calls	Input tokens	Output tokens
Qwen3-1.7B	2,564	12,206	73.13M	5.64M
Qwen2.5-3B	4,389	19,461	31.21M	8.56M

Appendix

Table 4: Training-time routing and teacher budget for the complete CataOPD runs. Self-rescue is conditional on initially all-failed groups, and teacher success is conditional on teacher-attempted groups; total recovery (Rec.) is the fraction of initially all-failed groups recovered by either route. Confidence intervals are 95% step-cluster bootstrap intervals. Token counts aggregate teacher-side prompt and completion tokens across training and exclude all student rollout tokens from the reported budget.

Trajectory likelihood diagnostics
Diagnostic	Mean	95% CI	Positive fraction (%)
Guided minus unaided likelihood gain	0.280	[0.228, 0.338]	99.0
Matched minus shuffled guidance gain	0.352	[0.281, 0.431]	100.0
Shuffled minus unaided likelihood gain	-0.072	[-0.111, -0.039]	22.5
Unaided internalization gain (Full $-$ base)	0.115	[0.079, 0.155]	71.5
Full $-$ no guidance	0.447	[0.299, 0.612]	80.0

Appendix

Table 5: Numerical mechanism diagnostics for 200 Qwen2.5-3B trajectories. Likelihood quantities are equal-weighted token-mean log-probability differences. Positive fraction is the percentage of trajectories with a positive diagnostic value. Barrier shares use uniform-mass references of 0.10, 0.20, and 0.30. Confidence intervals are 95% intervals under the fixed paired-trajectory bootstrap protocol with 10,000 trajectory-level resamples.

Figure 9: Staged recovery of initially all-failed training groups in the complete CataOPD runs. (a) partitions each group’s final disposition into self-rescued, teacher-recovered, or unrecovered. (b) reports cumulative recovery after self-rescue and successive teacher-guided rounds R1–R5. (c) separates the incremental recovery contributed by each stage. All percentages are conditional on initially all-failed groups and summarize training-time routing outcomes, rather than test-time accuracy.

Figure 10: Matched Qwen2.5-3B ablation over all 80 training steps. Compared with w/o Self-Rescue, Full CataOPD achieves higher all-failed-group recovery while using fewer teacher calls and teacher input/output tokens per verified target.

Figure 11: Matched Qwen2.5-3B ablation across non-overlapping 10-step windows. Full CataOPD exceeds w/o Self-Rescue in all-failed-group recovery and requires fewer teacher calls and teacher input tokens per verified target in every displayed window.

Figure 12: Case study on generation quality for a mathematical reasoning problem, comparing CataOPD with seven baseline methods.

Figure 13: Case study of training-time catalytic recovery after five failed initial rollouts. After unsuccessful unguided self-rescue, two rounds of catalytic guidance yield a verified student solution.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 511 papers

Beyond Imitation: Reflective On-Policy Self-Distillation for LLM Reasoning
May 27, 2026Ziqi Zhao, Xinyu Ma, Liu Yang +6On-Policy Self-Distillation Language Model Distillation
The Hong Kong Polytechnic University · Baidu Inc. · Shandong University +1
Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance
May 29, 2026Yuxuan Jiang, Francis FerraroCoT Distillation Language Model Distillation
University of Maryland, Baltimore County
OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning
May 12, 2026Yuxiao Yang, Xiaoyun Wang, Weitong ZhangLanguage Model Calibration Language Model Distillation
UNC Chapel Hill · NVIDIA

May 27, 2026J/K move · Enter open · S save

Beyond Imitation: Reflective On-Policy Self-Distillation for LLM Reasoning

Ziqi Zhao, Xinyu Ma, Liu Yang, Yujie Feng+5

The Hong Kong Polytechnic University · Baidu Inc. · Shandong University+1

On-Policy Self-Distillation Language Model Distillation Mode Collapse LLM Reasoning

Open paper AskPDF arXiv

On-policy self-distillation (OPSD) improves the reasoning capabilities of large language models (LLMs) by providing dense token-level supervision for on-policy rollouts. However, existing OPSD methods often yield limited gains on complex reasoning tasks and suffer from severe training instability. We identify two key causes: conditioning the self-teacher on a complete verified solution encourages imitation of complete reference trajectories rather than extraction of transferable reasoning insights, while indiscriminate full-response distillation imposes superfluous supervision on already-valid reasoning prefixes. Together, these issues suppress reasoning diversity and contribute to late-stage mode collapse. We propose Reflective On-policy Self-Distillation (ROSD), which distills transferable reasoning insights rather than complete reference trajectories. For each erroneous rollout, a self-reflector contrasts it with a correct rollout from the same group to derive a corrective idea and identify the sentence containing the first reasoning error. The corrective idea provides the self-teacher with targeted guidance, while the diagnosed error boundary allows ROSD to mask out the distillation loss over the valid prefix and apply token-level distillation only from the first erroneous sentence onward. Experiments across multiple reasoning benchmarks and model backbones show that ROSD consistently outperforms standard OPSD and reinforcement learning baselines, better preserves reasoning diversity, stabilizes training, and mitigates late-stage mode collapse. Code is available at https://github.com/ZiqiZhao1/ROSD.

Abstract

On-Policy Distillation

Explore similar work

Beyond Imitation: Reflective On-Policy Self-Distillation for LLM Reasoning

Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance

OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning