Updating Parametric Knowledge with Context Distillation Retains Post-Training Capabilities

Shankar Padmanabhan; Mustafa Omer Gul; Tanya Goyal

cs.CLFeb 17, 2026

Updating Parametric Knowledge with Context Distillation Retains Post-Training Capabilities

Authors: Shankar Padmanabhan, Mustafa Omer Gul, Tanya Goyal

Organizations: Department of Computer Science, Cornell University.

Abstract

Post-training endows pretrained LLMs with a variety of desirable skills, including instruction-following, reasoning, and others. However, these post-trained LLMs only encode knowledge up to a cut-off date, necessitating continual adaptation. Unfortunately, existing solutions cannot simultaneously learn new knowledge from an adaptation document corpora and mitigate the forgetting of earlier learned capabilities. To address this, we introduce Distillation via Split Contexts (DiSC), a simple context-distillation based approach for continual knowledge adaptation. \methodname~derives student and teacher distributions by conditioning on distinct segments of the training example and minimizes the KL divergence between the shared tokens. This allows us to efficiently apply context-distillation without requiring explicit generation steps during training. We run experiments on four post-trained models and two adaptation domains. Compared to prior finetuning and distillation methods for continual adaptation, DiSC consistently reports the best trade-off between learning new knowledge and mitigating forgetting of previously learned skills like instruction-following, reasoning, and factual knowledge.

Figures & tables

Figure 1: Distillation via Split Contexts. First, we sample multiple split points and extract corresponding prefixes and suffixes. Then, we minimize the KL over the suffix distributions with context conditioning (teacher model $M_{T}$ ) and without (student model $M_{S}$ ). Our training only updates the student model $M_{S}$ .

( Padmanabhan et al., 2023 )	$d\sim\mathcal{D}$	$y\sim P_{M_{T}}(\cdot\|d)$
Internalization Target	Context	Loss Tokens
Constitution	Static context $c$	$d\in\mathcal{D}$
( Askell et al., 2021 )	for all $d\in\mathcal{D}$	$d\in\mathcal{D}$
Instructions + Reasoning	In-context exs &	$y\sim P_{M_{T}}(\cdot\|d,c)$
( Snell et al., 2022 )	$c\sim P_{M_{T}}(\cdot\|d)$	$y\sim P_{M_{T}}(\cdot\|d,c)$
New Knowledge	$d\sim\mathcal{D}$	$y\sim P_{M_{T}}(\cdot\|d)$

Table 1: Representative prior works in context distillation.

Qwen-2.5-7B-Instruct
	KUP								BioASQ
Method	BBH	GPQA	MMLU-P	MuSR	IFEval	Math	Code	KUP	BBH	GPQA	MMLU-P	MuSR	IFEval	Math	Code	BioASQ
$M_{\text{post}}$	53.69	30.29	42.89	39.81	80.70	49.85	78.66	15.46	53.69	30.29	42.89	39.81	80.70	49.85	78.66	70.19
FT	52.25	28.44	38.33	40.74	65.83	24.47	69.51	40.90	54.07	30.87	37.86	43.25	75.18	35.05	73.78	77.40
+Rephrase	51.54	28.86	37.37	43.92	63.91	28.25	71.34	42.90	52.47	29.03	37.38	39.68	71.22	32.85	71.95	76.44
+TALR	52.70	28.86	38.90	44.18	65.83	26.51	75.61	36.90	54.05	29.28	41.30	39.15	77.70	46.98	81.1	78.37

Table 2: Performance of finetuning baselines and its variants on two adaptation tasks: KUP and BioASQ. We find that standard finetuning generally improves adaptation task performance, but at the cost of substantial degradation of

M_{\text{post}}

’s IFEval, Math and Code skills.

Qwen-2.5-7B-Instruct
	KUP								BioASQ
Method	BBH	GPQA	MMLU-P	MuSR	IFEval	Math	Code	KUP	BBH	GPQA	MMLU-P	MuSR	IFEval	Math	Code	BioASQ
$M_{\text{post}}$	53.69	30.29	42.89	39.81	80.70	49.85	78.66	15.46	53.69	30.29	42.89	39.81	80.70	49.85	78.66	70.19
FT (from § 5.1 )	52.25	28.44	38.33	40.74	65.83	24.47	69.51	40.90	54.07	30.87	37.86	43.25	75.18	35.05	73.78	77.40
FT ${}^{\text{CP}}$	53.58	31.80	42.01	41.14	76.74	45.24	81.71	36.3	52.66	29.78	42.67	39.02	80.46	50.53	80.49	73.08
CD base ${}^{\text{CP}}$	54.42	30.62	43.02	40.08	79.38	49.24	82.32	32.10	54.89	30.45	43.41	39.95	81.65	46.83	81.10	76.92

Table 3: Comparing DiSC to standard fine-tuning and CD base following adaptation on KUP and BioASQ. The superscript

{}^{\text{CP}}

indicates model checkpoints achieving high domain adaptation while experiencing limited forgetting. Across datasets, DiSC consistently reports a stronger trade-off between these two goals.

Figure 2: IFEval vs KUP performances at different learning rates on Qwen-2.5-7B-Instruct for standard finetuning and DiSC.

Model	BBH	GPQA	MMLU-P	MuSR	IFEval	Math	Code
Qwen-2.5-7B-Instruct
$M_{\text{base}}$	49.47	29.11	42.08	38.10	57.19	27.42	56.10
$M_{\text{post}}$	53.69	30.29	42.89	39.81	80.70	49.85	78.66
LLaMA3.1-8B-Instruct
$M_{\text{base}}$	46.02	31.54	32.39	37.17	15.59	5.06	43.29
$M_{\text{post}}$	50.63	29.70	37.68	38.76	85.25	20.92	70.12

Table 4: Comparing

M_{\text{base}}

and

M_{\text{post}}

models across capabilities. Post-training primarily improves on IFEval, Math and Code.

Model	Domain	IFEval	MATH
		Pearson	Pearson
LLaMA3.1-8B-Instruct	KUP	$+0.22$	$+0.54$
LLaMA3.1-8B-Instruct	BioASQ	$-0.54$	$+0.39$
Qwen-2.5-7B-Instruct	KUP	$+0.51$	$+0.50$
Qwen-2.5-7B-Instruct	BioASQ	$-0.55$	$-0.40$

Table 5: Per-dataset correlation between KL divergence (measured on the train domain) and forgetting magnitude on held-out tasks.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	Qwen2.5-7B-Instruct								LLaMA3.1-8B-Instruct
Method	BBH	GPQA	MMLU-P	MuSR	IFEval	Math	Code	KUP	BBH	GPQA	MMLU-P	MuSR	IFEval	Math	Code	KUP
$M_{\text{post}}$	53.69	30.29	42.89	39.81	80.70	49.85	78.66	15.46	50.63	29.70	37.68	38.76	85.25	20.92	70.12	20.10
FT (from § 5.1 )	52.25	28.44	38.33	40.74	65.83	24.47	69.51	40.9	49.02	28.36	34.60	39.42	68.11	14.73	61.59	35.10
DiSC ${}^{\text{Max KUP}}$	53.17	28.36	41.45	39.68	74.46	44.03	77.44	47.59	50.96	29.61	35.46	40.74	75.06	17.90	64.63	49.50

Appendix

Table 6: Comparing maximum in-domain performance of DiSC with that of standard fine-tuning

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 231 papers

Surgical Post-Training: Proximal On-Policy Distillation for Reasoning with Knowledge Retention
Mar 2, 2026Wenye Lin, Kai HanLLM Post-Training LLM Reasoning
The University of Hong Kong
Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall
Sep 1, 2026Jacqueline He, Howard Yen, Shuyue Stella Li +9Language Model Distillation Factual Knowledge in Language Models
Meta AI · University of Washington · Princeton University
Combating Catastrophic Forgetting in Continual Domain Adaptation via Knowledge Recasting
Oct 20, 2026Zijie Liu, Zhen Tan, Charles Fleming +1Continual Learning for LLMs Domain Adaptation
UNC at Chapel Hill · Stevens Institute of Technology · Cisco Research

Mar 2, 2026J/K move · Enter open · S save

Surgical Post-Training: Proximal On-Policy Distillation for Reasoning with Knowledge Retention

Wenye Lin, Kai Han

The University of Hong Kong

LLM Post-Training LLM Reasoning On-Policy Distillation Knowledge Distillation Continual Learning for LLMs

Open paper AskPDF arXiv

Injecting new reasoning knowledge into Large Language Models (LLMs) via post-training often induces catastrophic forgetting. Recent studies emphasize the importance of on-policy data but suggest that KL-divergence fails to mitigate forgetting. In contrast, we show, both analytically and empirically, that the KL-constrained reward formulation actually plays a critical role in retaining knowledge during post-training. This motivates our Surgical Post-Training (SPOT), a proximal on-policy distillation framework designed to optimize reasoning efficiently while preserving prior knowledge. SPOT consists of (1) a data rectification pipeline employing an Oracle to surgically correct erroneous steps via minimal edits, generating proximal on-policy data; and (2) a reward-based binary cross-entropy objective essential for enhancing reasoning and mitigating forgetting. Empirically, with only 4k rectified math pairs, SPOT improves Qwen3-8B's accuracy by 6.2% on average across in-domain and out-of-domain tasks, requiring merely 16-minute model training on 8x H800 GPUs. Moreover, SPOT provides a superior initialization for subsequent reinforcement learning, significantly elevating the performance ceiling. Code: https://github.com/Visual-AI/SPoT

Abstract

Continual Learning for LLMs

Explore similar work

Surgical Post-Training: Proximal On-Policy Distillation for Reasoning with Knowledge Retention

Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

Combating Catastrophic Forgetting in Continual Domain Adaptation via Knowledge Recasting