Combating Catastrophic Forgetting in Continual Domain Adaptation via Knowledge Recasting

Zijie Liu; Zhen Tan; Charles Fleming; Tianlong Chen

cs.LGOct 20, 2026

Combating Catastrophic Forgetting in Continual Domain Adaptation via Knowledge Recasting

Authors: Zijie Liu, Zhen Tan, Charles Fleming, Tianlong Chen

Organizations: UNC at Chapel Hill · Stevens Institute of Technology · Cisco Research

Abstract

Adapting language models to new domains during post-training risks degrading previously acquired capabilities, a phenomenon commonly known as catastrophic forgetting. Existing mitigation methods largely treat forgetting as a consequence of excessive model drift. They therefore constrain updates by preserving old outputs, replaying old data, or regularizing changes to parameters and representations so that the updated model remains close to its previous state. We take a different view: Rather than pulling the updated model back toward its old state, we translate the new domain into the model's pre-existing knowledge space, making unfamiliar knowledge learnable through familiar procedures. Therefore, we propose Skill Schema Transport (SST), a post-training framework that transports new knowledge into the model's existing skill space. SST abstracts each example type into a skill schema, preserving its essential operations, dependencies, and verification steps while removing domain-specific surface forms. By recasting new examples as instances of reusable procedures, SST turns knowledge update from trajectory memorization into procedural reuse. In continual domain adaptation settings, SST consistently reduces forgetting while improving generalization to related tasks. Across two 3B instruction-tuned backbones, SST improves average final performance on streamed domains by 2.53 to 5.75 points over sequential post-training. It also lowers max-drop forgetting in all evaluated settings, with the largest reduction from 42.41 to 7.10 points on the Qwen-3B MATH stream, while substantially recovering the OOD degradation induced by sequential updates.

Figures & tables

Figure 1: Overview of Skill Schema Transport (SST). SST recasts new-domain examples as instances of reusable skill schemas rather than anchoring old data, output distributions, or parameter values. The left panels illustrate the drift diagnostics used in our analysis, and the bottom pipeline shows how procedural skills are extracted, abstracted, stored, and used for schema-conditioned updates.

Figure 2: Surface drift and representation drift need not align. Domain-shifted updates induce different distributions of output-level and hidden-state change, motivating diagnostics that track both observable behavior and internal representations.

Figure 3: Errors cluster by procedure rather than by domain label. The same failed examples are projected and colored by procedural error type on the left and by MATH category on the right. Error clusters cut across category boundaries, motivating cross-domain skill schema reuse.

Field	Description
Skill Name	Algebraic Substitution and Equation Solving
Goal	Solve algebra problems via rewriting, substitution, and branch verification.
Applicability	Equations, expressions, radicals, and symbolic substitutions.
Keywords	equation, substitute, solve, factor, expression
Structural Cues	functional relations; inequalities or intervals
Output Format	integer, fraction, radical, interval

Table 1: Example of a structured skill schema used.

MATH stream: Algebra $\to$ Counting & Probability $\to$ Geometry $\to$ Intermediate Algebra
		Streamed domains						OOD generalization
Model	Method	Alg.	Count.	Geom.	Inter.	Stream Avg. $\uparrow$	Forget. $\downarrow$	MMLU	GSM8K	PIQA	CSQA	SciQ	OOD Avg. $\uparrow$
Qwen-3B	Base	44.99	28.48	22.34	9.19	26.25	–	65.45	63.00	77.69	75.54	92.50	74.84
Qwen-3B	Seq-SFT	26.62	21.31	18.16	13.73	19.96	42.41	57.18	7.58	74.43	61.59	93.80	58.92
Qwen-3B	SST	40.27	24.68	20.46	15.06	25.12	7.10	65.28	39.20	77.58	80.56	97.70	72.06
Llama-3B	Base	8.17	5.70	11.27	9.63	8.69	–	62.12	67.48	73.90	73.96	94.20	74.33

Table 2: Main streaming results. All numbers are percentages after the full stream. Stream Avg. : average over streamed domains. Forget. : average max-drop over prior streamed domains. OOD Avg. : average over MMLU, GSM8K, PIQA, CSQA, and SciQ (flex score for GSM8K). Sequential baseline is Seq-SFT for MATH and Seq-RL for agent tasks. SST uses the same update objective as its sequential counterpart, conditioned on skill schemas. Forgetting is not applicable to base models.

Figure 5: Baseline comparison on streaming MATH. Final accuracy after the four-domain stream. Left: seen domains. Right: held-out related MATH domains. SST yields a favorable retention-transfer trade-off on both models.

Figure 6: Underfitting does not explain SST’s retention. Underfit SFT retains more earlier-domain accuracy only by failing to learn the final domain and degrading related OOD generalization.

Figure 7: SST checkpoint trajectories on streaming MATH. Rows are checkpoints after each update and columns are evaluation domains. Partial recoveries after later updates are consistent with later domains reinforcing shared procedures.

Figure 8: Robustness to stream order. SST is trained on four permutations of the same MATH stream and evaluated after the final update. Performance is broadly consistent across orders, indicating that SST does not rely on a single favorable domain sequence.

Figure 9: Single-domain gain–drop diagnostic. Each MATH domain is trained independently from the same base checkpoint. Upward bars show target-domain gain, and downward hatched bars show average drop on the other streamed MATH domains. Compared with Full Raw , Full Skill yields a stronger gain–drop trade-off on both Qwen-3B and Llama-3B.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Category	Setting	Value
SFT hyperparameters
SFT	Training framework	LlamaFactory
SFT	Maximum sequence length	4096
SFT	Precision	bf16
SFT	Scheduler	cosine
SFT	Warmup ratio	0.03

Appendix

Table 3: Default training and evaluation settings.

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 231 papers

CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptation
Aug 31, 2026Kwangmin Ki, Yunhun Nam, Jongheon Jeong +1LLM Fine-Tuning Domain Adaptation for LLMs
Korea University · Yonsei University
RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting
May 29, 2026Yuduo Li, Xiaofeng Shi, Qian Kou +2Supervised Fine-Tuning Fine-Tuning
Beijing Academy of Artificial Intelligence (BAAI) · Beijing Jiaotong University (BJTU)
Updating Parametric Knowledge with Context Distillation Retains Post-Training Capabilities
Feb 17, 2026Shankar Padmanabhan, Mustafa Omer Gul, Tanya GoyalContinual Learning for LLMs Context Distillation
Department of Computer Science, Cornell University.

Aug 31, 2026J/K move · Enter open · S save

CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptation

Kwangmin Ki, Yunhun Nam, Jongheon Jeong, Jaehyung Kim

Korea University · Yonsei University

LLM Fine-Tuning Domain Adaptation for LLMs LLM Routing Catastrophic Forgetting

Open paper AskPDF arXiv

Supervised fine-tuning (SFT) is the de facto standard for adapting large language models (LLMs) to target domains, but it often degrades the model's general capabilities, a phenomenon known as catastrophic forgetting. Existing approaches typically modify the SFT loss to mitigate forgetting, but they inevitably operate along a domain-generality trade-off. In this work, we step outside this trade-off by decoupling the two capabilities at the model level: we keep the original base model for general capability, and selectively invoke the SFT expert only when domain-specific knowledge is required. Specifically, we propose CPR (Critical-Point Routing), a token-level routing framework between a base model and its expert derivative, based on critical tokens where the base model fails but the expert succeeds. We train a lightweight hierarchical router that estimates the expert-call probability per token, and pair it with a tailored inference procedure that combines momentum smoothing and threshold gating. Across diverse model-domain configurations, CPR achieves state-of-the-art across all settings, surpassing SFT expert by 1.4-5.5% in domain performance while recovering its general-capability drop from 3.4-14.5% to at most 0.5%, with minimal overhead from invoking the expert on only one-third of tokens.

Abstract

Continual Learning for LLMs

Explore similar work

CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptation

RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting

Updating Parametric Knowledge with Context Distillation Retains Post-Training Capabilities