Adapting language models to new domains during post-training risks degrading previously acquired capabilities, a phenomenon commonly known as catastrophic forgetting. Existing mitigation methods largely treat forgetting as a consequence of excessive model drift. They therefore constrain updates by preserving old outputs, replaying old data, or regularizing changes to parameters and representations so that the updated model remains close to its previous state. We take a different view: Rather than pulling the updated model back toward its old state, we translate the new domain into the model's pre-existing knowledge space, making unfamiliar knowledge learnable through familiar procedures. Therefore, we propose Skill Schema Transport (SST), a post-training framework that transports new knowledge into the model's existing skill space. SST abstracts each example type into a skill schema, preserving its essential operations, dependencies, and verification steps while removing domain-specific surface forms. By recasting new examples as instances of reusable procedures, SST turns knowledge update from trajectory memorization into procedural reuse. In continual domain adaptation settings, SST consistently reduces forgetting while improving generalization to related tasks. Across two 3B instruction-tuned backbones, SST improves average final performance on streamed domains by 2.53 to 5.75 points over sequential post-training. It also lowers max-drop forgetting in all evaluated settings, with the largest reduction from 42.41 to 7.10 points on the Qwen-3B MATH stream, while substantially recovering the OOD degradation induced by sequential updates.
Figures & tables
Figure 1: Overview of Skill Schema Transport (SST). SST recasts new-domain examples as instances of reusable skill schemas rather than anchoring old data, output distributions, or parameter values. The left panels illustrate the drift diagnostics used in our analysis, and the bottom pipeline shows how procedural skills are extracted, abstracted, stored, and used for schema-conditioned updates.
Figure 2: Surface drift and representation drift need not align. Domain-shifted updates induce different distributions of output-level and hidden-state change, motivating diagnostics that track both observable behavior and internal representations.
Figure 3: Errors cluster by procedure rather than by domain label. The same failed examples are projected and colored by procedural error type on the left and by MATH category on the right. Error clusters cut across category boundaries, motivating cross-domain skill schema reuse.
Field
Description
Skill Name
Algebraic Substitution and Equation Solving
Goal
Solve algebra problems via rewriting, substitution, and branch verification.
Applicability
Equations, expressions, radicals, and symbolic substitutions.
Keywords
equation, substitute, solve, factor, expression
Structural Cues
functional relations; inequalities or intervals
Output Format
integer, fraction, radical, interval
Table 1: Example of a structured skill schema used.
MATH stream: Algebra → Counting & Probability → Geometry → Intermediate Algebra
Streamed domains
OOD generalization
Model
Method
Alg.
Count.
Geom.
Inter.
Stream Avg. ↑
Forget. ↓
MMLU
GSM8K
PIQA
CSQA
SciQ
OOD Avg. ↑
Qwen-3B
Base
44.99
28.48
22.34
9.19
26.25
–
65.45
63.00
77.69
75.54
92.50
74.84
Qwen-3B
Seq-SFT
26.62
21.31
18.16
13.73
19.96
42.41
57.18
7.58
74.43
61.59
93.80
58.92
Qwen-3B
SST
40.27
24.68
20.46
15.06
25.12
7.10
65.28
39.20
77.58
80.56
97.70
72.06
Llama-3B
Base
8.17
5.70
11.27
9.63
8.69
–
62.12
67.48
73.90
73.96
94.20
74.33
Table 2: Main streaming results. All numbers are percentages after the full stream. Stream Avg. : average over streamed domains. Forget. : average max-drop over prior streamed domains. OOD Avg. : average over MMLU, GSM8K, PIQA, CSQA, and SciQ (flex score for GSM8K). Sequential baseline is Seq-SFT for MATH and Seq-RL for agent tasks. SST uses the same update objective as its sequential counterpart, conditioned on skill schemas. Forgetting is not applicable to base models.
Figure 5: Baseline comparison on streaming MATH. Final accuracy after the four-domain stream. Left: seen domains. Right: held-out related MATH domains. SST yields a favorable retention-transfer trade-off on both models.
Figure 6: Underfitting does not explain SST’s retention. Underfit SFT retains more earlier-domain accuracy only by failing to learn the final domain and degrading related OOD generalization.
Figure 7: SST checkpoint trajectories on streaming MATH. Rows are checkpoints after each update and columns are evaluation domains. Partial recoveries after later updates are consistent with later domains reinforcing shared procedures.
Figure 8: Robustness to stream order. SST is trained on four permutations of the same MATH stream and evaluated after the final update. Performance is broadly consistent across orders, indicating that SST does not rely on a single favorable domain sequence.
Figure 9: Single-domain gain–drop diagnostic. Each MATH domain is trained independently from the same base checkpoint. Upward bars show target-domain gain, and downward hatched bars show average drop on the other streamed MATH domains. Compared with Full Raw , Full Skill yields a stronger gain–drop trade-off on both Qwen-3B and Llama-3B.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Category
Setting
Value
SFT hyperparameters
SFT
Training framework
LlamaFactory
SFT
Maximum sequence length
4096
SFT
Precision
bf16
SFT
Scheduler
cosine
SFT
Warmup ratio
0.03
Appendix
Table 3: Default training and evaluation settings.
Supervised fine-tuning (SFT) is the de facto standard for adapting large language models (LLMs) to target domains, but it often degrades the model's general capabilities, a phenomenon known as catastrophic forgetting. Existing approaches typically modify the SFT loss to mitigate forgetting, but they inevitably operate along a domain-generality trade-off. In this work, we step outside this trade-off by decoupling the two capabilities at the model level: we keep the original base model for general capability, and selectively invoke the SFT expert only when domain-specific knowledge is required. Specifically, we propose CPR (Critical-Point Routing), a token-level routing framework between a base model and its expert derivative, based on critical tokens where the base model fails but the expert succeeds. We train a lightweight hierarchical router that estimates the expert-call probability per token, and pair it with a tailored inference procedure that combines momentum smoothing and threshold gating. Across diverse model-domain configurations, CPR achieves state-of-the-art across all settings, surpassing SFT expert by 1.4-5.5% in domain performance while recovering its general-capability drop from 3.4-14.5% to at most 0.5%, with minimal overhead from invoking the expert on only one-third of tokens.
Domain-specific supervised fine-tuning (SFT) often improves in-domain performance at the cost of degrading a model's general capabilities. We view this degradation through two practical gaps in domain SFT: a supervision-compatibility gap, where domain targets differ in style and reasoning format from the original model's natural responses, and a trajectory-preservation gap, where teacher-forced SFT optimizes fixed target tokens without constraining the model's behavior on its own generated prefixes. This process fails to preserve the model's original behavior. We propose RAFT (Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting), a two-stage framework that addresses both factors. First, RAFT constructs model-compatible supervision through self-conditioned rewriting, semantic filtering, and answer fusion. Second, RAFT performs Answer-Conditioned On-Policy Distillation, where the original instruction-tuned model provides soft targets on student-generated trajectories while being conditioned on the fused answer as helpful context. We further introduce top-K temperature distillation and EMA-based adaptive loss balancing to stabilize the domain-general trade-off. Across three instruction-tuned backbones and five domains, RAFT improves average domain accuracy by 23.2% over standard SFT, while recovering part of the SFT-induced degradation on MS-Bench and IFEval, with relative improvements of 18.2% and 10.2%, respectively. These results show that coupling data refinement with trajectory-level preservation provides an effective recipe for domain fine-tuning with alleviated forgetting.
Yuduo Li, Xiaofeng Shi, Qian Kou +2
Beijing Academy of Artificial Intelligence (BAAI) · Beijing Jiaotong University (BJTU)
Post-training endows pretrained LLMs with a variety of desirable skills, including instruction-following, reasoning, and others. However, these post-trained LLMs only encode knowledge up to a cut-off date, necessitating continual adaptation. Unfortunately, existing solutions cannot simultaneously learn new knowledge from an adaptation document corpora and mitigate the forgetting of earlier learned capabilities. To address this, we introduce Distillation via Split Contexts (DiSC), a simple context-distillation based approach for continual knowledge adaptation. \methodname~derives student and teacher distributions by conditioning on distinct segments of the training example and minimizes the KL divergence between the shared tokens. This allows us to efficiently apply context-distillation without requiring explicit generation steps during training. We run experiments on four post-trained models and two adaptation domains. Compared to prior finetuning and distillation methods for continual adaptation, DiSC consistently reports the best trade-off between learning new knowledge and mitigating forgetting of previously learned skills like instruction-following, reasoning, and factual knowledge.
Shankar Padmanabhan, Mustafa Omer Gul, Tanya Goyal
Department of Computer Science, Cornell University.