Organizations: MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics
Supervised fine-tuning (SFT) can degrade factual behavior outside the target domain. This degradation is often described as catastrophic forgetting, yet open-ended factual failures do not necessarily imply that the underlying facts have been erased. In this work, we identify a more specific phenomenon, factual access failure: after domain SFT, models can still recognize or rank the correct answer under constrained evaluation, while failing to produce it in closed-book generation. Through benchmark-level comparisons, same-fact multiple-choice and generation probes, and failure-mode analysis, we show that SFT-induced factual degradation reflects both genuine wrong-answer generations and expression-level failures such as verbosity, formatting mismatch, and exact-match artifacts. To address this problem, we introduce Recall-Anchored Distillation (RAD), a base-anchored self-distillation objective that preserves out-of-distribution generation behavior by aligning the adapted model with the original base model's soft continuation distribution on unlabeled OOD text. RAD requires no gold OOD answers, external judges, or labeled factual data. Across three backbones fine-tuned on MedMCQA, RAD recovers a consistent portion of the lost OOD recall while preserving target-domain adaptation. Compared with replay on the same OOD text, RAD shows that the key preservation signal is the base model's soft distribution rather than additional text exposure alone.
Figures & tables
Figure 1: Diagnostic overview of factual access failures. Panel (a) separates latent factual storage from observable behaviors: recognition, recall, and expression. Panel (b) shows the benchmark-level comparison in Finding 1. Panel (c) shows the same-fact diagnosis in Finding 2.
Figure 2: Failure modes among Base-correct/SFT-wrong examples. Most recall failures are format or verbosity mismatches rather than clean wrong-entity errors.
Figure 3: Overview of Recall-Anchored Distillation (RAD).
In-Domain ↑
OOD Recognition (multiple choice) ↑
OOD Recall (closed-book) ↑
MedMCQA
MMLU-Med
MMLU-Other
TruthfulQA
TriviaQA
PopQA
Method
Acc
Acc
Acc
MC2
EM
F1
EM
F1
Llama-3.1-8B
Base
58.14
71.54
64.82
44.17
65.03
71.04
19.30
23.67
+ Standard SFT
59.43
72.20
65.51
42.65
43.95
56.59
11.60
16.03
+ Replay
59.53
71.90
64.88
41.48
32.87
47.80
3.40
8.06
Table 1: Main results across three backbones on in-domain MedMCQA, OOD recognition, and OOD closed-book recall.
Figure 4: Qualitative examples showing how RAD repairs open-ended factual access failures on TriviaQA.
Variant
MedMCQA
TriviaQA EM
PopQA EM
Standard SFT
59.43
43.95
11.60
RAD default
60.05
51.62
15.20
Forward KL
59.89
52.88
16.74
Symmetric KL
59.93
52.57
16.58
No clipping
59.98
51.59
16.58
τ=2.5
59.98
52.17
16.77
Table 2: Ablation results of RAD.
Figure 5: Ablation and checkpoint-dynamics results on Llama-3.1-8B . Left: effect of anchoring strength α . Right: TriviaQA accuracy gains of RAD over Standard SFT at each training epoch.
Figure 6: Training time and GPU memory usage of Standard SFT and RAD across three backbones.
Large language models are prone to hallucinating factually incorrect statements. A key source of these errors is exposure to new factual information through supervised fine-tuning (SFT), which can increase hallucinations w.r.t.~knowledge acquired during pre-training. Since these errors arise as a by-product of knowledge degradation, we explore whether established continual learning tools can mitigate them. We propose a self-distillation-based SFT method that facilitates effective factual learning while minimizing hallucinations w.r.t.~pre-existing knowledge by regularizing output-distribution drift. We also show that when new knowledge acquisition is unnecessary, suppressing factual plasticity by freezing parameter groups preserves task performance while reducing hallucinations. Lastly, we investigate the mechanism, contrasting capacity limitations, behavior cloning, and localized interference. Our experiments show that a main driver is interference among overlapping semantic representations, which self-distillation mitigates and an associative-memory model explains: forgetting grows with the overlap between new and stored facts.
Guy Kaplan, Zorik Gekhman, Zhen Zhu +5
♡Hebrew University of Jerusalem · ♣Technion – Israel Institute of Technology · ♢University of Illinois Urbana-Champaign +1
Domain-specific supervised fine-tuning (SFT) often improves in-domain performance at the cost of degrading a model's general capabilities. We view this degradation through two practical gaps in domain SFT: a supervision-compatibility gap, where domain targets differ in style and reasoning format from the original model's natural responses, and a trajectory-preservation gap, where teacher-forced SFT optimizes fixed target tokens without constraining the model's behavior on its own generated prefixes. This process fails to preserve the model's original behavior. We propose RAFT (Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting), a two-stage framework that addresses both factors. First, RAFT constructs model-compatible supervision through self-conditioned rewriting, semantic filtering, and answer fusion. Second, RAFT performs Answer-Conditioned On-Policy Distillation, where the original instruction-tuned model provides soft targets on student-generated trajectories while being conditioned on the fused answer as helpful context. We further introduce top-K temperature distillation and EMA-based adaptive loss balancing to stabilize the domain-general trade-off. Across three instruction-tuned backbones and five domains, RAFT improves average domain accuracy by 23.2% over standard SFT, while recovering part of the SFT-induced degradation on MS-Bench and IFEval, with relative improvements of 18.2% and 10.2%, respectively. These results show that coupling data refinement with trajectory-level preservation provides an effective recipe for domain fine-tuning with alleviated forgetting.
Yuduo Li, Xiaofeng Shi, Qian Kou +2
Beijing Academy of Artificial Intelligence (BAAI) · Beijing Jiaotong University (BJTU)
Supervised fine-tuning (SFT) trains a base language model to imitate target responses, and these targets may require knowledge the base model has not robustly internalized. We study this as a source of hallucinations and frame a group of mitigation methods as knowledge-aligned SFT: constraining SFT training targets to the base model's parametric knowledge. Under a unified setup, we compare existing generation-based and estimation-based knowledge-alignment methods and introduce two new variants: Evidence Rewrite, which verifies base-model generations using external evidence, and Recall Rewrite, which retains claims only when they can be consistently recalled by the base model. Experiments with Qwen 3 4B and OLMo 3 7B show that knowledge-aligned SFT can reduce factual hallucinations on WildHalu and Biography while largely preserving general capabilities. Recall Rewrite yields the strongest factuality gains and improves refusal behavior on UnknownBench. It thereby confirms that SFT targets beyond the base model's knowledge drive hallucination behavior.
Arthur Becker, Jakob Kemmler, David Thulke +3
AppTek GmbH, Aachen, Germany · Machine Learning and Human Language Technology, RWTH Aachen University, Germany