Harmful SFT Leaves a Continuous Trace in LLM Checkpoint Updates

Ziqun Bao; Xinyu Zhang; Yuchen Shao; Chengcheng Wan

cs.LGOct 5, 2026

Harmful SFT Leaves a Continuous Trace in LLM Checkpoint Updates

Authors: Ziqun Bao, Xinyu Zhang, Yuchen Shao, Chengcheng Wan

Organizations: SEI, East China Normal University

Abstract

Safety auditing of post-trained large language models typically relies on model behavior, requiring model execution and depending on the coverage of available evaluations. This work asks a different question: Do the target behaviors optimized during supervised fine-tuning (SFT) leave readable evidence directly in checkpoint updates? We find that harmful-compliance SFT induces a continuous, objective-dependent ordering in checkpoint-update space. Using a reference geometry defined by pure harmful-compliance, safety-targeted, and benign-utility SFT, we find that a checkpoint-level coordinate s_H tracks controlled harmful-objective composition with Spearman correlations of 0.986-0.992 across four 7-8B backbones, with the same ordering persisting at larger model scales. Matched compliance-versus-refusal controls show that this checkpoint trace reflects the SFT objective rather than harmful-input exposure, while additional controls rule out simple explanations based on harmful-example count or generic training intensity. Building on this structure, we introduce TRACE, a weights-only auditing method that localizes an unknown checkpoint update relative to frozen harmful and non-harmful reference prototypes and converts this geometry into a continuous harmful-objective score. TRACE requires neither model queries nor access to the unknown SFT data, and can be evaluated directly from checkpoint updates. Across distribution shifts, unseen data, different SFT configurations, partial checkpoint access, and LoRA/full-parameter fine-tuning, the trace remains stable and is positively associated with independently measured attack success rates. TRACE remains informative even at low harmful-objective proportions, providing a complementary auditing signal when behavioral evaluation is unavailable or incomplete. Code is available at https://anonymous.4open.science/r/Code4TRACE-54D3.

Figures & tables

Figure 1: Objective-induced checkpoint geometry. Pure harmful-compliance (

C

), safety-targeted (

S

), and benign-utility (

B

) SFT define distinct reference regions in checkpoint-update space, while mixed-objective checkpoints occupy ordered intermediate positions and shift toward the harmful-compliance reference as the harmful-objective ratio increases. Here,

\mu_{C}

\mu_{S}

, and

\mu_{B}

denote the reference locations associated with the three pure objectives, and

\mu_{N}

denotes a combined non-harmful reference.

Figure 2: Matched

C/S

objective traces.

Panel of Figure 3 — Figure 3: Harmful SFT leaves a continuous trace in checkpoint updates. Lines show mean $s_{H}$ ; shaded bands denote $\pm 1$ standard deviation across trajectories.

Model	$\rho$	$n_{C}$	$\rho+n_{C}$
Llama	0.923	0.757	0.962
Mistral	0.969	0.606	0.970
Qwen	0.936	0.627	0.951
OLMo	0.922	0.757	0.980

Table 1: Seed-group CV

R^{2}

for predicting

s_{H}

Figure 4: Overview of TRACE. A frozen pure-objective geometry localizes an unknown checkpoint update relative to a trusted base and produces

s_{H}

Figure 5: TRACE design choices. Balanced

C/N

references, normalized scoring, and combined q/v updates provide the strongest joint preservation of objective ordering and compliance-versus-refusal separation.

Model	$K=1$	$K=2$	$K=4$	$K=8$	$K=12$	$K=16$	$K^{\ast}$
Llama	0.955/0.0917	0.959/0.0919	0.973/0.0570	0.973/0.0294	0.977/0.0155	0.976/0.0073	4
Mistral	0.919/0.1152	0.924/0.0577	0.924/0.0464	0.959/0.0091	0.965/0.0047	0.968/0.0010	4
Qwen	0.949/0.0725	0.949/0.0774	0.961/0.0444	0.966/0.0334	0.971/0.0059	0.971/0.0040	8
OLMo	0.959/0.0539	0.968/0.0527	0.976/0.0355	0.979/0.0242	0.980/0.0112	0.980/0.0610	8

Table 2: Reference-bank sensitivity. Each entry reports

\rho/\sigma_{s}

;

K=16

is the full bank.

Method	Access	ID	Matched $C/S$	OOD	Unseen
UpdateNorm	weights only	0.7090	0.7439	0.6425	0.7036
Spectral	weights only	0.6395	0.6814	0.6317	0.6636
WeightWatch-SVD	weights only	0.5864	0.5659	0.6134	0.6610
PEFTGuard	weights + labels	0.4345	0.1519	0.3334	0.3529
CANARY	forward + probes	0.6755	0.6605	0.7343	0.7274
TRACE	weights only	0.9110	0.9881	1.0000	0.9998

Table 3: Audit separation by setting. Macro AUROC across four backbones.

Method	Llama	Mistral	Qwen	OLMo	Macro
PEFTGuard-BD	0.464	0.510	0.854	0.432	0.565
WeightWatch-SVD	0.333	0.625	0.479	0.385	0.456
WeightWatch-SVD +BackdoorRef	0.531	0.583	0.281	0.375	0.443
TRACE original	0.260	0.385	0.146	0.135	0.232
TRACE +BackdoorRef	0.719	0.615	0.990	0.563	0.721

Table 4: Backdoor auditing across methods. AUROC on held-out

1\%

–

50\%

poison-rate checkpoints.

Setting	Trace $\rho$	$C/S$ AUROC
OOD	0.791–0.970	1.0000
Unseen	0.983–0.990	0.9997

Table 5: Generalization across SFT data distributions.

Model	Spearman $\rho$
Llama	0.997–1.000
Mistral	0.994–1.000
Qwen	0.993–1.000
OLMo	0.999–1.000

Table 6: SFT configuration robustness.

Model	$L\!\rightarrow\!L$	$F\!\rightarrow\!F$	$L\!\rightarrow\!F$	$F\!\rightarrow\!L$
Llama	0.991/0.976	0.940/0.949	0.967/0.915	0.975/0.921
Mistral	0.948/0.869	0.952/0.979	0.916/0.883	0.936/0.753
Qwen	0.923/0.956	0.979/0.988	0.916/0.827	0.963/0.975
OLMo	0.991/0.960	0.991/0.980	0.991/0.930	0.991/0.986

Table 7: Transfer between LoRA and full-parameter SFT. Entries report Spearman

\rho

R^{2}

Model	Spearman $\rho$	$R^{2}$
Llama	0.931	0.803
Mistral	0.716	0.485
Qwen	0.887	0.724
OLMo	0.854	0.546

Table 8:

s_{H}

vs. ASR.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Empirical validation of coordinate-level compositionality under realistic SFT optimization. (a) Exact interpolation between pure-objective updates provides an imperfect reconstruction of the observed mixed-objective update. (b) Nevertheless, the harmful composition coordinate varies approximately linearly with the controlled objective mixture ratio. (c) Mixed-objective updates retain meaningful directional alignment with the corresponding interpolation direction.

Panel of Figure 7 — Figure 7: Relative composition versus absolute harmful exposure. Left: dataset size varies at fixed harmful-compliance ratio. Right: harmful-compliance ratio varies at fixed harmful-example count.

Figure 8: SFT intensity and update magnitude. Update magnitude grows under all three objectives, while

s_{H}

remains strongly objective dependent.

Panel of Figure 9 — Figure 9: Controlling for sequence-length effects across four backbones.

Figure 10: Binary detection across harmful-objective proportions. Top: harmful-compliance checkpoints at proportion

r

are compared with benign-only checkpoints (

C_{r}

vs.

B_{0}

). Bottom: harmful-compliance checkpoints are compared with matched refusal-targeted checkpoints (

C_{r}

vs.

S_{r}

). Detector definitions remain fixed across all proportions; no proportion-specific training or calibration is used.

View	Method	$1\%$	$2\%$	$5\%$	$10\%$	$25\%$	$50\%$	$75\%$	$100\%$
$C_{r}$ vs. $B_{0}$	TRACE	0.718	0.843	0.995	1.000	1.000	1.000	1.000	1.000
	UpdateNorm	0.500	0.513	0.605	0.665	0.770	0.910	0.943	0.963
	Spectral	0.553	0.583	0.570	0.540	0.688	0.723	0.863	0.828
	WeightWatch-SVD	0.471	0.481	0.484	0.534	0.551	0.655	0.705	0.810
	PEFTGuard	0.481	0.509	0.435	0.439	0.404	0.409	0.323	0.316
	CANARY	0.548	0.573	0.650	0.768	0.793	0.785	0.790	0.735

Appendix

Table 9: Binary detection AUROC across harmful-objective proportions. Values are macro-averaged across the four model backbones.

	$C$ -trace Spearman $\rho$		$C/S$ AUROC
Model	PKU $\rightarrow$ BT	BT $\rightarrow$ PKU	PKU $\rightarrow$ BT	BT $\rightarrow$ PKU
Llama	0.9695	0.7911	1.0000	1.0000
Mistral	0.9384	0.9695	1.0000	1.0000
Qwen	0.9695	0.9695	1.0000	1.0000
OLMo	0.9695	0.9695	1.0000	1.0000

Appendix

Table 10: Dataset-family transfer in both directions.

Figure 11: Cross-family harmful-compliance traces. References and queries use different harmful-data families.

Figure 12: Traces on manually constructed unseen data. Solid and dashed lines denote

C

and

S

, respectively.

Model	$\rho_{C+B}$	$\rho_{C}$	$C/S$ AUROC
Llama	0.9873	0.9821	1.0000
Mistral	0.9887	0.9843	0.9989
Qwen	0.9831	0.9753	1.0000
OLMo	0.9901	0.9866	1.0000

Appendix

Table 11: Unseen-data generalization.

Model	Base	LR $\downarrow$	LR $\uparrow$	Step $\downarrow$	Step $\uparrow$	$r=4$	$r=16$	$\alpha=8$	$\alpha=32$
Llama	1.000	0.998	0.998	0.997	0.998	0.999	1.000	0.998	0.999
Mistral	1.000	1.000	0.996	1.000	0.998	0.994	0.998	1.000	0.998
Qwen	1.000	0.999	0.993	0.999	0.994	1.000	0.999	0.999	0.999
OLMo	1.000	1.000	0.999	0.999	1.000	0.999	1.000	1.000	1.000

Appendix

Table 12: Correlation with the canonical

s_{H}

curve under SFT configuration changes.

Figure 13: Harmful-objective traces under SFT configuration changes.

Panel of Figure 14 — Figure 14: Sensitivity of the checkpoint trace to audited layer scope.

Figure 15: Continuous harmful-objective traces on larger Qwen3 models. The reference geometry is constructed independently within each backbone; absolute

s_{H}

values are therefore not compared across models.

Model	Trace $\rho$	$R^{2}$	Matched C/S	5% C/B
Qwen3-14B	0.994	0.976	1.000	1.000
Qwen3-32B	0.960	0.997	1.000	1.000

Appendix

Table 13: TRACE on larger Qwen3 models.

Model	Spearman $\rho$	95% CI	Pearson	$R^{2}$	Partial Corr.
Llama	0.931	[0.890, 0.952]	0.896	0.803	0.52
Mistral	0.716	[0.568, 0.806]	0.697	0.485	-0.18
Qwen	0.887	[0.791, 0.936]	0.851	0.724	0.31
OLMo	0.854	[0.743, 0.912]	0.739	0.546	0.30

Appendix

Table 14: Behavioral validity of

s_{H}

Figure 16: Checkpoint-level association between $s_{H}$ and attack success rate. Each panel corresponds to one backbone.

Figure 17: Triggered ASR across poison rates.

Method	$1\%$	$2\%$	$5\%$	$10\%$	$25\%$	$50\%$
UpdateNorm	0.313	0.375	0.500	0.469	0.438	0.125
PEFTGuard-BD	0.531	0.531	0.656	0.406	0.281	0.188
WeightWatch-SVD	0.531	0.469	0.438	0.398	0.469	0.430
WeightWatch-SVD +BackdoorRef	0.438	0.438	0.547	0.422	0.438	0.375
TRACE original	0.250	0.250	0.250	0.062	0.000	0.000
TRACE +BackdoorRef	0.625	0.625	0.750	0.625	0.688	0.692

Appendix

Table 15: Backdoor detection across poison rates.

Figure 18: Backdoor auditing and reference adaptation. (A) Pooled held-out backdoor detection across baselines and reference-adapted variants over

1\%

–

50\%

poison rates. (B) Association between the TRACE checkpoint score and triggered ASR before and after adding the backdoor reference.

Setting	Original	$+\mu_{T}$	$\Delta$	95% CI
ID	0.9110	0.8745	-0.0365	$[-0.0710,-0.0045]$
Matched C/S	0.9881	0.9924	+0.0043	$[+0.0004,+0.0088]$
OOD	1.0000	1.0000	+0.0000	$[0.0000,0.0000]$
Unseen	0.9998	0.9543	-0.0455	$[-0.0762,-0.0181]$

Appendix

Table 16: Backward compatibility after adding the backdoor reference.

\Delta

reports the change in macro AUROC after reference-space expansion.

Figure 19: Backward compatibility of backdoor reference expansion. Left: change in the original harmful-SFT detection tasks after adding

\mu_{T}

. Right: retention of continuous harmful-objective ordering.

Figure 20: Construction of practical heterogeneous SFT trajectories. (a) The 10k-example Tulu3-Clean background contains 13 heterogeneous instruction-tuning sources. (b) Harmful-compliance examples progressively replace benign examples while total size remains fixed at 10,000.

Figure 21: TRACE under practical heterogeneous SFT. Harmful-objective proportion versus

s_{H}

, proportion versus ASR, and checkpoint-level

s_{H}

–ASR association. Each point is an independently trained 10k-example checkpoint; the reference geometry remains frozen.

Figure 22: Objective controls and training dynamics. (a–b) Matched

C/S

trajectories differ only in harmful-response targets. (c–d) Along

5\%

harmful-compliance trajectories,

s_{H}

and ASR generally increase during SFT. Thin lines show seeds and thick lines show means.

Explore similar work

Jun 26, 2026cs.LG

TRACE: Trajectory-Based Safety Patch Learning for LLM Post-Training Realignment

Fine-Tuning-as-a-Service (FTaaS) platforms let users perform supervised fine-tuning (SFT) on customized data, but this pipeline can erode model safety alignment. To recover safety without re-running full alignment, existing realignment methods focus on calibrating the integration of safety patches into fine-tuned models. These methods exhibit a persistent safety-utility trade-off: weak repair leaves harmful behavior intact, while stronger repair increasingly damages the benign task. This paper shifts the focus from online calibration to offline patch learning and aims to learn a safety patch that restores safety while preserving task-specific capabilities. To this end, we propose TRACE, which simulates harmful SFT trajectories to produce progressively corrupted model states, and optimizes a safety patch simultaneously across these states. TRACE trains the safety patch during the offline stage, and reuses it across all user checkpoints without per-user calibration. We evaluate two representative models using three harmful SFT datasets, together with three utility benchmarks. Across six benchmarks and two models, TRACE consistently dominates the safety-utility frontier. TRACE improves the safety rate by up to 77 percentage points over the best baselines, while maintaining comparable utility to the undefended fine-tuned model.

Changyue Li, Jiaming He, Youliang Yuan +4

The Chinese University of Hong Kong, Shenzhen · Ant Group · Lero the Research Ireland Centre for Software, University of Limerick

Supervised Fine-Tuning LLM Safety Alignment LLM Safety LLM Post-Training

Ask PDF

May 12, 2026cs.AI

A Mechanistic Investigation of Supervised Fine Tuning

The cosine similarity between a large language model's hidden activations before and after Supervised Fine-Tuning (SFT) remains very high. This, at first glance, suggests that SFT leaves the model's activation geometry largely undisturbed. However, projecting both sets of activations through a Sparse Autoencoder (SAE) pretrained on the base model reveals that the underlying sparse latents diverge significantly. We introduce a novel investigative pipeline which utilizes these pretrained SAEs as a high-resolution diagnostic tool to mechanistically investigate the drivers of this representational divergence. Through our analytical pipeline, we discover task-specific and layer-specific distributions of the precise semantic features that are systematically altered during supervised fine-tuning. We additionally identify a layer-wise update profile specific to safety alignment. All code, experimental scripts, and analysis files associated with this work are publicly available at: https://github.com/ruhzi/sae-investigation.

Ruhaan Chopra

Independent Researcher

Supervised Fine-Tuning LLM Alignment LLM Interpretability Representation Drift

Ask PDF

Jul 27, 2026cs.AI

Similar Models Learn Differently: Final-Window Pretraining Shapes Post-Training Beyond SFT

Developers judge a model checkpoint by how it behaves. After supervised fine-tuning (SFT), two checkpoints that perform about the same across relevant benchmarks are treated as interchangeable, equally ready for the next alignment stage, typically preference optimization. We ask whether this judgment misses a pretraining imprint: a difference that no post-SFT benchmark reveals, yet that decides how each checkpoint responds to further training. To find out, we run a controlled experiment on the final window of pretraining, the last data trained on before instruction tuning. Six branches fork from one partially pretrained checkpoint and differ only in this window: 500 million tokens, 0.1% to 1% of the tokens that precede it. Each branch trains its window on a single data source: generic web text, filtered web text, normative discourse, safety text, mathematical text, or synthetic educational text. SFT and post-training are then identical. After SFT the branches behave near-identically, within about one point on instruction following, refusal, and capability, yet the same post-training carries them to very different endpoints, under both a direct preference optimization update and a reinforcement learning update with a verifiable reward. We measure this deviation through refusal of harmful requests: when post-training begins the safety text branch refuses no more than the web text branch, yet by the end it has lost far less of its refusal. The other four branches gain little or no protection, so the effect is selective to what the window contained. The protection requires the safety text to arrive last rather than earlier in pretraining, and it reproduces on a second model family. What a model is pretrained on last shapes how it reacts to alignment. Therefore, a checkpoint should not be evaluated by its post-SFT behavior alone, and what it was trained on last should be reported with it.

Cen Lu, Yung-Chen Tang, Andrea Cavallaro

1EPFL · 2Idiap Research Institute

Language Model Pretraining LLM Safety Alignment Continued Pretraining

Ask PDF