Geometric Calibration of Supervised Fine-Tuning

Jianzhu Bao; Jia-Chen Gu; Qingyuan Liu; Haozhen Zhang; Kai-Wei Chang; Nanyun Peng; Wenya Wang

cs.LGOct 13, 2026

Geometric Calibration of Supervised Fine-Tuning

Authors: Jianzhu Bao, Jia-Chen Gu, Qingyuan Liu, Haozhen Zhang, Kai-Wei Chang, Nanyun Peng, Wenya Wang

Organizations: Nanyang Technological University · University of California, Los Angeles

Abstract

Recent studies investigate the weight geometry through its spectral structure, which characterizes how the weights transform different input directions. Within this view, supervised fine-tuning (SFT) is shown to induce dense updates that perturb the spectral geometry of pre-SFT weights along their principal directions, potentially disrupting the model's prior knowledge. These observations motivate using the geometry of pre-SFT weights as a reference for refining learned SFT updates. We study this perspective as Geometric Calibration and develop SPACE Subspace Projection for Adapted Checkpoint Enhancement), a lightweight post-hoc method that selectively reverts SFT update components within the principal subspace of the pre-SFT weights while preserving those in its orthogonal complement. SPACE is a data-free, training-free method that requires only the pre- and post-SFT checkpoints. We evaluate SPACE on Qwen2.5, Qwen3, Qwen3.5, Llama-3.1, and Olmo-3 models ranging from 2B to 14B, across mathematical, coding, and medical SFT, as well as several SFT variants. Overall, SPACE mitigates out-of-domain (OOD) capability regression while further improving in-domain (ID) performance over the uncalibrated SFT checkpoints. For math SFT, SPACE yields average ID/OOD gains of 2.85/2.26 points on Qwen2.5-7B, and 2.41/1.52 points on Qwen3-14B-Base, respectively. These results highlight how the geometric relationship between SFT updates and pre-SFT weights shapes the adapted model's capabilities and, more broadly, motivate geometry-aware parameter updates for continual model improvement.

Figures & tables

Figure 1: Overview of SPACE and results. (a) Method. SPACE uses the principal input subspace of the pre-SFT weight as a geometric reference for calibrating the post-SFT weights. The pre-SFT weight is visualized as rank-one SVD components ordered by singular value, while the post-SFT weight is decomposed along the same pre-SFT input directions. Blue and green denote pre- and post-SFT components, respectively, with color intensity reflecting the corresponding pre-SFT singular value. SPACE restores the pre-SFT transformation along the selected principal directions while retaining the post-SFT transformation along the remaining directions. (b) Results. After regular SFT on mathematical data, SPACE improves both in-domain (ID) and out-of-domain (OOD) performance over the uncalibrated SFT checkpoints.

Model	Benchmark	Base	SFT	WiSE-FT	TIES	LiNeS	DARE	POME	FAPM	DG-Hard	MonoSoup	SPACE
Qwen2.5-7B	In-domain (ID)
	MATH500	28.79	44.84	45.14	45.70	43.89	44.13	46.23	46.83	47.50	46.43	49.36
	AIME25	0.42	0.83	0.83	0.42	0.63	0.42	0.83	1.46	0.63	1.04	1.46
	AIME24	1.65	1.04	1.04	0.63	0.83	0.83	1.04	1.67	1.46	1.46	1.46
	OlympiadBench	7.69	14.08	17.06	14.62	14.34	13.82	15.75	17.44	15.39	15.06	17.72
	Minerva Math	9.26	12.68	11.60	12.82	10.62	11.53	14.27	11.79	13.26	12.68	15.26

Table 1: Main experimental results on mathematical reasoning SFT. We compare ID and OOD performance across different post-hoc methods. “Avg. (

\Delta

vs. SFT)” denotes the average score change relative to the SFT checkpoint. Bold denotes the best score excluding the Base model.

Figure 2: Ablation study on Qwen2.5-7B. “Avg. (

\Delta

vs. SFT)” denotes the average changes relative to the SFT checkpoint over the six ID mathematical benchmarks and the six OOD benchmarks. In (c), “Attn.” denotes attention projection matrices; “Gate”, “Up”, and “Down” denote the corresponding MLP matrices; “MLP” includes all MLP projection matrices; “Head+Emb.” denotes the output head and embedding matrices; and “All 2D” denotes all two-dimensional weight matrices. In (d), “R” and “L” denote the right and left singular vectors, while “top” and “bottom” denote the leading and trailing singular subspaces with their dimensions determined using the default threshold

\rho=0.5

. “

W_{0}

” and “

\Delta W

” indicate that the subspace is derived from the pre-SFT weights and the SFT update, respectively. Shading indicates the default SPACE configuration in each panel.

Coding SFT						Medical SFT
Benchmark	Base	SFT	DG-Hard	MonoSoup	SPACE	Benchmark	Base	SFT	DG-Hard	MonoSoup	SPACE
In-domain (ID), Coding						In-domain (ID), Medical
HumanEval	77.44	78.66	75.61	76.83	78.66	MedMCQA	53.74	54.79	54.22	54.12	55.25
HumanEval+	71.34	71.95	70.73	71.34	73.78	MedQA	58.68	65.12	60.80	63.39	64.65
MBPP	75.40	78.04	77.78	78.31	78.31	PubMedQA	71.70	70.00	70.60	70.00	73.20
MBPP+	65.61	67.99	67.99	68.78	67.99	MMLU-Pro (M)	55.83	57.65	55.83	59.15	59.54

Table 2: Experimental results of training Qwen2.5-7B on coding and medical data. “(M)” denotes the medical subset. Bold indicates the best result excluding the Base model.

Benchmark	Base	DFT			ASFT			RPSFT
Benchmark	Base	Original	MonoSoup	SPACE	Original	MonoSoup	SPACE	Original	MonoSoup	SPACE
In-domain (ID)
MATH500	28.79	54.40	57.33	58.64	60.55	61.56	61.12	45.98	46.64	48.69
AIME25	0.42	1.88	1.46	1.25	1.67	2.08	1.88	1.04	0.63	0.83
AIME24	1.65	2.08	3.54	3.75	5.83	5.00	5.21	1.46	1.25	1.25
OlympiadBench	7.69	23.22	26.29	27.18	27.98	28.81	28.45	14.40	15.53	17.03

Table 3: Compatibility with different SFT methods. “Original” denotes the checkpoint produced by each training method. “Avg. (

\Delta

vs. Original)” denotes the average score change relative to the original checkpoint. Bold indicates the best score within each method.

Benchmark	Qwen3.5-2B-Base			Qwen3-8B-Base			Olmo-3-7B-Instruct-SFT			Llama-3.1-8B-Instruct
Benchmark	Base	SFT	SPACE	Base	SFT	SPACE	Base	SFT	SPACE	Base	SFT	SPACE
In-domain (ID)
MATH500	72.40	76.47	78.00	55.93	83.73	84.87	60.80	87.93	89.33	39.07	49.20	53.33
AIME25	15.63	17.29	17.50	4.58	23.96	24.38	3.33	35.00	34.38	0.21	1.46	1.25
AIME24	12.92	16.67	15.21	7.29	22.71	22.71	6.67	41.25	41.46	1.88	1.67	2.29
OlympiadBench	37.09	40.15	41.38	24.05	46.86	48.79	25.88	55.26	56.79	10.47	18.96	21.04

Table 4: Generalization of SPACE across different model families and scales.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

In-domain (ID)
Benchmark	LoRA
Benchmark	Original	MonoSoup	SPACE
MATH500	45.30	46.06	46.34
AIME25	0.21	1.25	0.63
AIME24	0.83	0.83	2.50
OlympiadBench	14.82	14.95	15.40

Appendix

Table 5: Compatibility of SPACE with LoRA fine-tuning on Qwen2.5-7B (

r=64

\alpha=128

). “Original” denotes the LoRA fine-tuned checkpoint. “Avg. (

\Delta

vs. Original)” denotes the average score change after applying MonoSoup or SPACE relative to Original within each benchmark group. Bold indicates the best score.

In-domain (ID)
Benchmark	Qwen3.5-2B-Base				Qwen3-4B-Base				Qwen3-8B-Base
Benchmark	Base	SFT	MonoSoup	SPACE	Base	SFT	MonoSoup	SPACE	Base	SFT	MonoSoup	SPACE
MATH500	72.40	76.47	78.07	78.00	40.13	78.73	80.53	81.33	55.93	83.73	84.67	84.87
AIME25	15.63	17.29	17.29	17.50	2.29	17.50	20.00	18.13	4.58	23.96	23.96	24.38
AIME24	12.92	16.67	14.79	15.21	5.63	17.71	16.25	18.54	7.29	22.71	23.33	22.71
OlympiadBench	37.09	40.15	41.93	41.38	19.56	40.89	43.26	42.72	24.05	46.86	48.49	48.79

Appendix

Table 6: Generalization of SPACE across Qwen models at different scales (2B, 4B, and 8B).

In-domain (ID)
Benchmark	Olmo-3-7B-Instruct-SFT				Llama-3.1-8B-Instruct
Benchmark	Base	SFT	MonoSoup	SPACE	Base	SFT	MonoSoup	SPACE
MATH500	60.80	87.93	88.67	89.33	39.07	49.20	52.80	53.33
AIME25	3.33	35.00	30.83	34.38	0.21	1.46	1.04	1.25
AIME24	6.67	41.25	38.33	41.46	1.88	1.67	2.50	2.29
OlympiadBench	25.88	55.26	55.75	56.79	10.47	18.96	19.46	21.04

Appendix

Table 7: Generalization of SPACE across Olmo and Llama model families.

Figure 3: Performance of SFT and SPACE at different stages of the Qwen2.5-7B training trajectory. We report the average change over the six ID and six OOD benchmarks relative to the final 390-step SFT checkpoint. The 390-step checkpoint corresponds to one epoch of SFT.

In-domain (ID)
Benchmark	Batch size 128		Batch size 256 †		Batch size 512
Benchmark	SFT	SPACE	SFT	SPACE	SFT	SPACE
MATH500	43.40	48.88	44.84	49.36	45.31	48.45
AIME25	0.83	1.04	0.83	1.46	0.42	0.83
AIME24	0.21	2.08	1.04	1.46	0.63	1.25
OlympiadBench	13.06	18.49	14.08	17.72	14.31	17.70

Appendix

Table 8: Performance of SPACE across SFT batch sizes for Qwen2.5-7B fine-tuned on NuminaMath-CoT. The learning rate is fixed at

5\times 10^{-5}

, and training lasts one epoch. Projection uses

\rho=0.5

\alpha=1

, and all weight matrices. Bold marks the best score in each pair. † denotes the hyperparameters we used in the main experiment.

In-domain (ID)
Benchmark	LR 2e-5		LR 5e-5 †		LR 1e-4
Benchmark	SFT	SPACE	SFT	SPACE	SFT	SPACE
MATH500	45.60	44.80	44.84	49.36	39.41	49.21
AIME25	0.42	0.00	0.83	1.46	0.63	1.25
AIME24	1.67	1.46	1.04	1.46	1.04	1.25
OlympiadBench	14.83	14.29	14.08	17.72	10.98	18.18

Appendix

Table 9: Performance of SPACE across SFT learning rates for Qwen2.5-7B fine-tuned on NuminaMath-CoT. The batch size is fixed at 256, and training lasts one epoch. Projection uses

\rho=0.5

\alpha=1

, and all weight matrices. Bold marks the best score in each pair. † denotes the hyperparameters we used in the main experiment.

In-domain (ID)
Benchmark	Epochs 2		Epochs 4		Epochs 8
Benchmark	SFT	SPACE	SFT	SPACE	SFT	SPACE
MATH500	53.06	53.55	56.04	56.04	55.49	58.43
AIME25	1.88	1.25	2.29	1.04	2.29	2.50
AIME24	2.08	2.92	2.71	2.71	3.54	3.33
OlympiadBench	19.06	20.27	22.77	23.36	23.43	26.32

Appendix

Table 10: Performance of SPACE across SFT training epochs for Qwen2.5-7B fine-tuned on NuminaMath-CoT. The learning rate is fixed at

5\times 10^{-5}

, and the batch size is fixed at 256. Projection uses

\rho=0.5

\alpha=1

, and all weight matrices. Bold marks the best score in each pair.

In-domain (ID)
Benchmark	Base	SFT	RPSFT ( $\lambda=1$ )	Proj. Loss ( $\lambda=1$ )	Online Proj. ( $K=1$ )	Online Proj. ( $K=100$ )	SPACE (post-hoc)
MATH500	28.79	44.84	45.98	45.66	46.08	45.54	49.36
AIME25	0.42	0.83	1.04	0.63	0.42	0.63	1.46
AIME24	1.65	1.04	1.46	1.04	0.83	1.25	1.46
OlympiadBench	7.69	14.08	14.40	13.89	14.54	14.73	17.72
Minerva Math	9.26	12.68	13.01	12.73	12.80	12.64	15.26

Appendix

Table 11: Comparison of training-time and post-hoc projection on Qwen2.5-7B. “Proj. Loss” introduces projection-based regularization during SFT with weight

\lambda=1

. “Online Proj.” applies the projection every

K

optimization steps and at the end of training. SPACE denotes the default post-hoc projection. “Avg. (

\Delta

vs. SFT)” denotes the average score change relative to SFT. Bold denotes the best score excluding the Base model.

Explore similar work

Fora: From Weight-Space to Function-Space Protection in Capability-Preserving Fine-Tuning
Jun 30, 2026Rui Zhou, Tianci XieFine-Tuning LLM Fine-Tuning
Dalian University of Technology · Zhejiang University
Rotation-Preserving Supervised Fine-Tuning
May 8, 2026Hangzhan Jin, Tianwei Ni, Lu Li +3Supervised Fine-Tuning Fine-Tuning
1Mila - Quebec AI Institute · 2Polytechnique Montréal · 3Université de Montréal +3
Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts
Feb 16, 2026Buze Zhang, Jinkai Tao, Zilang Zeng +4Parameter-Efficient Fine-Tuning Representation Geometry in Language Models
School of Software Engineering, Xi’an Jiaotong University, China · School of Information, Central University of Finance and Economics, Beijing, China · Department of Artificial Intelligence, Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong, China +2

Jun 30, 2026J/K move · Enter open · S save

Fora: From Weight-Space to Function-Space Protection in Capability-Preserving Fine-Tuning

Rui Zhou, Tianci Xie

Dalian University of Technology · Zhejiang University

Fine-Tuning LLM Fine-Tuning

Open paper AskPDF arXiv

Full fine-tuning adapts large language models to new tasks but can erode capabilities they already possess. Existing remedies protect through proxies such as parameter distances, importance penalties, output matching, or dominant singular directions of the weights, but none directly asks which activation directions the preserved capability relies on. We argue that a capability is characterized more faithfully by the activation subspace it induces than by the singular geometry of the weight matrix, and develop function-space protection, instantiated as FORA (Function-space Orthogonal Residual Adaptation). From label-free calibration inputs, FORA estimates, per layer, the principal directions

Q

of the input-activation covariance and forms a right projector

P_Q = I - QQ^T

. Paired with a left projector

P_U

from the weight SVD, the update is

ΔW = P_U M P_Q + U_2 D_δ V_2^T

: a high-capacity branch structurally barred from reading capability-relevant function directions, plus a narrow spectral channel for controlled plasticity. The construction extends to parameter-efficient adaptation via

M \to (α/r) BA

. Across three settings on Qwen3-1.7B, including COGS and GSM8K learned while preserving translation and translation learned while preserving math, FORA consistently improves preservation over weight-space projection and standard regularization, with only a small new-task trade-off in the math-preservation setting. A controlled ablation isolating the projection source shows that the advantage comes not from projection itself, but from projecting onto capability-derived rather than weight-derived directions. Code is available at https://github.com/zrui239/FORA.

Abstract

Explore similar work

Fora: From Weight-Space to Function-Space Protection in Capability-Preserving Fine-Tuning

Rotation-Preserving Supervised Fine-Tuning

Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts