Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature

Angelo Porrello; Pietro Buzzega; Felix Dangel; Thomas Sommariva; Riccardo Salami; Lorenzo Bonicelli; Simone Calderara

cs.AIFeb 19, 2026

Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature

Authors: Angelo Porrello, Pietro Buzzega, Felix Dangel, Thomas Sommariva, Riccardo Salami, Lorenzo Bonicelli, Simone Calderara

Organizations: University of Modena and Reggio Emilia, Italy · Vector Institute, Toronto

Abstract

Task Arithmetic yields a modular, scalable way to adapt foundation models. Combining multiple task vectors, however, can lead to cross-task interference, causing representation drift and degraded performance. Representation drift regularization provides a natural remedy to disentangle task vectors; however, existing approaches typically require external task data, conflicting with modularity and data availability constraints (e.g., privacy requirements). We propose a dataless approach by framing regularization against representation drift as a curvature matrix approximation problem. This allows us to leverage well-established techniques; in particular, we adopt Kronecker-Factored Approximate Curvature and obtain a practical regularizer that achieves state-of-the-art results in task addition and negation. Our method has constant complexity in the number of tasks and promotes robustness to task vector rescaling, eliminating the need for held-out tuning.

Figures & tables

Panel of Figure 1 — Figure 1: Weight disentanglement (left) without and (right) with Jacobian Gram regularization.

Figure 2

Figure 2: Impact of regularization on “8 Vision” — CLIP ViT-B/16 (abs. accuracy). Left : linearized fine-tuning regime . Right : non-linear regime . See the Appendix for CLIP ViT-B/32 and -L/14.

Method	Dataless	$\mathbf{\alpha}$	ViT-B/32		ViT-B/16		ViT-L/14
Method	Dataless	$\mathbf{\alpha}$	Abs.	Norm.	Abs.	Norm.	Abs.	Norm.
Pre-trained	–	–	$48.4$	–	$55.4$	–	$65.0$	–
Individual	–	–	$90.9$	–	$92.4$	–	$93.8$	–
Linear Fine-Tuning
Linear FT	–	$1.0$	$76.7$	$87.2$	$80.2$	$88.9$	$88.0$	$94.8$
Linear FT	–	Best	$78.8$	$89.9$	$82.0$	$90.9$	$88.0$	$94.8$

Table 1: Task addition results on 8 Vision. The “

\alpha

” column specifies how task vector coefficients are chosen. “

1.0

” denotes that all coefficients are fixed to

1.0

, with no tuning. Numbers marked with

\dagger

for TaLoS ( Iurada et al., 2025 ) are taken from the original paper. See Fig. 2 for a task-wise plot.

Method	Dataless	ViT-B/32		ViT-B/16		ViT-L/14
Method	Dataless	Targ. $\downarrow$	Cont. $\uparrow$	Targ. $\downarrow$	Cont. $\uparrow$	Targ. $\downarrow$	Cont. $\uparrow$
Pre-trained	–	$48.4$	$63.3$	$55.4$	$68.3$	$65.0$	$75.5$
Non-linear FT	–	$20.4$	$60.5$	$20.4$	$65.3$	$18.1$	$72.4$
Linear FT	–	$9.3$	$60.5$	$8.3$	$65.5$	$7.5$	$72.1$
TaLoS † ( Iurada et al., 2025 )	✓	$11.0$	$60.7$	$10.6$	$66.1$	$10.7$	$73.6$
$\tau$ Jp ( Yoshida et al., 2025 )	×	$6.7$	$60.8$	$4.7$	$66.0$	$3.7$	$\mathbf{73.0}$

Table 2: Task negation on 8 Vision. We report the minimum accuracy on target tasks while preserving at least 95% of the pretrained model’s accuracy on control tasks.

Panel of Figure 3 — Figure 3: Results for language tasks. Left : impact of different training strategies and sensitivity to $\alpha$ hyperparameter. Right : effects of different regularizations on linear and non-linear fine-tuning.

Panel of Figure 4 — Figure 4: For ViT-B/32 (8 Vision), we analyze the sensitivity of different merging strategies to the scaling coefficient $\alpha$ ; a similar analysis for ViT-B/16 is reported in the Appendix. Left: $\alpha$ -sweep accuracy of post-hoc merging strategies in the non-linear regime, compared with our linearized and regularized models. Right: performance of merging methods on linearized checkpoints.

Figure 5: Distribution of $\left\lVert\mathrm{J}_{{\bm{\theta}}}f({\bm{x}},{\bm{\theta}}_{0})\bm{\tau}_{t}\right\rVert_{2}^{2}$ for inputs originating from the training distribution of task $t$ (inliers) versus from other tasks (outliers), under both regularized and non-regularized FT.

Method	Complexity	$\alpha$	ViT-B/32		ViT-B/16		T5-base
Method	Complexity	$\alpha$	Abs.	Norm.	Abs.	Norm.	Abs.	Norm.
Naïve Multi-Task FT	$\mathcal{O}(T)$	$1.0$	$86.5$	$98.4$	88.0	$97.5$	$78.5$	$97.0$
Naïve Multi-Task FT	$\mathcal{O}(T)$	Best	$86.6$	$98.5$	$88.1$	$97.6$	$78.5$	$97.0$
Accumulated reg. (TAK)	$\mathcal{O}(1)$	$1.0$	$85.8$	$97.6$	$88.3$	$97.9$	$78.6$	$98.7$
Accumulated reg. (TAK)	$\mathcal{O}(1)$	Best	$86.0$	$97.8$	$88.3$	$98.1$	$78.7$	$98.9$

Table 3: Our Kronecker-accumulation heuristic vs. the idealized multi-task formulation.

Panel of Figure 6 — Figure 6: Analysis of the overhead of KFAC regularization during training and pre-computation.

Panel of Figure 7 — Figure 7: Effect of KFAC approximation efficiency on performance. Left: impact of the number of datapoints used to estimate the KFAC matrices on downstream accuracy of the merged model. Right: accuracy–memory trade-off induced different KFAC matrix compression strategies.

Figure 8: Applying the KFAC loss every N steps.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Visualization of weight disentanglement ( Ortiz-Jimenez et al., 2023 ) in ViT-B/16. Non linear fine-tuning Ilharco et al. (2022) , Linear fine-tuning Ortiz-Jimenez et al. (2023) , Attention-Only fine-tuning Jin et al. (2025) , Linear fine-tuning with KFAC regularization.

Figure 10: Impact of training and regularization choices on vision tasks (absolute accuracy). Top: linearized regime, compared against the diagonal approximation. Bottom: non-linear regime, compared against attention-only fine-tuning.

Method	$\alpha$	ViT-B/32		ViT-B/16
Method	$\alpha$	Abs.	Norm.	Abs.	Norm.
Linear FT + TIES Yadav et al. (2023)	$1.0$	$77.1$	$87.8$	$80.1$	$88.7$
Linear FT + TIES Yadav et al. (2023)	Best	$77.2$	$87.8$	$80.3$	$88.9$
Linear FT + TSV Gargiulo et al. (2025)	$1.0$	$84.2$	$96.3$	$86.7$	$96.5$
Linear FT + TSV Gargiulo et al. (2025)	Best	$84.2$	$96.3$	$86.8$	$96.6$
Linear FT + ISO Marczak et al. (2025)	$1.0$	$83.8$	$95.8$	$86.9$	$96.7$

Appendix

Table 4: Comparison of different merging strategies in the linear fine-tuning regime, with and without KFAC regularization. Results are reported for

\alpha=1.0

and the best-performing

\alpha

Panel of Figure 11 — Figure 11: For ViT-B/16 (8 Vision), we analyze the sensitivity of different merging strategies to the scaling coefficient $\alpha$ . Left: $\alpha$ -sweep accuracy of post-hoc merging strategies in the non-linear regime, compared with our linearized and regularized models. Right: performance of merging methods on linearized checkpoints.

Panel of Figure 12 — Figure 12: Sensitivity to the scaling coefficient $\alpha$ in the non-linear fine-tuning regime. We report $\alpha$ -sweep results for ViT-B/32 (left) and ViT-B/16 (right), comparing standard non-linear fine-tuning, attention-only fine-tuning Jin et al. (2025) , and its variant regularized with the KFAC.

ViT-B/32
$\lambda$	Seed $7$	Seed $21$	Seed $42$	AVG.
$0$	$75.0$	$75.4$	$75.1$	$75.2\pm 0.028$
$1$	$82.2$	$82.4$	$80.6$	$81.7\pm 0.648$
$10$	$85.2$	$85.1$	$85.1$	$85.1\pm 0.002$
$100$	$86.2$	$85.8$	$86.0$	$86.0\pm 0.026$
$1000$	$86.5$	$86.4$	$86.4$	$86.4\pm 0.002$

Appendix

Table 5: On 8 Vision, ablation of

\lambda

on ViT-B/32 (left) and ViT-B/16 (right). All performances are reported in terms of absolute accuracy using

\alpha=1

Method	Dataless	$\alpha$	ViT-B/32		ViT-B/16
Method	Dataless	$\alpha$	Abs.	Norm.	Abs.	Norm.
Linear FT	–	1.0	$76.7$	$87.2$	$80.2$	$88.9$
Linear FT	–	Best	$78.8$	$89.9$	$82.0$	$90.9$
TAK, Ours	✓	1.0	$\bf{85.8}$	$\bf{97.6}$	$\bf{88.3}$	$\bf{97.9}$
TAK, Ours	✓	Best	$\bf{86.0}$	$\bf{97.8}$	$\bf{88.3}$	$\bf{98.1}$
ImageNet-TAK, Ours	✓	1.0	$84.7$	$97.0$	$86.0$	$95.4$

Appendix

Table 6: Task addition results on the eight vision datasets when using either task-specific KFAC factors or a single shared KFAC computed on ImageNet-1k. Results show that a universal, task-agnostic KFAC (ImageNet-KFAC) retains most of the benefits of our regularizer while requiring no access to auxiliary task-specific data.

Figure 13: Task localization under non-linear fine-tuning . We report the distribution of the Jacobian-projected normalcy scores $\left\lVert\mathrm{J}_{{\bm{\theta}}}f({\bm{x}},{\bm{\theta}}_{0})\,\bm{\tau}_{t}\right\rVert_{2}^{2}$ for inputs belonging to task $t$ (in-task) versus inputs from all other tasks (out-of-task).

Figure 14: Task localization under linearized fine-tuning with block-compressed KFAC. The separation between the two distributions closely matches that of the full KFAC model, indicating that the block-based compression has negligible impact on task localization and that curvature-based task isolation remains robust even under aggressive memory reductions.

Model	ImageNet-R	EUROSAT	RESISC45
Zero-shot	$77.72$	$49.48$	$66.02$
Non-linear FT	$82.32$	$71.21$	$73.85$
Linear FT	$81.66$	$70.40$	$72.28$
Linear FT w. Diag. GN	$81.64$	$73.94$	$74.04$
$\tau$ jP Yoshida et al. (2025)	$81.28$	$84.36$	$84.83$
TAK, Ours (naive penalty)	$82.64$	$79.64$	$78.91$

Appendix

Table 7: Performance comparison across different regularization strategies on ViT-B/16

Method	Dataless	Abs.	Norm.
Individual	–	$85.9$	–
Non-lin. FT	–	$65.5$	$75.9$
Linear FT	–	$76.1$	$92.0$
Attn-Only FT	–	$67.0$	$78.3$
TaLoS	✓	$75.8$	$92.8$
$\tau$ Jp	×	$\bf{81.0}$	$\bf{99.5}$

Appendix

Table 8: Task addition results for T5-base with

\alpha=1

Kronecker-Factored Approximation

See all 14 papers

Understanding and Enforcing Weight Disentanglement in Task Arithmetic
Apr 18, 2026Shangge Liu, Yuehan Yin, Lei Wang +5Task Vectors Weight-Space Analysis
State Key Laboratory for Novel Software Technology, Nanjing University, China · University of Wollongong, Australia · Nanyang Technological University, Singapore
Distilling Linearized Behavior into Non-Linear Fine-Tuning for Effective Task Arithmetic
May 18, 2026Thomas Sommariva, Francesca Morandi, Simone Calderara +1Fine-Tuning Task Arithmetic
ImageLab, Department of Engineering “Enzo Ferrari”, University of Modena and Reggio Emilia, Italy · University of Pisa, Italy
Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmetic
Feb 3, 2025Siqi Zeng, Yifei He, Meitong Liu +5Model Compression Task Vectors
University of Illinois Urbana-Champaign · University of Pennsylvania · Okinawa Institute of Science and Technology

Apr 18, 2026J/K move · Enter open · S save

Understanding and Enforcing Weight Disentanglement in Task Arithmetic

Shangge Liu, Yuehan Yin, Lei Wang, Qi Fan+4

State Key Laboratory for Novel Software Technology, Nanjing University, China · University of Wollongong, Australia · Nanyang Technological University, Singapore

Task Vectors Weight-Space Analysis Task Arithmetic

Open paper AskPDF arXiv

Task arithmetic provides an efficient, training-free way to edit pre-trained models, yet lacks a fundamental theoretical explanation for its success. The existing concept of ``weight disentanglement" describes the ideal outcome of non-interfering task composition but does not reveal its underlying cause. Crucially, what intrinsic properties of the pre-trained model (

θ_0

) or the task vectors (

τ_t

) enable this disentanglement remains underexplored. In this paper, we introduce Task-Feature Specialization (TFS), a model's ability to allocate distinct internal features to different tasks, as the fundamental principle. We first prove that TFS is a sufficient condition for weight disentanglement. More importantly, we find that TFS also gives rise to an observable geometric consequence: weight vector orthogonality. This positions TFS as the common cause for both the desired functional outcome (disentanglement) and a measurable geometric property (orthogonality). This relationship provides the key insight for our method: since the abstract TFS property is intractable to enforce directly, we can instead promote weight disentanglement by shaping its concrete geometric consequence, orthogonality. Therefore, we propose OrthoReg, a simple and effective regularization method that actively enforces an internal orthogonal structure on weight updates (

ΔW

) that constitute

τ_t

during fine-tuning. And we theoretically prove that OrthoReg promotes disentanglement. Extensive experiments demonstrate that OrthoReg consistently and significantly enhances the performance of various task arithmetic methods. Code is available at https://github.com/RL-MIND/OrthoReg.

Abstract

Kronecker-Factored Approximation

Explore similar work

Understanding and Enforcing Weight Disentanglement in Task Arithmetic

Distilling Linearized Behavior into Non-Linear Fine-Tuning for Effective Task Arithmetic

Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmetic