When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging

Shangge Liu; Yuehan Yin; Yinghuan Shi; Lei Wang; Wenbin Li

cs.LGAug 27, 2026

When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging

Authors: Shangge Liu, Yuehan Yin, Yinghuan Shi, Lei Wang, Wenbin Li

Organizations: State Key Laboratory for Novel Software Technology, Nanjing University · University of Wollongong

Abstract

Continual learning (CL) and model merging (MM) both aim to obtain a single model that performs well across multiple tasks, challenged respectively by catastrophic forgetting and weight-disentanglement error. In the literature, these difficulties are merely treated separately and mitigated through a variety of solutions, while the geometry induced by the base optimizer is treated as an implementation detail. In this work, we show that the two difficulties are in fact two instances of the same phenomenon: a parameter update useful for one task shifts the model's outputs on another. We formalize this shared phenomenon as task interference and reduce it to a common layer-wise Frobenius inner product $\langle ΔW_\ell, J_\ell(x)\rangle_F$ . This quantity, in turn, is utilized to expose the role of the optimizer. We theoretically derive an upper bound that isolates the spectral norm $\|ΔW_\ell\|_2$ as an optimizer-controllable factor of task interference, and a per-mode analysis shows that this bound tracks the dominant part of the empirical interference. Specifically, we then identify the recent Muon optimizer as a mechanism that regulates this factor by construction. Our work reveals that its elegant control on spectral norm tightens the interference bound for both CL and MM, positioning Muon as a principled optimizer-centric approach complementary to existing solutions. Our theoretcal analysis is well validated by experimental results. Replacing the AdamW optimizer with Muon improves accuracy by up to +5.02 points on the eight-task model-merging benchmark across three CLIP backbones. For continual learning, Muon also delivers uniformly positive gains across ten class-incremental protocols, three task-incremental protocols, and the 11-task MTIL benchmark.

Figures & tables

Figure 1: Overview of this study. We reinterpret the Muon optimizer through the lens of continual learning and model merging, and provide a unified theoretical analysis together with empirical validation on both paradigms.

Figure 2: A unified spectral framework for CL and MM. Left: Both paradigms target a shared multi-task optimum $\theta^{\star}$ through different protocols. Middle: A first-order Taylor expansion unifies catastrophic forgetting (CL) and disentanglement error (MM) as the same layer-wise term $\langle\Delta W_{\ell},J_{\ell}(x)\rangle_{F}$ , which Theorem 1 bounds by $\|J_{\ell}(x)\|_{*}\cdot\|\Delta W_{\ell}\|_{2}$ and isolates the spectral norm as an optimizer-controllable factor. Right: Theorem 2 identifies Muon as the closed-form spectral-norm steepest descent, which uniformizes the update spectrum and thus tightens the interference bound for both paradigms.

Figure 3: Empirical validation of Remark 1 on ViT-B/32. We plot the per-mode contribution of the SVD modes of

\Delta W_{\ell}

. (a) Cumulative contribution of the top-10% singular modes; (b) per-mode contribution on a log scale; (c) full cumulative curve across all modes. Under AdamW, the top-10% of modes account for

63.5\%

of the total interference, confirming that interference is sharply concentrated on a few dominant modes. Under Muon, the same fraction is

29.0\%

and the curve is nearly flat. This validates Remark 1 : shrinking

\|\Delta W_{\ell}\|_{2}

suppresses precisely the modes that dominate the true interference, so the bound of Theorem 1 tracks the actual quantity rather than merely upper-bounding it.

Figure 4: Per-mode contribution of the MoE-Adapter updates

\Delta W

along the MTIL sequence. (a)

\mathrm{Contrib}_{i}

vs. singular rank

i

on a log scale. (b) Cumulative contribution over the top-

k

modes. The CL counterpart of Figure 3 .

	ViT-B/32, 8 tasks		ViT-B/16, 8 tasks		ViT-L/14, 8 tasks
Method	Abs.Acc. $\uparrow$	Norm.Acc. $\uparrow$	Abs.Acc. $\uparrow$	Norm.Acc. $\uparrow$	Abs.Acc. $\uparrow$	Norm.Acc. $\uparrow$
Zero-shot	47.74	–	54.22	–	64.54	–
Non-linear FT + AdamW	70.32	77.56	75.39	75.39	84.07	89.19
Non-linear FT + Muon	75.34	82.79	78.81	84.64	87.89	93.26
Gain	+5.02	+5.23	+3.42	+9.25	+3.82	+4.07
TTA + AdamW	74.68	85.27	78.97	87.48	86.19	93.14

Table 1: Eight-task addition on CLIP backbones. Absolute Accuracy (Abs.Acc.) and Normalized Accuracy (Norm.Acc.) are reported. “+ Muon” denotes optimizing all matrix-valued parameters with Muon and the remaining parameters with auxiliary AdamW. Across all three backbones, replacing AdamW with Muon improves both metrics under every pre-merging strategy.

Figure 5: Weight-disentanglement heatmap for Cars + RESISC45. Each cell reports the prediction disagreement (%) and lower is better. (a) AdamW: low-disagreement region narrowly confined near the origin. (b) Muon: the same region expands in both axes. This matches the reduction in interfence predicted by Theorem 1 .

Figure 6: Per-task accuracy on ViT-B/32 under AdamW and Muon. (a, b) Grey dashed: zero-shot backbone. Light dashed: accuracy of each independently fine-tuned expert on its own task. Bold solid: accuracy of the merged model. (c) Direct comparison of the two merged models. Single-task accuracies are nearly identical between optimizers, but the merged accuracies expand substantially under Muon.

Figure 7: Relative interference

R_{j\to\text{SVHN}}=I_{j\to\text{SVHN}}/S_{j}

on ViT-B/32 for each source task vector

j

. Lower is better. Muon consistently reduces relative interference.

Figure 8: Stable rank of $\Delta W$ on CLIP ViT-B/32 under AdamW and Muon. (a) Per-dataset mean stable rank. Muon increases the mean by roughly $11\times$ on every task. (b, c) Layer-wise stable rank for every matrix type of all blocks, under AdamW and Muon respectively.

			AdamW		Muon		Gain
	B	Tasks	Avg. $\uparrow$	Last $\uparrow$	Avg. $\uparrow$	Last $\uparrow$	Avg.	Last
CIL	I	10	79.43	71.22	81.55	75.12	+2.12	+3.90
	I	20	71.04	64.92	75.19	68.43	+4.15	+3.51
	C	50	62.04	47.87	64.39	51.01	+2.35	+3.14
TIL	C	5	93.92	91.81	95.61	93.45	+1.69	+1.64
	C	10	91.68	90.15	95.07	91.94	+3.39	+1.79

Table 2: LoRA baseline with AdamW vs. Muon on CIL and TIL protocols. Muon uniformly improves both metrics. B (benchmark): I = ImageNet-R, C = CIFAR-100.

Benchmark	Protocol	AdamW		Muon
Benchmark	Protocol	Avg. $\uparrow$	Last $\uparrow$	Avg. $\uparrow$	Last $\uparrow$
CIFAR-100	10 tasks	85.30	77.56	85.98	78.82
	20 tasks	84.36	76.61	84.87	77.27
	50 tasks	83.39	73.76	83.84	74.02
TinyImageNet	5 steps	80.85	77.04	81.62	77.57
	10 steps	80.26	75.79	81.14	76.75

Table 3: CIL results on MoE-Adapters4CL. Replacing AdamW with Muon improves both Avg. and Last across all seven protocols.

Method	Transfer $\uparrow$	Average $\uparrow$	Last $\uparrow$
AdamW	66.4	76.9	86.3
Muon	67.1	78.3	87.9

Table 4: 11-task MTIL evaluation on MoE-Adapters4CL. Replacing AdamW with Muon improves Transfer, Average, and Last by

+0.7

+1.4

, and

+1.6

points, respectively. Complete per-task results are provided in Appendix Table.

Figure 9: Mean stable rank of

\Delta W

on the MTIL sequence. AdamW:

[2.3,4.4]

across all datasets; Muon:

[27.0,36.8]

, roughly

10\times

higher and uniform across datasets. Higher stable rank at fixed Frobenius scale forces a smaller spectral norm and a tighter Theorem 1 bound at every transition.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Per-task accuracy of the merged model on ViT-B/32 (left) and ViT-L/14 (right).

Figure 11: Sensitivity of the merged model to the scaling coefficient

\alpha

on ViT-L/14. Muon reaches a higher peak than AdamW and, more importantly, degrades much more gracefully as

\alpha

moves away from the optimum.

LoRA4TIL
	5 tasks		10 tasks		20 tasks
Method	Avg.	Last	Avg.	Last	Avg.	Last
AdamW	93.92	91.81	91.68	90.15	91.81	90.44
Muon	95.61	93.45	95.07	91.94	96.07	93.64

Appendix

Table 5: CIFAR-100 (5/10/20-task protocols) + Lora for TIL.

LoRA4TIL
	ImageNet-R				TinyImageNet
	10 tasks		20 tasks		20 tasks
Method	Avg.	Last	Avg.	Last	Avg.	Last
AdamW	87.97	87.87	90.33	89.18	91.60	87.88
Muon	91.17	88.48	91.85	88.45	91.75	88.04

Appendix

Table 6: ImageNet-R (10/20-task) and TinyImageNet (20-task) + LoRA for TIL.

LoRA4CIL
	10 tasks		50 tasks
Method	Avg.	Last	Avg.	Last
AdamW	77.56	66.56	62.04	47.87
Muon	78.73	67.69	64.39	51.01

Appendix

Table 7: CIFAR-100 (10/50-task protocols) + Lora for CIL.

	10 tasks		20 tasks		50 tasks
Method	Avg.	Last	Avg.	Last	Avg.	Last
UCIR ( Hou et al. 2019 )	58.66	43.39	58.17	40.63	56.86	37.09
BiC ( Wu et al. 2019 )	68.80	53.54	66.48	47.02	62.09	41.04
PODNet ( Douillard et al. 2020 )	58.03	41.05	53.97	35.02	51.19	32.99
DER ( Yan et al. 2021 )	74.64	64.35	73.98	62.55	72.05	59.76
DyTox+ ( Douillard et al. 2022 )	74.10	62.34	71.62	57.43	68.90	51.09

Appendix

Table 8: class-incremental CIFAR-100 + MoE-Adapter for CL.

LoRA4CIL
	10 tasks		20 tasks
Method	Avg.	Last	Avg.	Last
AdamW	79.43	71.22	71.04	64.92
Muon	81.55	75.12	75.19	68.43

Appendix

Table 9: ImageNet-R (10/20-task protocols) + Lora for CIL.

LoRA4CIL
	10 tasks		20 tasks
Method	Avg.	Last	Avg.	Last
AdamW	74.81	64.44	67.44	56.31
Muon	75.00	64.93	68.72	57.65

Appendix

Table 10: TinyImageNet (10/20-task protocols) + Lora for CIL.

	5 steps		10 steps		20 steps
Method	Avg.	Last	Avg.	Last	Avg.	Last
EWC ( Kirkpatrick et al. 2016 )	19.01	6.00	15.82	3.79	12.35	4.73
EEIL ( Castro et al. 2018 )	47.17	35.12	45.03	34.64	40.41	29.72
UCIR ( Hou et al. 2019 )	50.30	39.42	48.58	37.29	42.84	30.85
MUC ( Liu et al. 2020b )	32.23	19.20	26.67	15.33	21.89	10.32
PASS ( Zhu et al. 2021 )	49.54	41.64	47.19	39.27	42.01	32.93

Appendix

Table 11: TinyImageNet + MoE-Adapter for CL (5/10/20-step protocols , each with a 100-class base task).

Metric	Method	Aircraft	Caltech101	CIFAR-100	DTD	EuroSAT	Flowers	Food-101	MNIST	OxfordPet	Cars	SUN397	Mean
Transfer	Continual-FT	–	67.1	46.0	32.1	35.6	35.0	57.7	44.1	60.8	20.5	46.6	44.6
	LwF ( Li and Hoiem 2018 )	–	74.5	56.9	39.1	51.1	52.6	72.8	60.6	75.1	30.3	55.9	58.9
	iCaRL ( Rebuffi et al. 2017 )	–	56.6	44.6	32.7	39.3	46.6	68.0	46.0	77.4	31.9	60.5	50.4
	LwF-VR ( Ding et al. 2022 )	–	77.1	61.0	40.5	45.3	54.4	74.6	47.9	76.7	36.3	58.6	57.2
	WiSE-FT ( Wortsman et al. 2022b )	–	73.5	55.6	35.6	41.5	47.0	68.3	53.9	69.3	26.8	51.9	52.3
	ZSCL ( Zheng et al. 2023 )	–	86.0	67.4	45.4	50.4	69.1	87.6	61.8	86.8	60.1	66.8	68.1

Appendix

Table 12: Full 11-task MTIL evaluation on the benchmark of ( Zheng et al. 2023 ) . We report Transfer, Average, and Last. Relative to AdamW, Muon improves the mean Transfer by

+0.6

, Average by

+1.4

, and Last by

+1.6

points.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 294 papers

Muon Can Outperform Dedicated Continual Learning Methods
Sep 21, 2026Sebastian George Sincari, Bogdan Alexandru Gheorghe, Antonio BarbalauContinual Learning Muon Optimizer
Faculty of Mathematics and Computer Science, University of Bucharest, Bucharest, Romania · Bitdefender, Bucharest, Romania
Revitalizing the Beginning: Avoiding Storage Dependency for Model Merging in Continual Learning
May 8, 2026Xi Wang, Cheng DengStorage-Efficient Continual Learning Continual Learning
School of Electronic Engineering, Xidian University, Xi’an, Shaanxi, China.
PMF-CL: Pareto-Minimal-Forgetting Continual Learner for Conflicting Tasks
May 18, 2026Srijith Nair, Atilla Eryilmaz, Jia LiuContinual Learning Catastrophic Forgetting
Department of Electrical and Computer Engineering The Ohio State University Columbus, OH 43201

Sep 21, 2026J/K move · Enter open · S save

Muon Can Outperform Dedicated Continual Learning Methods

Sebastian George Sincari, Bogdan Alexandru Gheorghe, Antonio Barbalau

Faculty of Mathematics and Computer Science, University of Bucharest, Bucharest, Romania · Bitdefender, Bucharest, Romania

Continual Learning Muon Optimizer Low-Rank Adaptation Incremental Learning Continual LoRA

Open paper AskPDF arXiv

Continual learning with Low-Rank Adapters (LoRA) typically mitigates forgetting by penalizing the overlap between a new update and the accumulated past weights, which discourages certain update directions without controlling how an update distributes its energy over the ones that remain. We ask whether that restriction has to be task-aware, or whether a generic one supplied by the optimizer is enough. We train a plain incremental LoRA (IncLoRA) with Muon, which orthogonalizes each update, and compare it against O-LoRA and ELLA over five seeds and three task orders on the Standard CL Benchmark and three seeds on TRACE. IncLoRA+Muon reaches the accuracy band of the dedicated methods on Standard CL and improves on every AdamW configuration on TRACE. One update-constraining mechanism is enough, whether it comes from the loss or from the optimizer; on Standard CL a second one does not help, and for the most restrictive method it costs 8.4 points of accuracy and the plasticity to fit each task. What separates the two optimizers is not the size of the update, which under Muon is 0.91 to 2.06 times that under AdamW, but how it is distributed. AdamW confines it to between 1.4 and 1.8 effective singular directions, Muon spreads it over 7.0, and the two do not overlap in any tracked run. Part of the advantage usually attributed to dedicated CL methods may therefore be explained by the geometry of the optimizer's updates.

Abstract

Continual Learning

Explore similar work

Muon Can Outperform Dedicated Continual Learning Methods

Revitalizing the Beginning: Avoiding Storage Dependency for Model Merging in Continual Learning

PMF-CL: Pareto-Minimal-Forgetting Continual Learner for Conflicting Tasks