cs.LGJan 24, 2026

Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning

Authors: Fatema Siddika, Md Anwar Hossen, Tanwi Mallick, Ali Jannesari

Organizations: Iowa State University, Ames, USA · Argonne National Laboratory, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typically treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities. We introduce Mixture of Sparse Experts for Task-Agnostic Continual Learning (SETA), a framework that resolves the plasticity-stability conflict through adaptive sparse subspace decomposition into task-specific expert modules. Unlike standard updates, where tasks compete for the same parameters, SETA separates knowledge into unique experts, designed to isolate task-specific patterns, and shared experts, responsible for capturing common features. SETA shares an overlapping parameter block only when the new task's update passes a subspace-agreement test and a sign test against its prior expert; shared knowledge is protected by adaptive elastic anchoring, and independently gated routers retrieve the relevant experts at inference without task identifiers. Extensive experiments across diverse domain-specific benchmarks demonstrate that SETA achieves competitive or superior overall performance relative to state-of-the-art continual learning baselines, improving overall performance over the strongest baseline by 1.1 to 1.7 points on TRACE and by up to 5.5 points on the O-LoRA benchmark, with backward transfer between -2.2 and -1.3 on TRACE, against 4.6 to 6.3 points of forgetting for MoE-CL and 8 to 15 points for the other LoRA baselines.

Figures & tables

Figure 1: Overview of SETA. (a–b) High-utility V-projection blocks are selected from the pretrained LLM by gradient scores. (c) Split-on-Share turns overlapping blocks that pass the direction test into plastic shared experts ( EsE_{\text{s}} ) and keeps the remaining blocks in frozen unique experts ( EuE_{\text{u}} ). (d) The gating network grows with logit invariance, so routing decisions are preserved when an expert splits. (e) At inference, independent sigmoid gates weight all experts for each token, without task identifiers.
 Qwen3-4BLLaMA-2-7BLLaMA-3-8B
MethodOP ↑\uparrowBWT ↑\uparrowOP ↑\uparrowBWT ↑\uparrowOP ↑\uparrowBWT ↑\uparrow
FIX46.55±0.7246.55_{\pm 0.72}–30.29±1.8630.29_{\pm 1.86}–36.50±0.1136.50_{\pm 0.11}–
SeqLoRA45.71±3.2945.71_{\pm 3.29}−7.97±1.83-7.97_{\pm 1.83}41.97±0.8441.97_{\pm 0.84}−9.72±0.13-9.72_{\pm 0.13}46.92±2.6646.92_{\pm 2.66}−12.32±3.12-12.32_{\pm 3.12}
O-LoRA45.13±1.9445.13_{\pm 1.94}−9.93±1.06-9.93_{\pm 1.06}40.32±1.9340.32_{\pm 1.93}−11.76±2.11-11.76_{\pm 2.11}49.54±0.9649.54_{\pm 0.96}−9.14±2.55-9.14_{\pm 2.55}
HiDe-LoRA43.73±0.7743.73_{\pm 0.77}−9.35±0.81-9.35_{\pm 0.81}38.42±2.8338.42_{\pm 2.83}−14.87±3.15-14.87_{\pm 3.15}50.51±2.9050.51_{\pm 2.90}−8.46±4.18-8.46_{\pm 4.18}
TreeLoRA46.07±3.1446.07_{\pm 3.14}−8.50±2.24-8.50_{\pm 2.24}39.30±2.4439.30_{\pm 2.44}−12.69±2.21-12.69_{\pm 2.21}47.58±1.3547.58_{\pm 1.35}−11.52±2.12-11.52_{\pm 2.12}
Table 1: Comparison of continual learning methods on TRACE-8 with three backbones. We report OP and BWT (%) after training on the full sequence as mean ±std{}_{\pm\text{std}} over two seeds. FIX prompts the frozen backbone with in-context examples, so it has no BWT. SETA uses β\beta =12%. Bold marks the best mean in each column.
Figure 2: Final performance (OP) against backward transfer (BWT) on TRACE-8 for three backbones. Points are means over seeds 100 and 42, and error bars show one standard deviation on each axis. The horizontal line marks zero forgetting, so methods toward the upper right both learn more and forget less. SETA is shown at block budgets of 12% and 25%.
 SeqLoRAO-LoRAHiDe-LoRATreeLoRAMoE-CLSETA
OrderOPBWTOPBWTOPBWTOPBWTOPBWTOPBWT
O170.00±0.0070.00_{\pm 0.00}−6.67±1.41-6.67_{\pm 1.41}70.00±1.0670.00_{\pm 1.06}−7.00±0.00-7.00_{\pm 0.00}69.50±0.3569.50_{\pm 0.35}−5.83±0.24-5.83_{\pm 0.24}69.50±0.3569.50_{\pm 0.35}−7.50±1.18-7.50_{\pm 1.18}70.00±1.7770.00_{\pm 1.77}−7.33±2.83-7.33_{\pm 2.83}75.50±1.06\mathbf{75.50}_{\pm 1.06}−2.00±0.94\mathbf{-2.00}_{\pm 0.94}
O271.38±1.2471.38_{\pm 1.24}−6.67±0.47-6.67_{\pm 0.47}71.00±0.7171.00_{\pm 0.71}−6.67±2.36-6.67_{\pm 2.36}69.75±0.0069.75_{\pm 0.00}−6.00±0.47-6.00_{\pm 0.47}71.50±1.0671.50_{\pm 1.06}−6.00±1.89-6.00_{\pm 1.89}68.12±1.5968.12_{\pm 1.59}−9.33±1.41-9.33_{\pm 1.41}77.00±0.35\mathbf{77.00}_{\pm 0.35}−1.33±0.47\mathbf{-1.33}_{\pm 0.47}
O374.00±0.3574.00_{\pm 0.35}−4.00±0.47-4.00_{\pm 0.47}73.38±0.1873.38_{\pm 0.18}−5.00±0.00-5.00_{\pm 0.00}72.50±0.3572.50_{\pm 0.35}−4.17±1.65-4.17_{\pm 1.65}73.50±0.3573.50_{\pm 0.35}−4.33±0.00-4.33_{\pm 0.00}73.88±0.1873.88_{\pm 0.18}−2.33±0.00-2.33_{\pm 0.00}74.00±1.06\mathbf{74.00}_{\pm 1.06}−1.67±0.47\mathbf{-1.67}_{\pm 0.47}
O476.15±0.2076.15_{\pm 0.20}−4.06±0.26-4.06_{\pm 0.26}75.87±1.2375.87_{\pm 1.23}−4.35±1.27-4.35_{\pm 1.27}74.42±2.1974.42_{\pm 2.19}−5.33±2.05-5.33_{\pm 2.05}76.31±0.0276.31_{\pm 0.02}−3.84±0.02-3.84_{\pm 0.02}76.33±0.2876.33_{\pm 0.28}−3.42±0.15\mathbf{-3.42}_{\pm 0.15}77.14±2.06\mathbf{77.14}_{\pm 2.06}−4.16±1.86-4.16_{\pm 1.86}
O579.17±0.3379.17_{\pm 0.33}−1.49±0.57-1.49_{\pm 0.57}79.44±0.2679.44_{\pm 0.26}−1.02±0.45-1.02_{\pm 0.45}79.36±0.1679.36_{\pm 0.16}−0.41±0.03\mathbf{-0.41}_{\pm 0.03}78.65±0.0578.65_{\pm 0.05}−2.00±0.38-2.00_{\pm 0.38}78.01±0.7578.01_{\pm 0.75}−1.94±0.24-1.94_{\pm 0.24}80.06±2.89\mathbf{80.06}_{\pm 2.89}−0.80±0.42-0.80_{\pm 0.42}
Table 2: Comparison of continual learning methods on the O-LoRA benchmark with Qwen3-4B. O1–O3 are orders of the four standard text-classification tasks and O4–O6 orders of the fifteen-task long sequence, following Wang et al. (2023a) and Razdaibiedina et al. (2023) . We report OP and BWT (%) after the final task as mean ±std{}_{\pm\text{std}} over two seeds. Bold marks the best mean for each metric in each row.
Figure 3: Expert structure of SETA along the TRACE-8 task sequence at block budgets of 12% (solid) and 25% (dashed). (a) LLaMA-2-7B and (b) Qwen3-4B: distinct V-projection block positions used by any expert, against the count if every task took fresh blocks (dotted); the shaded gap is the number of positions reused by later tasks (reuse saves positions, not stored parameters; Appendix D.1 ). (c) Number of experts and (d) mean expert size in blocks for all three backbones; at 25% each task adds exactly one expert.
Routingβ=12%\beta=12\%β=25%\beta=25\%
Task-Agnostic49.1649.60
Oracle49.2151.09
Top-132.5334.20
Top-227.6937.83
Top-332.6540.00
Table 3: Routing rules at inference on TRACE-8 with Qwen3-4B (seed 100, OP %). Each final checkpoint is evaluated with SETA’s own gates, oracle routing by task identity, or only the top- kk gates. Keeping only the top- kk gates lowers OP at both budgets, so predictions draw on several experts at once.
Testφ\varphiPassedSharedUndone (%)Cos << .02 (%)OPBWT
Strict0.006144646867.60.0 ∗34.69−- 6.29
Normal0.0063226107466.731.234.74−- 5.56
Relaxed0.0063851118269.337.734.65−- 6.66
Default0.012235817992.433.037.15−- 4.95
Table 4: Effect of the direction test on TRACE-8 with Qwen3-4B (reduced protocol, seed 100). Strict uses ZZ =3, τ\tau =+0.02; normal ZZ =2, τ\tau =0; relaxed ZZ =1.5, τ\tau = −- 0.02; the default ZZ =3, τ\tau =0. Passed and Shared count blocks over tasks 2–8 that pass the test and that stay shared after the size rule; Undone is the share of passed blocks the size rule removes, and Cos << .02 the share with gradient cosine below 0.02.
λbase\lambda_{\text{base}}LAOPBWTΔ\Delta OP
0 (no anchoring)40.5634.34−- 7.11−- 2.81
0.0540.6034.96−- 6.44−- 2.19
0.1 (default)41.4837.15−- 4.95 
0.242.2436.79−- 6.22−- 0.36
0.542.3535.31−- 8.05−- 1.84
Table 5: Sensitivity to the anchoring strength λbase\lambda_{\text{base}} on TRACE-8 with Qwen3-4B, under the reduced protocol of Table 7 . All runs use the sign test and differ only in λbase\lambda_{\text{base}} ; Δ\Delta OP is the change from the default, and λbase\lambda_{\text{base}} =0 turns anchoring off. The default gives both the highest OP and the least forgetting.
VariantLA ↑\uparrowOP ↑\uparrowBWT ↑\uparrowΔ\Delta OP
SETA ( β\beta =12%, V)50.5349.16−- 1.56 
Q projection40.9334.90−- 6.89−- 14.26
MLP projections42.4435.82−- 7.56−- 13.34
No shared experts52.3551.70−- 0.85+2.54
Table 6: Ablations of SETA on TRACE-8 with Qwen3-4B under the full protocol of Table 1 (5,000 training samples per task, seed 100), changing the adapted projection, the shared experts. Δ\Delta OP is the change from the first row.
VariantLA ↑\uparrowOP ↑\uparrowBWT ↑\uparrowΔ\Delta OP
SETA, all components41.4837.15−- 4.95 
Sign test off40.9634.77−- 7.07−- 2.38
No direction test (positional split)39.4635.37−- 4.67−- 1.78
No projection of own copies40.9737.51−- 3.96+0.36
Earlier gate rows trainable40.9335.89−- 5.76−- 1.26
No elastic anchoring ( λbase\lambda_{\text{base}} =0)40.5634.34−- 7.11−- 2.81
Table 7: Component ablations of SETA on TRACE-8 with Qwen3-4B under a reduced protocol (500 training samples per task, fixed warm-up, seed 100). Each row changes one setting of the full system, which here freezes the gate rows of earlier experts. Δ\Delta OP is relative to the first row. Scores are not comparable with Table 6 .
Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

 SettingSplit structure (tasks 2–8)Performance
ArmZZτ\tauφ\varphiPassedSharedUndone (%)cos << 0.02 (%)ExpertsLAOPBWT
Strict3.0+0.02+0.020.006144646867.60.0 ∗2640.1934.69−6.29-6.29
Normal2.00.000.000.0063226107466.731.24139.6034.74−5.56-5.56
Relaxed1.5−0.02-0.020.0063851118269.337.74340.4734.65−6.66-6.66
Default3.00.000.000.012235817992.433.01741.4837.15−4.95-4.95
Appendix
Table 8: Effect of the direction test’s strictness on Qwen3-4B (TRACE-8, reduced protocol with the sign test on). ZZ is the agreement bar, τ\tau the minimum gradient cosine for sharing and φ\varphi the minimum expert size as a fraction of the block budget. Split structure counts the blocks that pass the direction test, those kept as shared after the size rule, the share of passing blocks undone by that rule, the share of passing blocks with a gradient cosine below 0.02, and the experts after the last task. A looser test passes and shares more blocks, but the size rule undoes most of them and performance barely changes among the three arms at φ\varphi =0.006. ∗ Zero by construction.
MethodC-STANCEFOMCMeetingB.Py150SciQANG-cmNG-ds20Min.OP ↑\uparrowBWT ↑\uparrow
LLaMA-2-7B
FIX        30.29±1.8630.29_{\pm 1.86}–
SeqLoRA32.8±2.532.8_{\pm 2.5}46.9±3.646.9_{\pm 3.6}35.9±0.535.9_{\pm 0.5}42.0±3.242.0_{\pm 3.2}54.5±7.254.5_{\pm 7.2}37.0±3.537.0_{\pm 3.5}46.2±0.446.2_{\pm 0.4}40.4±0.840.4_{\pm 0.8}41.97±0.8441.97_{\pm 0.84}−9.72±0.13-9.72_{\pm 0.13}
after own task51.2±1.151.2_{\pm 1.1}65.7±0.665.7_{\pm 0.6}39.2±0.339.2_{\pm 0.3}60.9±0.160.9_{\pm 0.1}58.7±1.658.7_{\pm 1.6}37.0±3.537.0_{\pm 3.5}50.6±0.750.6_{\pm 0.7}40.4±0.840.4_{\pm 0.8}  
O-LoRA22.3±17.422.3_{\pm 17.4}43.6±0.743.6_{\pm 0.7}36.8±0.036.8_{\pm 0.0}42.2±11.042.2_{\pm 11.0}52.6±5.452.6_{\pm 5.4}35.8±5.235.8_{\pm 5.2}48.5±0.748.5_{\pm 0.7}40.7±0.240.7_{\pm 0.2}40.32±1.9340.32_{\pm 1.93}−11.76±2.11-11.76_{\pm 2.11}
after own task51.2±0.851.2_{\pm 0.8}65.9±1.165.9_{\pm 1.1}39.4±0.239.4_{\pm 0.2}60.9±0.160.9_{\pm 0.1}59.6±2.059.6_{\pm 2.0}38.3±3.538.3_{\pm 3.5}48.8±0.248.8_{\pm 0.2}40.7±0.240.7_{\pm 0.2}  
Appendix
Table 9: Per-task scores (%) on TRACE-8 in training order. For each trained method, the first row gives the score on each task after the last task (its mean is OP) and the second row the score right after that task was learned (its mean is LA), so comparing the two rows shows where each method forgets. All entries are mean ±std{}_{\pm\text{std}} over two seeds (FIX runs on the LLaMA backbones have no per-task scores).
 C-STFOMCMeetBPy150SciQANG-cmNG-ds20Min
β\beta =12%
Qwen3-4B, experts1345691013
shared blocks formed05500022031
layers kept (of 36)3433343434303331
trainable params (%)0.280.280.280.280.280.270.280.28
LLaMA-2-7B, experts13579111619
Appendix
Table 10: Expert structure of SETA after each task on TRACE-8 for three backbones at block budgets at β=12%\beta=12\% , read from the split reports and training logs. Experts counts all experts after the task’s split; shared blocks formed counts the blocks moved into new shared experts; layers kept counts the layers that retain the task’s blocks after the minimum-size rule; trainable parameters are the experts updated for the task, as a share of the backbone. At the larger budget no shared expert forms, so each task adds one isolated expert (Figure 3 c).
Figure 4: How SETA’s experts grow on TRACE-8. (a) Number of experts after each task’s split; at the larger budget every task adds exactly one unique expert. (b) Blocks held by shared experts at β\beta =12% (none form at 25%). (c) Size of every expert after the last task at β\beta =12% (log scale), unique (filled) against shared (hollow). (d) Share of the backbone’s layers that keep the task’s blocks after the minimum-size rule. Shared experts stay small, so most of each task’s update remains in its own frozen expert.
Figure 5: Layer-wise share of top-ranked gradients in the Q, K and V projections. Nearly all high-magnitude gradients fall in the Value projection.
Figure 6: Selected blocks per layer and task on TRACE-8 for Qwen3-4B (left), LLaMA-2-7B (middle) and LLaMA-3-8B (right); darker cells mark more blocks. All three backbones spread their selection over every layer with a task-dependent profile.
Figure 7: Rate of novel blocks during selection for Qwen3-4B (left), LLaMA-2-7B (middle) and LLaMA-3-8B (right): the fraction of the first nn selected blocks that no earlier task selected. The rate falls over the sequence as later tasks reuse blocks selected before them.
Figure 8: Distinct V-projection blocks adapted by any task so far on TRACE-8 ( β\beta =12%) for Qwen3-4B (left), LLaMA-2-7B (middle) and LLaMA-3-8B (right), against the count if every task took blocks no earlier task had selected (dashed). Reuse keeps the adapted set growing more slowly than the number of tasks.
Figure 9: Blocks per layer and task that no earlier task selected at the same position, for Qwen3-4B (left), LLaMA-2-7B (middle) and LLaMA-3-8B (right); darker cells mark more new coverage. Later tasks add less new coverage as more layers are already covered.
SettingLAOPBWTΔ\Delta OP
Full system41.7334.29−- 8.50 
ZZ =139.9834.78−- 5.95+0.49
ZZ =240.7937.17−- 4.14+2.87
ZZ =442.2337.75−- 5.12+3.46
φ\varphi =0.00641.7635.91−- 6.69+1.61
φ\varphi =0.02539.4433.64−- 6.62−- 0.65
Appendix
Table 11: Sensitivity to the split-rule settings on TRACE-8 with Qwen3-4B under the reduced protocol. Each row changes one setting from its default: the agreement bar ZZ , the size floor φ\varphi or the kept fraction share q of a shared piece. Δ\Delta OP is the change from the full-system run of the same round (first row), whose sign-test setting was not recorded. Only a size floor larger than a typical layer’s selection breaks the method. † Run with the sign test, so no Δ\Delta OP is given. The share q =0.5 run lacks one final score, so only its LA is reported.
Figure 10: Sensitivity to one setting at a time on TRACE-8 with Qwen3-4B under the reduced protocol: OP (top) and BWT (bottom) against the agreement bar ZZ , the size floor φ\varphi and the anchoring strength λbase\lambda_{\text{base}} . Blue marks the default and the dashed line its value. The ZZ and φ\varphi runs share the full-system run of Table 11 , and the λbase\lambda_{\text{base}} runs that of Table 7 ; share q is not drawn. Only a size floor that is too large breaks the method.
SettingSETALoRA baselines
BackbonesLLaMA-2-7B, LLaMA-3-8B, Qwen3-4B; the same local weights for all methods
PromptInstruction prompt of Appendix G , without the model’s chat template
Training data5,000 samples per task.
EvaluationFirst 500 test samples per task.
Max. sequence length512 tokens; MeetingBank 2,048 and 20Minuten 768 (1,536 and 512 on LLaMA-3-8B)
Learning rate9.65×10−69.65\times 10^{-6}9.65×10−69.65\times 10^{-6}
Appendix
Table 12: Training settings of SETA and the LoRA baselines (TreeLoRA code base). All methods train on the same task streams and data and use the same prompt format.
More in this topic

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 223 papers

Explore similar work

CardsList

  1. Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning

    Jun 5, 2026Fatema Siddika, Md Anwar Hossen, Tanwi Mallick +1Continual Learning for LLMsMixture-of-Experts Language Models

    Iowa State University, Ames, USA · Argonne National Laboratory, USA

  2. Mixtures of SubExperts for Large Language Continual Learning

    Nov 9, 2025Haeyong Kang, Hee Suk Yoon, Dahua Feng +1Continual Learning for LLMsMixture-of-Experts Language Models

    Korea Advanced Institute of Science and Technology (KAIST)

  3. CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning

    May 18, 2026Yang Liu, Toan Nguyen, Flora D. SalimContinual Learning for LLMsVision-Language Models

    School of Computer Science and Engineering University of New South Wales Australia