Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning

Fatema Siddika; Md Anwar Hossen; Tanwi Mallick; Ali Jannesari

cs.LGJan 24, 2026

Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning

Authors: Fatema Siddika, Md Anwar Hossen, Tanwi Mallick, Ali Jannesari

Organizations: Iowa State University, Ames, USA · Argonne National Laboratory, USA

Abstract

Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typically treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities. We introduce Mixture of Sparse Experts for Task-Agnostic Continual Learning (SETA), a framework that resolves the plasticity-stability conflict through adaptive sparse subspace decomposition into task-specific expert modules. Unlike standard updates, where tasks compete for the same parameters, SETA separates knowledge into unique experts, designed to isolate task-specific patterns, and shared experts, responsible for capturing common features. SETA shares an overlapping parameter block only when the new task's update passes a subspace-agreement test and a sign test against its prior expert; shared knowledge is protected by adaptive elastic anchoring, and independently gated routers retrieve the relevant experts at inference without task identifiers. Extensive experiments across diverse domain-specific benchmarks demonstrate that SETA achieves competitive or superior overall performance relative to state-of-the-art continual learning baselines, improving overall performance over the strongest baseline by 1.1 to 1.7 points on TRACE and by up to 5.5 points on the O-LoRA benchmark, with backward transfer between -2.2 and -1.3 on TRACE, against 4.6 to 6.3 points of forgetting for MoE-CL and 8 to 15 points for the other LoRA baselines.

Figures & tables

Figure 1: Overview of SETA. (a–b) High-utility V-projection blocks are selected from the pretrained LLM by gradient scores. (c) Split-on-Share turns overlapping blocks that pass the direction test into plastic shared experts ( $E_{\text{s}}$ ) and keeps the remaining blocks in frozen unique experts ( $E_{\text{u}}$ ). (d) The gating network grows with logit invariance, so routing decisions are preserved when an expert splits. (e) At inference, independent sigmoid gates weight all experts for each token, without task identifiers.

	Qwen3-4B		LLaMA-2-7B		LLaMA-3-8B
Method	OP $\uparrow$	BWT $\uparrow$	OP $\uparrow$	BWT $\uparrow$	OP $\uparrow$	BWT $\uparrow$
FIX	$46.55_{\pm 0.72}$	–	$30.29_{\pm 1.86}$	–	$36.50_{\pm 0.11}$	–
SeqLoRA	$45.71_{\pm 3.29}$	$-7.97_{\pm 1.83}$	$41.97_{\pm 0.84}$	$-9.72_{\pm 0.13}$	$46.92_{\pm 2.66}$	$-12.32_{\pm 3.12}$
O-LoRA	$45.13_{\pm 1.94}$	$-9.93_{\pm 1.06}$	$40.32_{\pm 1.93}$	$-11.76_{\pm 2.11}$	$49.54_{\pm 0.96}$	$-9.14_{\pm 2.55}$
HiDe-LoRA	$43.73_{\pm 0.77}$	$-9.35_{\pm 0.81}$	$38.42_{\pm 2.83}$	$-14.87_{\pm 3.15}$	$50.51_{\pm 2.90}$	$-8.46_{\pm 4.18}$
TreeLoRA	$46.07_{\pm 3.14}$	$-8.50_{\pm 2.24}$	$39.30_{\pm 2.44}$	$-12.69_{\pm 2.21}$	$47.58_{\pm 1.35}$	$-11.52_{\pm 2.12}$

Table 1: Comparison of continual learning methods on TRACE-8 with three backbones. We report OP and BWT (%) after training on the full sequence as mean

{}_{\pm\text{std}}

over two seeds. FIX prompts the frozen backbone with in-context examples, so it has no BWT. SETA uses

\beta

=12%. Bold marks the best mean in each column.

Figure 2: Final performance (OP) against backward transfer (BWT) on TRACE-8 for three backbones. Points are means over seeds 100 and 42, and error bars show one standard deviation on each axis. The horizontal line marks zero forgetting, so methods toward the upper right both learn more and forget less. SETA is shown at block budgets of 12% and 25%.

	SeqLoRA		O-LoRA		HiDe-LoRA		TreeLoRA		MoE-CL		SETA
Order	OP	BWT	OP	BWT	OP	BWT	OP	BWT	OP	BWT	OP	BWT
O1	$70.00_{\pm 0.00}$	$-6.67_{\pm 1.41}$	$70.00_{\pm 1.06}$	$-7.00_{\pm 0.00}$	$69.50_{\pm 0.35}$	$-5.83_{\pm 0.24}$	$69.50_{\pm 0.35}$	$-7.50_{\pm 1.18}$	$70.00_{\pm 1.77}$	$-7.33_{\pm 2.83}$	$\mathbf{75.50}_{\pm 1.06}$	$\mathbf{-2.00}_{\pm 0.94}$
O2	$71.38_{\pm 1.24}$	$-6.67_{\pm 0.47}$	$71.00_{\pm 0.71}$	$-6.67_{\pm 2.36}$	$69.75_{\pm 0.00}$	$-6.00_{\pm 0.47}$	$71.50_{\pm 1.06}$	$-6.00_{\pm 1.89}$	$68.12_{\pm 1.59}$	$-9.33_{\pm 1.41}$	$\mathbf{77.00}_{\pm 0.35}$	$\mathbf{-1.33}_{\pm 0.47}$
O3	$74.00_{\pm 0.35}$	$-4.00_{\pm 0.47}$	$73.38_{\pm 0.18}$	$-5.00_{\pm 0.00}$	$72.50_{\pm 0.35}$	$-4.17_{\pm 1.65}$	$73.50_{\pm 0.35}$	$-4.33_{\pm 0.00}$	$73.88_{\pm 0.18}$	$-2.33_{\pm 0.00}$	$\mathbf{74.00}_{\pm 1.06}$	$\mathbf{-1.67}_{\pm 0.47}$
O4	$76.15_{\pm 0.20}$	$-4.06_{\pm 0.26}$	$75.87_{\pm 1.23}$	$-4.35_{\pm 1.27}$	$74.42_{\pm 2.19}$	$-5.33_{\pm 2.05}$	$76.31_{\pm 0.02}$	$-3.84_{\pm 0.02}$	$76.33_{\pm 0.28}$	$\mathbf{-3.42}_{\pm 0.15}$	$\mathbf{77.14}_{\pm 2.06}$	$-4.16_{\pm 1.86}$
O5	$79.17_{\pm 0.33}$	$-1.49_{\pm 0.57}$	$79.44_{\pm 0.26}$	$-1.02_{\pm 0.45}$	$79.36_{\pm 0.16}$	$\mathbf{-0.41}_{\pm 0.03}$	$78.65_{\pm 0.05}$	$-2.00_{\pm 0.38}$	$78.01_{\pm 0.75}$	$-1.94_{\pm 0.24}$	$\mathbf{80.06}_{\pm 2.89}$	$-0.80_{\pm 0.42}$

Table 2: Comparison of continual learning methods on the O-LoRA benchmark with Qwen3-4B. O1–O3 are orders of the four standard text-classification tasks and O4–O6 orders of the fifteen-task long sequence, following Wang et al. (2023a) and Razdaibiedina et al. (2023) . We report OP and BWT (%) after the final task as mean

{}_{\pm\text{std}}

over two seeds. Bold marks the best mean for each metric in each row.

Figure 3: Expert structure of SETA along the TRACE-8 task sequence at block budgets of 12% (solid) and 25% (dashed). (a) LLaMA-2-7B and (b) Qwen3-4B: distinct V-projection block positions used by any expert, against the count if every task took fresh blocks (dotted); the shaded gap is the number of positions reused by later tasks (reuse saves positions, not stored parameters; Appendix D.1 ). (c) Number of experts and (d) mean expert size in blocks for all three backbones; at 25% each task adds exactly one expert.

Routing	$\beta=12\%$	$\beta=25\%$
Task-Agnostic	49.16	49.60
Oracle	49.21	51.09
Top-1	32.53	34.20
Top-2	27.69	37.83
Top-3	32.65	40.00

Table 3: Routing rules at inference on TRACE-8 with Qwen3-4B (seed 100, OP %). Each final checkpoint is evaluated with SETA’s own gates, oracle routing by task identity, or only the top-

k

gates. Keeping only the top-

k

gates lowers OP at both budgets, so predictions draw on several experts at once.

Test	$\varphi$	Passed	Shared	Undone (%)	Cos $<$ .02 (%)	OP	BWT
Strict	0.006	1446	468	67.6	0.0 ∗	34.69	$-$ 6.29
Normal	0.006	3226	1074	66.7	31.2	34.74	$-$ 5.56
Relaxed	0.006	3851	1182	69.3	37.7	34.65	$-$ 6.66
Default	0.012	2358	179	92.4	33.0	37.15	$-$ 4.95

Table 4: Effect of the direction test on TRACE-8 with Qwen3-4B (reduced protocol, seed 100). Strict uses

Z

=3,

\tau

=+0.02; normal

Z

=2,

\tau

=0; relaxed

Z

=1.5,

\tau

-

0.02; the default

Z

=3,

\tau

=0. Passed and Shared count blocks over tasks 2–8 that pass the test and that stay shared after the size rule; Undone is the share of passed blocks the size rule removes, and Cos

<

.02 the share with gradient cosine below 0.02.

$\lambda_{\text{base}}$	LA	OP	BWT	$\Delta$ OP
0 (no anchoring)	40.56	34.34	$-$ 7.11	$-$ 2.81
0.05	40.60	34.96	$-$ 6.44	$-$ 2.19
0.1 (default)	41.48	37.15	$-$ 4.95
0.2	42.24	36.79	$-$ 6.22	$-$ 0.36
0.5	42.35	35.31	$-$ 8.05	$-$ 1.84

Table 5: Sensitivity to the anchoring strength

\lambda_{\text{base}}

on TRACE-8 with Qwen3-4B, under the reduced protocol of Table 7 . All runs use the sign test and differ only in

\lambda_{\text{base}}

;

\Delta

OP is the change from the default, and

\lambda_{\text{base}}

=0 turns anchoring off. The default gives both the highest OP and the least forgetting.

Variant	LA $\uparrow$	OP $\uparrow$	BWT $\uparrow$	$\Delta$ OP
SETA ( $\beta$ =12%, V)	50.53	49.16	$-$ 1.56
Q projection	40.93	34.90	$-$ 6.89	$-$ 14.26
MLP projections	42.44	35.82	$-$ 7.56	$-$ 13.34
No shared experts	52.35	51.70	$-$ 0.85	+2.54

Table 6: Ablations of SETA on TRACE-8 with Qwen3-4B under the full protocol of Table 1 (5,000 training samples per task, seed 100), changing the adapted projection, the shared experts.

\Delta

OP is the change from the first row.

Variant	LA $\uparrow$	OP $\uparrow$	BWT $\uparrow$	$\Delta$ OP
SETA, all components	41.48	37.15	$-$ 4.95
Sign test off	40.96	34.77	$-$ 7.07	$-$ 2.38
No direction test (positional split)	39.46	35.37	$-$ 4.67	$-$ 1.78
No projection of own copies	40.97	37.51	$-$ 3.96	+0.36
Earlier gate rows trainable	40.93	35.89	$-$ 5.76	$-$ 1.26
No elastic anchoring ( $\lambda_{\text{base}}$ =0)	40.56	34.34	$-$ 7.11	$-$ 2.81

Table 7: Component ablations of SETA on TRACE-8 with Qwen3-4B under a reduced protocol (500 training samples per task, fixed warm-up, seed 100). Each row changes one setting of the full system, which here freezes the gate rows of earlier experts.

\Delta

OP is relative to the first row. Scores are not comparable with Table 6 .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Arm	$Z$	$\tau$	$\varphi$	Passed	Shared	Undone (%)	cos $<$ 0.02 (%)	Experts	LA	OP	BWT
	Setting			Split structure (tasks 2–8)					Performance
Strict	3.0	$+0.02$	0.006	1446	468	67.6	0.0 ∗	26	40.19	34.69	$-6.29$
Normal	2.0	$0.00$	0.006	3226	1074	66.7	31.2	41	39.60	34.74	$-5.56$
Relaxed	1.5	$-0.02$	0.006	3851	1182	69.3	37.7	43	40.47	34.65	$-6.66$
Default	3.0	$0.00$	0.012	2358	179	92.4	33.0	17	41.48	37.15	$-4.95$

Appendix

Table 8: Effect of the direction test’s strictness on Qwen3-4B (TRACE-8, reduced protocol with the sign test on).

Z

is the agreement bar,

\tau

the minimum gradient cosine for sharing and

\varphi

the minimum expert size as a fraction of the block budget. Split structure counts the blocks that pass the direction test, those kept as shared after the size rule, the share of passing blocks undone by that rule, the share of passing blocks with a gradient cosine below 0.02, and the experts after the last task. A looser test passes and shares more blocks, but the size rule undoes most of them and performance barely changes among the three arms at

\varphi

=0.006. ∗ Zero by construction.

Method	C-STANCE	FOMC	MeetingB.	Py150	SciQA	NG-cm	NG-ds	20Min.	OP $\uparrow$	BWT $\uparrow$
LLaMA-2-7B
FIX									$30.29_{\pm 1.86}$	–
SeqLoRA	$32.8_{\pm 2.5}$	$46.9_{\pm 3.6}$	$35.9_{\pm 0.5}$	$42.0_{\pm 3.2}$	$54.5_{\pm 7.2}$	$37.0_{\pm 3.5}$	$46.2_{\pm 0.4}$	$40.4_{\pm 0.8}$	$41.97_{\pm 0.84}$	$-9.72_{\pm 0.13}$
after own task	$51.2_{\pm 1.1}$	$65.7_{\pm 0.6}$	$39.2_{\pm 0.3}$	$60.9_{\pm 0.1}$	$58.7_{\pm 1.6}$	$37.0_{\pm 3.5}$	$50.6_{\pm 0.7}$	$40.4_{\pm 0.8}$
O-LoRA	$22.3_{\pm 17.4}$	$43.6_{\pm 0.7}$	$36.8_{\pm 0.0}$	$42.2_{\pm 11.0}$	$52.6_{\pm 5.4}$	$35.8_{\pm 5.2}$	$48.5_{\pm 0.7}$	$40.7_{\pm 0.2}$	$40.32_{\pm 1.93}$	$-11.76_{\pm 2.11}$
after own task	$51.2_{\pm 0.8}$	$65.9_{\pm 1.1}$	$39.4_{\pm 0.2}$	$60.9_{\pm 0.1}$	$59.6_{\pm 2.0}$	$38.3_{\pm 3.5}$	$48.8_{\pm 0.2}$	$40.7_{\pm 0.2}$

Appendix

Table 9: Per-task scores (%) on TRACE-8 in training order. For each trained method, the first row gives the score on each task after the last task (its mean is OP) and the second row the score right after that task was learned (its mean is LA), so comparing the two rows shows where each method forgets. All entries are mean

{}_{\pm\text{std}}

over two seeds (FIX runs on the LLaMA backbones have no per-task scores).

	C-ST	FOMC	MeetB	Py150	SciQA	NG-cm	NG-ds	20Min
$\beta$ =12%
Qwen3-4B, experts	1	3	4	5	6	9	10	13
shared blocks formed	0	55	0	0	0	22	0	31
layers kept (of 36)	34	33	34	34	34	30	33	31
trainable params (%)	0.28	0.28	0.28	0.28	0.28	0.27	0.28	0.28
LLaMA-2-7B, experts	1	3	5	7	9	11	16	19

Appendix

Table 10: Expert structure of SETA after each task on TRACE-8 for three backbones at block budgets at

\beta=12\%

, read from the split reports and training logs. Experts counts all experts after the task’s split; shared blocks formed counts the blocks moved into new shared experts; layers kept counts the layers that retain the task’s blocks after the minimum-size rule; trainable parameters are the experts updated for the task, as a share of the backbone. At the larger budget no shared expert forms, so each task adds one isolated expert (Figure 3 c).

Figure 4: How SETA’s experts grow on TRACE-8. (a) Number of experts after each task’s split; at the larger budget every task adds exactly one unique expert. (b) Blocks held by shared experts at

\beta

=12% (none form at 25%). (c) Size of every expert after the last task at

\beta

=12% (log scale), unique (filled) against shared (hollow). (d) Share of the backbone’s layers that keep the task’s blocks after the minimum-size rule. Shared experts stay small, so most of each task’s update remains in its own frozen expert.

Figure 5: Layer-wise share of top-ranked gradients in the Q, K and V projections. Nearly all high-magnitude gradients fall in the Value projection.

Figure 6: Selected blocks per layer and task on TRACE-8 for Qwen3-4B (left), LLaMA-2-7B (middle) and LLaMA-3-8B (right); darker cells mark more blocks. All three backbones spread their selection over every layer with a task-dependent profile.

Figure 7: Rate of novel blocks during selection for Qwen3-4B (left), LLaMA-2-7B (middle) and LLaMA-3-8B (right): the fraction of the first

n

selected blocks that no earlier task selected. The rate falls over the sequence as later tasks reuse blocks selected before them.

Figure 8: Distinct V-projection blocks adapted by any task so far on TRACE-8 (

\beta

=12%) for Qwen3-4B (left), LLaMA-2-7B (middle) and LLaMA-3-8B (right), against the count if every task took blocks no earlier task had selected (dashed). Reuse keeps the adapted set growing more slowly than the number of tasks.

Figure 9: Blocks per layer and task that no earlier task selected at the same position, for Qwen3-4B (left), LLaMA-2-7B (middle) and LLaMA-3-8B (right); darker cells mark more new coverage. Later tasks add less new coverage as more layers are already covered.

Setting	LA	OP	BWT	$\Delta$ OP
Full system	41.73	34.29	$-$ 8.50
$Z$ =1	39.98	34.78	$-$ 5.95	+0.49
$Z$ =2	40.79	37.17	$-$ 4.14	+2.87
$Z$ =4	42.23	37.75	$-$ 5.12	+3.46
$\varphi$ =0.006	41.76	35.91	$-$ 6.69	+1.61
$\varphi$ =0.025	39.44	33.64	$-$ 6.62	$-$ 0.65

Appendix

Table 11: Sensitivity to the split-rule settings on TRACE-8 with Qwen3-4B under the reduced protocol. Each row changes one setting from its default: the agreement bar

Z

, the size floor

\varphi

or the kept fraction share q of a shared piece.

\Delta

OP is the change from the full-system run of the same round (first row), whose sign-test setting was not recorded. Only a size floor larger than a typical layer’s selection breaks the method. † Run with the sign test, so no

\Delta

OP is given. The share q =0.5 run lacks one final score, so only its LA is reported.

Figure 10: Sensitivity to one setting at a time on TRACE-8 with Qwen3-4B under the reduced protocol: OP (top) and BWT (bottom) against the agreement bar

Z

, the size floor

\varphi

and the anchoring strength

\lambda_{\text{base}}

. Blue marks the default and the dashed line its value. The

Z

and

\varphi

runs share the full-system run of Table 11 , and the

\lambda_{\text{base}}

runs that of Table 7 ; share q is not drawn. Only a size floor that is too large breaks the method.

Setting	SETA	LoRA baselines
Backbones	LLaMA-2-7B, LLaMA-3-8B, Qwen3-4B; the same local weights for all methods
Prompt	Instruction prompt of Appendix G , without the model’s chat template
Training data	5,000 samples per task.
Evaluation	First 500 test samples per task.
Max. sequence length	512 tokens; MeetingBank 2,048 and 20Minuten 768 (1,536 and 512 on LLaMA-3-8B)
Learning rate	$9.65\times 10^{-6}$	$9.65\times 10^{-6}$

Appendix

Table 12: Training settings of SETA and the LoRA baselines (TreeLoRA code base). All methods train on the same task streams and data and use the same prompt format.

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 223 papers

Jun 5, 2026

Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning

Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typically treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities. We introduce Mixture of Sparse Experts for Task Agnostic Continual Learning (SETA), a framework that resolves the plasticity-stability conflict through adaptive sparse subspace decomposition into task-specific expert modules. Unlike standard updates, where tasks compete for the same parameters, SETA separates knowledge into unique experts, designed to isolate task-specific patterns, and shared experts, responsible for capturing common features. This structure is maintained through adaptive elastic anchoring and a routing-aware regularization that jointly protect shared knowledge at both the weight and routing levels and enable a unified gating network to automatically retrieve the correct expert combination during inference. Extensive experiments across diverse domain-specific benchmarks demonstrate that SETA achieves competitive or superior overall performance relative to state-of-the-art continual learning baselines, with particularly strong retention of early-task knowledge and improved backward transfer on LLaMA-2 7B and Qwen3-4B.

Fatema Siddika, Md Anwar Hossen, Tanwi Mallick +1

Iowa State University, Ames, USA · Argonne National Laboratory, USA

Continual Learning for LLMs Mixture-of-Experts Language Models Task-Free Continual Learning Catastrophic Forgetting

Ask PDF

Nov 9, 2025

Mixtures of SubExperts for Large Language Continual Learning

Enabling lifelong learning in LLMs demands resolving the stability-plasticity dilemma (i.e., models must incorporate new knowledge without overwriting prior representations) while maintaining scalability under bounded parameter growth. Existing PEFT methods fail to satisfy this triad; shared-parameter approaches suffer from catastrophic interference, while task-isolated expansions preclude knowledge transfer and scale linearly. We propose Mixtures of SubExperts (MoSEs), a modular and sparse framework that factorizes model capacity into reusable, compositional primitives. MoSEs augment transformer layers with lightweight SubExperts and a learned sub-routing function that dynamically selects and composes a sparse subset of modules conditioned on task inputs. This induces a structured decomposition of the parameter space where knowledge is localized yet accessible, mitigating interference while preserving reuse. Specifically, MoSEs balance the dilemma via three pillars: (i) stability by isolating knowledge within sparsely activated modules, (ii) plasticity through routing-driven recombination and selective expansion, and (iii) scalability via sublinear growth in effective capacity. Notably, the routing mechanism enables compositional generalization, allowing new tasks to be represented as combinations of previously acquired sub-functions. We empirically validate MoSEs on TRACE and SuperNI, showing reduced forgetting, improved forward transfer, and better parameter efficiency over strong PEFT baselines. MoSEs establish a new Pareto frontier, achieving state-of-the-art performance while maintaining strict parameter budgets. Our results suggest that modular sparsity and compositional routing are key inductive biases for building foundation models that continually learn without saturation.

Haeyong Kang, Hee Suk Yoon, Dahua Feng +1

Korea Advanced Institute of Science and Technology (KAIST)

Continual Learning for LLMs Mixture-of-Experts Language Models Incremental Learning Parameter-Efficient Fine-Tuning Catastrophic Forgetting

Ask PDF

May 18, 2026

CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning

Catastrophic forgetting remains a major obstacle to continual learning in large language models (LLMs) and vision--language models (VLMs). Although Mixture-of-Experts (MoE) architectures offer an efficient path to scaling, existing LoRA-based MoE continual learning methods still face a fundamental trade-off: they either isolate experts too aggressively, limiting knowledge transfer across tasks, or allow task-specific updates to overwrite important existing parameters, leading to severe forgetting. To address this, we propose CP-MoE, a continual learning framework built around a transient expert that captures early task-specific updates and guides their integration into stable experts. CP-MoE introduces a consistency-preserving routing bias, which uses the transient expert to estimate representation similarity with stable experts and steer routing towards more compatible expert selection, and a transient expert-guided regularisation mechanism, which selectively protects important historical parameters during merging. Together, these components reduce parameter interference and forgetting while preserving cross-task knowledge transfer. We validate CP-MoE on both unimodal and multimodal continual learning benchmarks with LLM-based and VLM-based MoE models. On SuperNI benchmark, spanning diverse sequential language tasks, CP-MoE achieves state-of-the-art performance and stronger zero-shot transfer to unseen tasks. On VQA v2 dataset, it scales effectively to multimodal visual reasoning, consistently reduces forgetting, and outperforms strong MoE baselines.

Yang Liu, Toan Nguyen, Flora D. Salim

School of Computer Science and Engineering University of New South Wales Australia

Continual Learning for LLMs Vision-Language Models Mixture-of-Experts Language Models Continual Learning for VLMs Catastrophic Forgetting

Ask PDF

Abstract

Continual Learning for LLMs

Explore similar work

Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning

Mixtures of SubExperts for Large Language Continual Learning

CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning