FedSEAM: Federated Continual Learning of LLMs via Subspace Expert Agreement Merging

Fatema Siddika; Abhijit Chunduru; Zilinghan Li; Ali Jannesari; Ravi Madduri

cs.LGOct 24, 2026

FedSEAM: Federated Continual Learning of LLMs via Subspace Expert Agreement Merging

Authors: Fatema Siddika, Abhijit Chunduru, Zilinghan Li, Ali Jannesari, Ravi Madduri

Organizations: Iowa State University, Ames, USA · Argonne National Laboratory, USA · University of Massachusetts Amherst, Amherst, USA

Abstract

Federated continual learning trains a model on a sequence of tasks across several clients without sharing their data while retaining knowledge of earlier tasks. Sparse mixture-of-experts models support this setting by adapting selected parameters on a frozen backbone. However, clients learn different expert content as tasks evolve, making it difficult to share knowledge across clients without disrupting task-specific and client-specific specialization. Matching tile coordinates alone does not establish whether their learned content is compatible. We present FedSEAM, a federated continual learning framework that organizes task knowledge into sparse experts and selectively shares their content across tasks and clients. On each client, task-aware tile selection allocates capacity to a new task, while a subspace-based compatibility test determines which existing expert tiles to reuse and where to introduce fresh capacity. On the server, Subspace Expert Agreement Merging (SEAM) compares client updates as subspaces and moves each client towards consensus only along agreed-upon directions, leaving the rest of its update unchanged. We identify two remaining sources of forgetting, routing drift and merge erasure, and address them with three protections, one of which exactly preserves tile components in a selected prior-overlap subspace at every merge. We evaluate FedSEAM on TRACE using Qwen3-4B and Llama-3.2-3B, with component-wise ablations and studies of data size to assess its balance between learning new tasks and retaining prior knowledge.

Figures & tables

Figure 1: LA and OP of FedSEAM and FedAvg on Qwen3-4B with 5,000 training samples per task (client mean); labels give BWT. FedSEAM achieves comparable LA and an OP that is 2.5 points higher.

Figure 2: Overview of FedSEAM. Once per task, each client selects tiles for $T_{t}$ (1) and shares a tile with an earlier expert only if both updates use the same input directions and agree in sign (2). In each round, clients train the current experts (3); the server keeps the directions $\mathcal{C}$ that most clients move along in the same sense (4) and merges the tiles along them (5); each client then recalibrates its current gate rows (6). (i)–(iii) mark the protections against routing drift and merge erasure.

Method	C-ST	FOMC	MB	Py150	SciQA	NG-cm	NG-ds	20Min	LA	OP	BWT
FedAvg over tiles	60.48	65.36	37.34	56.23	79.62	56.92	50.77	38.61	55.67	49.27	$-$ 6.99
FedSEAM	60.68	64.84	37.95	56.32	80.47	55.13	51.02	38.56	55.62	51.52	$-$ 4.68

Table 1: Main results on TRACE with Qwen3-4B and 5,000 training samples per task (1,000 per client, equal split): score on each task right after it is learned (

R_{j,j}

, client mean), LA, OP and BWT. Bold: the higher of the two methods.

Group	Setting	Value
Federation	clients, rounds per task, local passes per round	5, 3, 3
	training instances per client and task	1,000 (100 in the low-data setting)
Architecture	adapted matrices; tile side	value projections; 128
	budget per task; expansion limit $\kappa$	15% (864 tiles); 1
	router	sigmoid, initial bias 1.0
Warm-up	steps; instances per step	at most one pass; 2

Table 6: Settings of the main configuration.

Figure 3: Router change made by calibration in each task and round: the norm of the difference between a client’s router after calibration and after local training. Lines are client means; bands span the five clients.

Figure 4: FedSEAM minus FedAvg on each task right after it is learned (

R_{j,j}

), with 500 and with 5,000 training samples per task.

Figure 5: Per-task scores of FedSEAM and FedAvg on Qwen3-4B with 5,000 training samples per task (client mean). (a) Score right after each task is learned. (b) Score of the model after the eighth task. The radial axis starts at 20.

After $\backslash$ on	C-STANCE	FOMC	MeetingBank	Py150	ScienceQA	NumGLUE-cm
T1	60.68
T2	59.20	64.84
T3	54.76	59.23	37.95
T4	57.50	55.34	29.81	56.32
T5	49.90	52.82	28.70	33.58	80.47
T6	54.35	55.70	31.13	53.52	82.50	55.13

Table 7: Backward evaluation of FedSEAM, Qwen3-4B, 5,000 samples per task: score on each task (column) after learning each task (row). Unmarked cells average all five clients; a superscript gives the number of clients averaged.

After $\backslash$ on	C-STANCE	FOMC	MeetingBank	Py150	ScienceQA	NumGLUE-cm
T1	60.48
T2	59.16	65.36
T3	53.76	60.89	37.34
T4	56.25	54.59	29.37	56.23
T5	48.50	50.00	29.51	28.96	79.62
T6	54.30	55.44	32.65	52.44	81.40	56.92

Table 8: Backward evaluation of FedAvg over tiles, Qwen3-4B, 5,000 samples per task, as in Table 7 .

Figure 6: Gain over local training on each task (500 samples per task, equal split, client mean). Top: right after the task is learned (

R_{j,j}

). Bottom: at the end of the stream (

R_{8,j}

). NumGLUE-cm accounts for most of the difference between the two merges.

Figure 7: Change of each task from right after learning it to the end of the stream ( $R_{8,j}-R_{j,j}$ , 500 samples per task, equal split, client mean), tasks 1–7. Red is forgetting, blue is a gain.

Figure 8: NumGLUE-cm score of each client right after the task is learned (equal split).

Figure 9: LA and OP of every client’s model (small markers) and the mean of each method (large markers; the Local mean is drawn hollow because it lies almost on the FedSEAM mean). The star is the centralized model.

Figure 10: Server aggregation on Llama-3.2-3B. (a) Share of a client’s update that lies in the agreed subspace,

e_{C}

, mean over the three rounds of each task. (b–d) The rounds of NumGLUE-cm (task 6): (b) agreed share, (c) share of bins left private, (d) median change the merge makes to a client’s tile, relative to the tile. FedSEAM+R also averages the part of the update outside the agreed subspace.

$t$	task	Own	Experts	Shared	Own	Experts	Shared	$\Delta$ Own
		direction-aware			positional
1	C-STANCE	800	1	0	800	1	0	$0$
2	FOMC	473	3	351	500	3	327	$-27$
3	MeetingBank	869	4	351	715	7	404	$+154$
4	ScienceQA	649	9	657	601	13	746	$+48$
5	NumGLUE-cm	753	12	657	614	21	860	$+139$

Table 19: Expert structure along the task stream. Own is the number of tiles in the task’s own expert, Experts the number of experts present after the task, and Shared the number of tiles held by shared experts. Tasks are listed in the order used in this pass.

After the eighth task	Direction	Position
Mean tiles in own expert ( $t{\geq}2$ )	727	611
Experts present	32	68
Median expert size (tiles)	51	21
Tiles in shared experts	1127	1596
Total tiles held	2933	2940

Table 20: Expert structure under the direction-aware split and sharing by position (structure-only pass, one client holding all data, Qwen3-4B, budget 12%).

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 233 papers

Task-Agnostic Federated Continual Learning via Replay-Free Gradient Projection
Sep 25, 2025Seohyeon Cha, Huancheng Chen, Haris VikaloFederated Continual Learning Gradient Projection
Department of Electrical and Computer Engineering The University of Texas at Austin
Task Relevance-Driven Prompt Expansion for Federated Continual Learning
Oct 25, 2026Jiahua Dong, Wenqi Liang, Yang Cong +6Continual Learning Federated Continual Learning
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · University of Trento, Trento, Italy · South China University of Technology, Guangzhou, China +4
ReSCENE: Server-Side Replay for Structural Mitigation of Catastrophic Forgetting in Federated Continual Learning
Sep 30, 2026Sungmin Kang, Zhengzhong Tu, Sunwoo LeeFederated Continual Learning Replay-Based Continual Learning
Texas A&M University · Inha University

Sep 25, 2025J/K move · Enter open · S save

Task-Agnostic Federated Continual Learning via Replay-Free Gradient Projection

Seohyeon Cha, Huancheng Chen, Haris Vikalo

Department of Electrical and Computer Engineering The University of Texas at Austin

Federated Continual Learning Gradient Projection Task-Free Continual Learning Federated Learning

Open paper AskPDF arXiv

Federated continual learning (FCL) enables collaborative model training across distributed clients on sequentially arriving tasks without revisiting past data. However, existing approaches often suffer from catastrophic forgetting, rely on replay buffers or generative models that may violate privacy constraints, or assume knowledge of task identities during inference. We propose FedProTIP (Federated Projection-based Continual Learning with Task Identity Prediction), a replay-free FCL framework that maintains shared task-specific feature subspaces across clients. Each client extracts low-rank core bases from intermediate activations using randomized singular value decomposition, capturing dominant feature directions associated with the current task. These bases are transmitted to the server and aggregated to construct global task subspaces that capture shared feature directions across clients without requiring data sharing. During training, client updates are projected onto the orthogonal complement of previously learned subspaces to reduce cross-task interference and mitigate catastrophic forgetting. The learned subspaces are also reused during inference to estimate task identity via subspace relevance, enabling task-agnostic prediction without requiring explicit task labels. Experiments on CIFAR100, ImageNet-R, and DomainNet demonstrate that FedProTIP consistently outperforms state-of-the-art federated continual learning baselines while maintaining lower training time, memory footprint, and communication cost.

Abstract

Continual Learning for LLMs

Explore similar work

Task-Agnostic Federated Continual Learning via Replay-Free Gradient Projection

Task Relevance-Driven Prompt Expansion for Federated Continual Learning

ReSCENE: Server-Side Replay for Structural Mitigation of Catastrophic Forgetting in Federated Continual Learning