Unifying Detection and Adaptation in Task-Free Continual Learning

Dezheng Han; Anbang Zhang; Zhihao Zhu; Shuaishuai Guo

cs.LGAug 27, 2026

Unifying Detection and Adaptation in Task-Free Continual Learning

Authors: Dezheng Han, Anbang Zhang, Zhihao Zhu, Shuaishuai Guo

Organizations: School of Control Science and Engineering, Shandong University, Jinan, China · Department of ECE, The Hong Kong University of Science and Technology, Hong Kong · National Medicine-Engineering Interdisciplinary Industry-Education Integration Innovation Platform, Shandong University, Jinan, China · Shandong Key Laboratory: Magnetic Field-free Medicine & Functional Imaging, Qilu Hospital of Shandong University, Jinan, China

Abstract

To mitigate catastrophic forgetting in downstream continual learning (CL) for large language models (LLMs), existing methods typically constrain parameter updates or introduce task-specific adaptation modules. However, these methods often rely on explicit task boundaries during training, limiting their applicability to realistic task-free scenarios. In this paper, we propose a Fisher-guided unified (FiUni) framework for batch-level task detection and parameter-efficient continual adaptation. FiUni is motivated by a key observation about the Fisher information matrix (FIM) of pre-trained models: the orthogonality among the principal subspaces of its Kronecker-Factored Approximate Curvature (K-FAC) approximation, estimated from a small number of downstream task samples, can reflect the similarity between different tasks. Based on this observation, FiUni constructs FIM-derived frozen subspaces to guide low-rank adaptation (LoRA), while matching the Fisher principal subspace of each incoming batch window with historical subspaces. This enables FiUni to adaptively determine whether to reuse existing knowledge, expand a related subspace, or create a new subspace, dynamically balancing knowledge sharing and task isolation. Experiments show that FiUni can effectively infer latent batch-level task affiliations and achieve competitive performance against advanced task-aware CL methods with fewer trainable parameters.

Figures & tables

Figure 1: Fisher principal subspaces encode task similarity. For each downstream task, we estimate the K-FAC approximation and extract the top- $r$ principal subspaces of its gradient and activation covariance factors. Higher subspace similarity indicates more similar task geometry, while lower similarity indicates stronger orthogonality.

Figure 2: Overview of FiUni. For each incoming batch in a task-free stream, FiUni estimates its Fisher principal subspace and matches it with the historical subspace pool. Based on the subspace similarity, FiUni adaptively decides to create a New subspace, Expand a related subspace, or Reuse an existing one.

Figure 3: Empirical analysis of Fisher subspace similarity and FiUni decisions on T5-base. Panels (a)–(c) estimate Fisher subspaces using 32 samples per task over all linear layers with $r_{det}=8$ : (a) shows cross-task Fisher subspace similarity, (b) shows within-task self-similarity from two independent samplings, and (c) compares within-task similarity with task-to-other similarity. Panel (d) shows the batch-level decisions produced by FiUni on a task-free stream using 32-shot Fisher estimation, selected Q/K/V/O layers, and thresholds $\tau_{\mathrm{low}}=0.65$ and $\tau_{\mathrm{high}}=0.85$ .

	Methods	Standard CL Benchmark (SC)				Long Sequence Benchmark (LS)				TRACE
	Methods	Order 1	Order 2	Order 3	Avg	Order 4	Order 5	Order 6	Avg	Order 7
T5-Large	SeqLoRA †	39.5	31.9	46.6	39.3	4.9	3.5	4.2	4.2	12.1
	SeqLoRAReplay	74.0	73.1	73.0	73.3	74.2	72.7	73.9	73.6	34.0
	IncLoRA	63.4	62.2	65.1	63.6	63.0	57.9	60.4	60.5	-
	EWC Kirkpatrick et al. (2017)	46.3	45.3	52.1	47.9	44.9	44.0	45.4	44.8	-
	L2P Wang et al. (2022)	60.3	61.7	61.1	60.7	57.5	53.8	56.9	56.1	-

Table 1: Comparison results on Standard CL Benchmark (SC), Long Sequence Benchmark (LS), and TRACE. FiUni results are averaged over three random seeds. Methods marked with † do not rely on task boundaries during training. Gray-colored methods use replay data, generated pseudo-data, or external memory during continual learning. Bold indicates the best result, and underline indicates the second-best result among non-replay methods.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset Name	Category	Task	Domain	Metric
Yelp	CL Benchmark	Sentiment Analysis	Yelp Reviews	Accuracy
Amazon	CL Benchmark	Sentiment Analysis	Amazon Reviews	Accuracy
DBpedia	CL Benchmark	Topic Classification	Wikipedia	Accuracy
Yahoo	CL Benchmark	Topic Classification	Yahoo Q&A	Accuracy
AG News	CL Benchmark	Topic Classification	News	Accuracy
MNLI	GLUE	Natural Language Inference	Various	Accuracy

Appendix

Table 2: Task details of the Long Sequence Benchmark (LS). The first five datasets correspond to the standard CL benchmark, while the remaining tasks are drawn from GLUE, SuperGLUE, and IMDB.

Dataset	Source	Avg. Len.	Metric	Language	#Data
Domain-specific
ScienceQA	Science	210	Accuracy	English	5,000
FOMC	Finance	51	Accuracy	English	5,000
MeetingBank	Meeting	2853	ROUGE-L	English	5,000
Multi-lingual
C-STANCE	Social Media	127	Accuracy	Chinese	5,000

Appendix

Table 3: Task details of TRACE. The benchmark covers domain-specific tasks, multilingual tasks, code completion, and mathematical reasoning.

Benchmark	Order	Task Sequence
Standard CL Benchmark	1	DBpedia $\rightarrow$ Amazon $\rightarrow$ Yahoo $\rightarrow$ AG News
	2	DBpedia $\rightarrow$ Amazon $\rightarrow$ AG News $\rightarrow$ Yahoo
	3	Yahoo $\rightarrow$ Amazon $\rightarrow$ AG News $\rightarrow$ DBpedia
Long Sequence Benchmark	4	MNLI $\rightarrow$ CB $\rightarrow$ WiC $\rightarrow$ COPA $\rightarrow$ QQP $\rightarrow$ BoolQA $\rightarrow$ RTE $\rightarrow$ IMDB $\rightarrow$ Yelp $\rightarrow$ Amazon $\rightarrow$ SST-2 $\rightarrow$ DBpedia $\rightarrow$ AG News $\rightarrow$ MultiRC $\rightarrow$ Yahoo
	5	MultiRC $\rightarrow$ BoolQA $\rightarrow$ WiC $\rightarrow$ MNLI $\rightarrow$ CB $\rightarrow$ COPA $\rightarrow$ QQP $\rightarrow$ RTE $\rightarrow$ IMDB $\rightarrow$ SST-2 $\rightarrow$ DBpedia $\rightarrow$ AG News $\rightarrow$ Yelp $\rightarrow$ Amazon $\rightarrow$ Yahoo
	6	Yelp $\rightarrow$ Amazon $\rightarrow$ MNLI $\rightarrow$ CB $\rightarrow$ COPA $\rightarrow$ QQP $\rightarrow$ RTE $\rightarrow$ IMDB $\rightarrow$ SST-2 $\rightarrow$ DBpedia $\rightarrow$ AG News $\rightarrow$ Yahoo $\rightarrow$ MultiRC $\rightarrow$ BoolQA $\rightarrow$ WiC

Appendix

Table 4: Task orders used in our continual learning experiments. Orders 1–3 correspond to SC, Orders 4–6 correspond to LS, and Order 7 corresponds to TRACE.

Hyperparameter	T5-large			Llama-3.1-8B
Hyperparameter	SC	LS	TRACE	SC	LS	TRACE
Batch size	32
Learning rate	1e-3			1e-4
Epochs	1
FiLoRA rank $r$	32			128
Dropout	0	0.05	0	0.1	0.1	0

Appendix

Table 5: Training and adaptation hyperparameters for FiUni. Batch size denotes the effective batch size, computed as the raw batch size multiplied by the number of gradient accumulation steps.

Hyperparameter	T5-large			Llama-3.1-8B
	SC	LS	TRACE	SC	LS	TRACE
Detection rank $r_{\mathrm{det}}$	4			8
Expansion rank $\Delta r$	0	4	4	8	8	8
$\tau_{\mathrm{low}}$	0.5			0.6
$\tau_{\mathrm{high}}$	0.7
Detection layers $\mathcal{L}_{\mathrm{det}}$	23			31

Appendix

Table 6: Detection and decision hyperparameters for FiUni.

Figure 4: Ablation on the detection rank $r_{\mathrm{det}}$ for Fisher subspace similarity computation. A moderate rank provides a stable separation between within-task self-similarity and task-to-other similarity.

Figure 5: Ablation on the number of samples used for Fisher estimation. More samples lead to more stable Fisher subspaces and clearer separation between within-task and cross-task similarities.

Figure 6: Ablation on module selection for Fisher subspace similarity computation. Different module choices preserve similar task-level structures, showing that the similarity signal is not tied to a single module configuration.

Figure 7: Ablation on layer selection for Fisher subspace similarity computation. The resulting similarity structures remain broadly consistent across different layer choices, supporting efficient detection with selected layers.

Figure 8: Training loss and FiUni decision trajectory on the LS benchmark with T5-large. The upper panel shows the training loss along the online data stream, and the lower panel shows the corresponding batch-level Reuse , Expand , and New decisions.

Figure 9: Training loss and FiUni decision trajectory on the SC benchmark with T5-large. Even when some task transitions do not produce obvious loss spikes, FiUni can still identify the corresponding latent shifts through Fisher subspace matching and trigger appropriate batch-level decisions.

Metric	O-LoRA	SpaRTA	FiUni (ours)
Trainable Params	35.39M	44.24M	3.62M

Appendix

Table 7: Trainable parameter comparison on the LS benchmark. For FiUni, we report the dynamic trainable parameter count obtained on Order 4.

Task-Free Continual Learning

See all 19 papers

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
Jul 26, 2026Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2Continual Learning for LLMs Continual LoRA
College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA +1
Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning
Jan 24, 2026Fatema Siddika, Md Anwar Hossen, Tanwi Mallick +1Continual Learning for LLMs Continual Learning
Iowa State University, Ames, USA · Argonne National Laboratory, USA
Continual Fine-Tuning of Large Language Models via Program Memory
May 13, 2026Hung Le, Svetha VenkateshContinual Learning for LLMs Memory-Augmented Language Models
Deakin Applied AI Initiative, Deakin University, Australia

Jul 26, 2026J/K move · Enter open · S save

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli, Pavan Turaga+1

College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA+1

Continual Learning for LLMs Continual LoRA Incremental Learning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially, leading to catastrophic forgetting. To mitigate this, LoRA-based continual learning methods allocate a separate low-rank adapter per task, yet existing approaches either require task identity at inference or sum all adapters indiscriminately, letting irrelevant branches distort the output. Recent gating-based solutions route inputs to the correct adapter but introduce trainable parameters that themselves need protection against forgetting. In this work, we observe that pooled token embeddings from a frozen LLM embedding layer already separate task distributions throughout the learning sequence. A Gaussian mixture model fitted on these embeddings, without any gradient-based training, is sufficient for task-agnostic adapter selection at test time. This eliminates the need for a learned gating module. On the adapter side, constraining each task's parameters to the principal subspace of the pretrained weights via SVD yields a compact latent-space parameterization. Within this subspace, orthogonal regularization directly controls inter-task interference. The resulting system, Latent-LoRA, is replay-free, requires no trainable routing component, and uses substantially fewer parameters per task. Experiments across five model scales and two established continual learning benchmarks show state-of-the-art performance with near-zero forgetting.

Abstract

Task-Free Continual Learning

Explore similar work

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning

Continual Fine-Tuning of Large Language Models via Program Memory