cs.LGAug 27, 2026

Unifying Detection and Adaptation in Task-Free Continual Learning

Authors: Dezheng Han, Anbang Zhang, Zhihao Zhu, Shuaishuai Guo

Organizations: School of Control Science and Engineering, Shandong University, Jinan, China · Department of ECE, The Hong Kong University of Science and Technology, Hong Kong · National Medicine-Engineering Interdisciplinary Industry-Education Integration Innovation Platform, Shandong University, Jinan, China · Shandong Key Laboratory: Magnetic Field-free Medicine & Functional Imaging, Qilu Hospital of Shandong University, Jinan, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

To mitigate catastrophic forgetting in downstream continual learning (CL) for large language models (LLMs), existing methods typically constrain parameter updates or introduce task-specific adaptation modules. However, these methods often rely on explicit task boundaries during training, limiting their applicability to realistic task-free scenarios. In this paper, we propose a Fisher-guided unified (FiUni) framework for batch-level task detection and parameter-efficient continual adaptation. FiUni is motivated by a key observation about the Fisher information matrix (FIM) of pre-trained models: the orthogonality among the principal subspaces of its Kronecker-Factored Approximate Curvature (K-FAC) approximation, estimated from a small number of downstream task samples, can reflect the similarity between different tasks. Based on this observation, FiUni constructs FIM-derived frozen subspaces to guide low-rank adaptation (LoRA), while matching the Fisher principal subspace of each incoming batch window with historical subspaces. This enables FiUni to adaptively determine whether to reuse existing knowledge, expand a related subspace, or create a new subspace, dynamically balancing knowledge sharing and task isolation. Experiments show that FiUni can effectively infer latent batch-level task affiliations and achieve competitive performance against advanced task-aware CL methods with fewer trainable parameters.

Figures & tables

Figure 1: Fisher principal subspaces encode task similarity. For each downstream task, we estimate the K-FAC approximation and extract the top- rr principal subspaces of its gradient and activation covariance factors. Higher subspace similarity indicates more similar task geometry, while lower similarity indicates stronger orthogonality.
Figure 2: Overview of FiUni. For each incoming batch in a task-free stream, FiUni estimates its Fisher principal subspace and matches it with the historical subspace pool. Based on the subspace similarity, FiUni adaptively decides to create a New subspace, Expand a related subspace, or Reuse an existing one.
Figure 3: Empirical analysis of Fisher subspace similarity and FiUni decisions on T5-base. Panels (a)–(c) estimate Fisher subspaces using 32 samples per task over all linear layers with rdet=8r_{det}=8 : (a) shows cross-task Fisher subspace similarity, (b) shows within-task self-similarity from two independent samplings, and (c) compares within-task similarity with task-to-other similarity. Panel (d) shows the batch-level decisions produced by FiUni on a task-free stream using 32-shot Fisher estimation, selected Q/K/V/O layers, and thresholds τlow=0.65\tau_{\mathrm{low}}=0.65 and τhigh=0.85\tau_{\mathrm{high}}=0.85 .
 MethodsStandard CL Benchmark (SC)Long Sequence Benchmark (LS)TRACE
Order 1Order 2Order 3AvgOrder 4Order 5Order 6AvgOrder 7
T5-LargeSeqLoRA †39.531.946.639.34.93.54.24.212.1
SeqLoRAReplay74.073.173.073.374.272.773.973.634.0
IncLoRA63.462.265.163.663.057.960.460.5-
EWC Kirkpatrick et al. (2017)46.345.352.147.944.944.045.444.8-
L2P Wang et al. (2022)60.361.761.160.757.553.856.956.1-
Table 1: Comparison results on Standard CL Benchmark (SC), Long Sequence Benchmark (LS), and TRACE. FiUni results are averaged over three random seeds. Methods marked with † do not rely on task boundaries during training. Gray-colored methods use replay data, generated pseudo-data, or external memory during continual learning. Bold indicates the best result, and underline indicates the second-best result among non-replay methods.
Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset NameCategoryTaskDomainMetric
YelpCL BenchmarkSentiment AnalysisYelp ReviewsAccuracy
AmazonCL BenchmarkSentiment AnalysisAmazon ReviewsAccuracy
DBpediaCL BenchmarkTopic ClassificationWikipediaAccuracy
YahooCL BenchmarkTopic ClassificationYahoo Q&AAccuracy
AG NewsCL BenchmarkTopic ClassificationNewsAccuracy
MNLIGLUENatural Language InferenceVariousAccuracy
Appendix
Table 2: Task details of the Long Sequence Benchmark (LS). The first five datasets correspond to the standard CL benchmark, while the remaining tasks are drawn from GLUE, SuperGLUE, and IMDB.
DatasetSourceAvg. Len.MetricLanguage#Data
Domain-specific
ScienceQAScience210AccuracyEnglish5,000
FOMCFinance51AccuracyEnglish5,000
MeetingBankMeeting2853ROUGE-LEnglish5,000
Multi-lingual
C-STANCESocial Media127AccuracyChinese5,000
Appendix
Table 3: Task details of TRACE. The benchmark covers domain-specific tasks, multilingual tasks, code completion, and mathematical reasoning.
BenchmarkOrderTask Sequence
Standard CL Benchmark1DBpedia →\rightarrow Amazon →\rightarrow Yahoo →\rightarrow AG News
 2DBpedia →\rightarrow Amazon →\rightarrow AG News →\rightarrow Yahoo
 3Yahoo →\rightarrow Amazon →\rightarrow AG News →\rightarrow DBpedia
Long Sequence Benchmark4MNLI →\rightarrow CB →\rightarrow WiC →\rightarrow COPA →\rightarrow QQP →\rightarrow BoolQA →\rightarrow RTE →\rightarrow IMDB →\rightarrow Yelp →\rightarrow Amazon →\rightarrow SST-2 →\rightarrow DBpedia →\rightarrow AG News →\rightarrow MultiRC →\rightarrow Yahoo
 5MultiRC →\rightarrow BoolQA →\rightarrow WiC →\rightarrow MNLI →\rightarrow CB →\rightarrow COPA →\rightarrow QQP →\rightarrow RTE →\rightarrow IMDB →\rightarrow SST-2 →\rightarrow DBpedia →\rightarrow AG News →\rightarrow Yelp →\rightarrow Amazon →\rightarrow Yahoo
 6Yelp →\rightarrow Amazon →\rightarrow MNLI →\rightarrow CB →\rightarrow COPA →\rightarrow QQP →\rightarrow RTE →\rightarrow IMDB →\rightarrow SST-2 →\rightarrow DBpedia →\rightarrow AG News →\rightarrow Yahoo →\rightarrow MultiRC →\rightarrow BoolQA →\rightarrow WiC
Appendix
Table 4: Task orders used in our continual learning experiments. Orders 1–3 correspond to SC, Orders 4–6 correspond to LS, and Order 7 corresponds to TRACE.
HyperparameterT5-largeLlama-3.1-8B
SCLSTRACESCLSTRACE
Batch size32
Learning rate1e-31e-4
Epochs1
FiLoRA rank rr32128
Dropout00.0500.10.10
Appendix
Table 5: Training and adaptation hyperparameters for FiUni. Batch size denotes the effective batch size, computed as the raw batch size multiplied by the number of gradient accumulation steps.
HyperparameterT5-largeLlama-3.1-8B
SCLSTRACESCLSTRACE
Detection rank rdetr_{\mathrm{det}}48
Expansion rank Δr\Delta r044888
τlow\tau_{\mathrm{low}}0.50.6
τhigh\tau_{\mathrm{high}}0.7
Detection layers Ldet\mathcal{L}_{\mathrm{det}}2331
Appendix
Table 6: Detection and decision hyperparameters for FiUni.
Figure 4: Ablation on the detection rank rdetr_{\mathrm{det}} for Fisher subspace similarity computation. A moderate rank provides a stable separation between within-task self-similarity and task-to-other similarity.
Figure 5: Ablation on the number of samples used for Fisher estimation. More samples lead to more stable Fisher subspaces and clearer separation between within-task and cross-task similarities.
Figure 6: Ablation on module selection for Fisher subspace similarity computation. Different module choices preserve similar task-level structures, showing that the similarity signal is not tied to a single module configuration.
Figure 7: Ablation on layer selection for Fisher subspace similarity computation. The resulting similarity structures remain broadly consistent across different layer choices, supporting efficient detection with selected layers.
Figure 8: Training loss and FiUni decision trajectory on the LS benchmark with T5-large. The upper panel shows the training loss along the online data stream, and the lower panel shows the corresponding batch-level Reuse , Expand , and New decisions.
Figure 9: Training loss and FiUni decision trajectory on the SC benchmark with T5-large. Even when some task transitions do not produce obvious loss spikes, FiUni can still identify the corresponding latent shifts through Fisher subspace matching and trigger appropriate batch-level decisions.
MetricO-LoRASpaRTAFiUni (ours)
Trainable Params35.39M44.24M3.62M
Appendix
Table 7: Trainable parameter comparison on the LS benchmark. For FiUni, we report the dynamic trainable parameter count obtained on Order 4.
More in this topic

Task-Free Continual Learning

See all 19 papers

Explore similar work

CardsList
Jul 26, 2026

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially, leading to catastrophic forgetting. To mitigate this, LoRA-based continual learning methods allocate a separate low-rank adapter per task, yet existing approaches either require task identity at inference or sum all adapters indiscriminately, letting irrelevant branches distort the output. Recent gating-based solutions route inputs to the correct adapter but introduce trainable parameters that themselves need protection against forgetting. In this work, we observe that pooled token embeddings from a frozen LLM embedding layer already separate task distributions throughout the learning sequence. A Gaussian mixture model fitted on these embeddings, without any gradient-based training, is sufficient for task-agnostic adapter selection at test time. This eliminates the need for a learned gating module. On the adapter side, constraining each task's parameters to the principal subspace of the pretrained weights via SVD yields a compact latent-space parameterization. Within this subspace, orthogonal regularization directly controls inter-task interference. The resulting system, Latent-LoRA, is replay-free, requires no trainable routing component, and uses substantially fewer parameters per task. Experiments across five model scales and two established continual learning benchmarks show state-of-the-art performance with near-zero forgetting.

Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2

College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA +1

Continual Learning for LLMsContinual LoRAIncremental LearningParameter-Efficient Fine-Tuning
AskPDF
Jan 24, 2026

Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning

Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typically treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities. We introduce Mixture of Sparse Experts for Task-Agnostic Continual Learning (SETA), a framework that resolves the plasticity-stability conflict through adaptive sparse subspace decomposition into task-specific expert modules. Unlike standard updates, where tasks compete for the same parameters, SETA separates knowledge into unique experts, designed to isolate task-specific patterns, and shared experts, responsible for capturing common features. SETA shares an overlapping parameter block only when the new task's update passes a subspace-agreement test and a sign test against its prior expert; shared knowledge is protected by adaptive elastic anchoring, and independently gated routers retrieve the relevant experts at inference without task identifiers. Extensive experiments across diverse domain-specific benchmarks demonstrate that SETA achieves competitive or superior overall performance relative to state-of-the-art continual learning baselines, improving overall performance over the strongest baseline by 1.1 to 1.7 points on TRACE and by up to 5.5 points on the O-LoRA benchmark, with backward transfer between -2.2 and -1.3 on TRACE, against 4.6 to 6.3 points of forgetting for MoE-CL and 8 to 15 points for the other LoRA baselines.

Fatema Siddika, Md Anwar Hossen, Tanwi Mallick +1

Iowa State University, Ames, USA · Argonne National Laboratory, USA

Continual Learning for LLMsContinual LearningTask-Free Continual LearningMixture-of-Experts Language Models
AskPDF
May 13, 2026

Continual Fine-Tuning of Large Language Models via Program Memory

Parameter-Efficient Fine-Tuning (PEFT), particularly Low-Rank Adaptation (LoRA), has become a standard approach for adapting Large Language Models (LLMs) under limited compute. However, in continual settings where models are updated sequentially with small datasets, conventional LoRA updates struggle to balance rapid adaptation and knowledge retention. Existing methods typically treat the low-rank space as a homogeneous update region, lacking mechanisms to regulate how short-term updates are consolidated over time. We propose a continual LoRA framework with Program memory, inspired by Complementary Learning Systems in neuroscience. Our approach, dubbed ProCL, organizes LoRA adapters into structured program memory slots that are dynamically retrieved through input-conditioned attention. This enables rapid and localized adaptation, encouraging similar inputs to reuse shared adapter regions while reserving unused capacity for future data. The slots are then combined with the underlying adapter, which maintains a distributed representation that gradually accumulates knowledge across tasks to balance plasticity and stability. Our method operates entirely within the LoRA parameterization and incurs no additional inference cost. Experiments on diverse benchmarks demonstrate improved retention and reduced catastrophic forgetting over other continual LoRA strategies.

Hung Le, Svetha Venkatesh

Deakin Applied AI Initiative, Deakin University, Australia

Continual Learning for LLMsMemory-Augmented Language ModelsLow-Rank AdaptationContinual LoRAParameter-Efficient Fine-Tuning
AskPDF