Task-Oriented Rank Adaptation for Continual Learning in Text Classification

Rey Sanchez Lopez; Eduardo Morales Manzanares; Hugo Jair Escalante

cs.CLOct 1, 2026

Task-Oriented Rank Adaptation for Continual Learning in Text Classification

Authors: Rey Sanchez Lopez, Eduardo Morales Manzanares, Hugo Jair Escalante

Organizations: INAOE, Puebla, Mexico · The University of Texas at El Paso, El Paso, TX, USA

Abstract

Continual learning (CL) in text classification faces two critical challenges: catastrophic forgetting and negative transfer across sequential tasks. Parameter-Efficient Fine-Tuning (PEFT) methods such as LoRA enable efficient adaptation by learning low-rank updates of the model parameters. However, these compact representations are normally trained in isolation, limiting their reuse across related tasks. We introduce Task-Oriented Rank Adaptation (TORA), a geometric routing framework that leverages the low-rank structure of LoRA adapters to decide whether to transfer knowledge from the most compatible expert (Boosting) or isolate the new task (Shielding) based on structural similarity. Evaluated across 15 diverse text classification benchmarks, TORA consistently avoids harmful routing decisions: compatible tasks exceed their isolated performance while reducing training time, and structurally distant tasks are protected from interference with no loss in accuracy. With a single geometric threshold and no reliance on task identities or predefined sequences, TORA provides a simple and effective approach for dynamic adapter routing in sequential text classification systems.

Figures & tables

Figure 1: Architectural diagram of the Task-Oriented Rank Adaptation (TORA) framework. The diagram illustrates the routing process, structural affinity evaluation and the memory of task experts.

Task	Benchmark	Type	Description
Yelp Polarity	CL Benchmark	Binary classification	Sentiment in Yelp reviews
Amazon Polarity	CL Benchmark	Binary classification	Sentiment in Amazon reviews
DBPedia-14	CL Benchmark	14-class classification	DBPedia article categories
Yahoo Answers	CL Benchmark	10-class classification	Topics in Yahoo questions
AG News	CL Benchmark	4-class classification	News classification
MNLI	GLUE	Natural language inference	Entailment/contradiction/neutral

Table 1: Evaluation tasks for the TORA framework.

Hyperparameter	Value
Base model	DistilBERT-base-uncased
LoRA rank ( $r$ )	64
LoRA alpha ( $\alpha$ )	16
LoRA dropout	0.1
LoRA bias	none
LoRA target modules	q_lin, k_lin, v_lin

Table 2: Experimental hyperparameter configuration.

Task	$\theta_{1}$	$\theta_{2}$	$\theta_{3}$	$\theta_{4}$	$\theta_{5}$	$\bar{\theta}_{1}$	$\bar{\theta}_{2}$	$\bar{\theta}_{3}$	$\bar{\theta}_{4}$	$\bar{\theta}_{5}$	Decision
IMDB	75.9348	78.2152	81.5054	81.9472	83.3364	75.9348	77.0750	78.5518	79.4006	80.1878	Transfer
WIC	78.2600	80.1432	86.1518	86.2642	86.3813	78.2600	79.2016	81.5183	82.7048	83.4401	Transfer
RTE	78.6749	79.5074	87.4172	87.7247	87.7818	78.6749	79.0912	81.8665	83.3311	84.2212	Transfer
COPA	79.5401	81.8497	82.7471	82.9056	83.0827	79.5401	80.6949	81.3790	81.7606	82.0250	Transfer
YELP	79.8080	84.1105	88.0673	88.8443	91.1074	79.8080	81.9593	83.9953	85.2075	86.3875	Transfer
MULTIRC	80.4721	80.7501	81.1396	82.0910	82.0954	80.4721	80.6111	80.7873	81.1132	81.3096	Transfer

Table 3: Cosine similarity (in degrees) between each incoming task and its Top-5 nearest experts, along with cumulative averages

\bar{\theta}_{k}

for

k=1,\ldots,5

. The routing threshold of

84.5^{\circ}

applied to

\bar{\theta}_{3}

provides the clearest separation between transfer and shielding decisions.

Task	Forced Transfer	Scratch (TORA)	Avoided Degradation	Epochs Saved	Routing Decision
CB	0.820	0.820	0.000	10	Isolation (Scratch)
MNLI	0.628	0.652	+0.024	10	Isolation (Scratch)
SST2	0.870	0.870	0.000	10	Isolation (Scratch)

Table 4: Interference Prevention: Performance and computational cost of forcing geometric transfer versus applying the Shielding protocol.

Task (Domain)	Isolation Prob. (Scratch)	Transfer Prob.	Geometric Classification
The Isolated Cases (Isolators)
SST2	99.50%	0.50%	Highly Isolated Node
CB	99.50%	0.50%	Highly Isolated Node
MNLI	99.50%	0.50%	Highly Isolated Node
AGNEWS	69.00%	31.00%	Rigid Knowledge Node
The Compatible Cases (Hubs)

Table 6: Routing probability distribution following Monte Carlo simulation (

N=100,000

Method / Configuration	Order	$\boldsymbol{A_{T}}$ (%)
CL Ours (Similar Tasks First)	1	60.75
CL Ours (Reverse Order)	2	66.99
CL Ours Avg	–	63.87
O-LoRA [ 23 ]	3	72.3
O-LoRA [ 23 ]	4	64.8
O-LoRA [ 23 ]	5	71.6

Table 7: Continual learning performance comparison across 15 text classification tasks. We report the Average Final Accuracy (

A_{T}

) for different sequence configurations.

Explore similar work

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning
Jun 26, 2026Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar +2Low-Rank Adaptation Adapters Model Fine-Tuning
Department of Computer Science, ETH Zurich, Zurich, Switzerland.
Energy-Structured Low-Rank Adaptation for Continual Learning
May 26, 2026Longhua Li, Lei Qi, Qi Tian +1Replay-Based Continual Learning Continual Learning
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shenzhen, China
Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
Jul 26, 2026Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2Replay-Based Continual Learning Catastrophic Forgetting
College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA +1

Jun 26, 2026, cs.LGJ/K move · Enter open · S save

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning

Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar, Louis Barinka+1

Department of Computer Science, ETH Zurich, Zurich, Switzerland.

Low-Rank Adaptation Adapters Model Fine-Tuning Continual Fine-Tuning Continual Learning Low-Rank Structure Redundancy

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) has become the standard tool for parameter-efficient fine-tuning of large pretrained models. When applied sequentially across tasks in Continual Learning (CL), the standard assumption is that each new task requires a dedicated low-rank adapter. In this work, we challenge this assumption empirically and structurally. We show that task-specific LoRA adapters in CL exhibit significant low-rank redundancy: the subspaces spanned by adapters trained on different tasks substantially overlap, and in many cases earlier adapters can faithfully represent later tasks. Building on this observation, we propose LiteLoRA, a plug-and-play gating mechanism that learns at train time whether to recruit a new adapter or reuse existing low-rank representations. Our method reduces the number of active adapters by 20-70% while matching or exceeding state-of-the-art performance on standard CL benchmarks, revealing that structural redundancy is pervasive and that selective learning is sufficient to achieve stability without sacrificing plasticity.

Abstract

Explore similar work

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning

Energy-Structured Low-Rank Adaptation for Continual Learning

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning