Geo-LoRA: Geometry-Aware Subspace Evolution for Low-Rank Adaptation in Continual Learning

Yibo Feng

cs.CVAug 27, 2026

Geo-LoRA: Geometry-Aware Subspace Evolution for Low-Rank Adaptation in Continual Learning

Authors: Yibo Feng

Organizations: University of Electronic Science and Technology of China Chengdu, China

Abstract

Rehearsal-free class-incremental learning (CIL) with LoRA adapters remains challenging because the low-rank subspaces updated across tasks evolve without geometric control, causing unstable shared representations and repetitive collapse of task-specific updates into previously occupied directions. We introduce Geo-LoRA, a geometry-aware framework that explicitly regulates how low-rank subspaces, both shared and task-specific, evolve during continual learning. For the shared branch, Subspace Projection Preservation (SPP) constrains consecutive updates to follow smooth trajectories on the Grassmann manifold, and Adaptive Core-Slack Alignment (ACSA) decomposes transitions into principal and residual components, aligning the former while modulating the latter to balance stability and plasticity. For the task-specific branch, Median-Calibrated Block Overlap (MCBO) imposes a statistical constraint via normalized projection overlap, penalizing excessive reuse to mitigate subspace crowding. These constraints jointly regulate the evolution of all LoRA subspaces across layers and tasks without introducing additional adapter types beyond standard LoRA. Geo-LoRA provides a principled geometric formulation for continual low-rank adaptation and consistently achieves state-of-the-art performance across multiple benchmark datasets and different task lengths.

Figures & tables

Figure 1. Comparison of different PTM-based CIL methods on ImageNet-R dataset (20 tasks). The X-axis stands for the number of trainable parameters, and the Y-axis represents the average accuracy. For a fair comparison, all methods are based on pre-trained ViT-B/16-IN1K.

Figure 2. Overview of Geo-LoRA. Geo-LoRA regulates the geometric evolution of LoRA-induced subspaces through three lightweight constraints. (1) SPP enforces smooth transitions of the shared low-rank subspace by minimizing Grassmannian projector drift. (2) ACSA decomposes subspace changes into core and slack components, preserving aligned directions while adaptively controlling residual activation. (3) MCBO suppresses excessive reuse of historical directions in deep task-specific blocks via median-calibrated normalized overlaps. Together, these objectives coordinate stable yet flexible subspace evolution across tasks without increasing inference cost.

Method	ImageNet-R ( $T{=}20$ )		ImageNet-R ( $T{=}10$ )		ImageNet-A ( $T{=}20$ )		ImageNet-A ( $T{=}10$ )		VTAB ( $T{=}5$ )		CUB-200 ( $T{=}10$ )
Method	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$
L2P ( Wang et al., 2022b )	68.97	74.16	71.26	76.13	38.48	47.16	42.94	51.40	80.83	81.19	65.18	76.12
DualPrompt ( Wang et al., 2022a )	65.23	71.30	68.22	73.81	50.23	59.54	45.49	54.68	79.79	82.89	68.00	79.40
CODA-Prompt ( Smith et al., 2023 )	69.38	73.95	74.05	78.14	35.02	47.29	45.36	57.03	82.33	83.88	71.92	78.76
InfLoRA ( Liang and Li, 2024 )	68.89	76.68	74.75	80.67	46.21	59.71	49.20	60.92	87.63	88.90	70.82	81.39
SD-LoRA ( Wu et al., 2025 )	75.26	80.22	77.34	82.04	52.71	62.21	55.96	64.95	75.71	87.03	77.48	85.59

Table 1. Results (%) on ImageNet-R, ImageNet-A, VTAB and CUB200 with ViT-B/16-IN1K

Method	ImageNet-R ( $T{=}20$ )		ImageNet-R ( $T{=}10$ )		ImageNet-A ( $T{=}20$ )		ImageNet-A ( $T{=}10$ )		VTAB ( $T{=}5$ )		CUB-200 ( $T{=}10$ )
Method	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$
L2P ( Wang et al., 2022b )	62.15	68.35	64.94	70.33	39.30	46.67	37.62	39.81	76.41	78.96	63.24	75.01
DualPrompt ( Wang et al., 2022a )	66.89	73.07	62.24	74.63	48.78	58.45	47.45	56.43	80.94	82.51	69.22	79.80
CODA-Prompt ( Smith et al., 2023 )	67.53	73.64	72.15	77.51	37.06	50.73	51.61	60.70	89.49	92.27	70.11	80.74
InfLoRA ( Liang and Li, 2024 )	71.01	77.28	75.65	80.82	41.61	56.84	47.04	56.91	87.16	88.83	70.47	81.02
SD-LoRA ( Wu et al., 2025 )	74.31	80.00	77.01	81.92	45.89	56.60	50.10	59.17	72.77	85.24	63.74	82.61

Table 2. Results (%) on ImageNet-R, ImageNet-A, VTAB and CUB200 with ViT-B/16-IN21K

Method	ImageNet-R		ImageNet-A
Method	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$
L2P	60.62	65.82	27.42	38.09
DualPrompt	61.73	67.41	42.75	52.26
CODA-Prompt	63.93	70.39	27.93	38.39
InfLoRA	64.51	73.22	40.02	52.97
SD-LoRA	70.07	76.21	42.49	53.07

Table 3. Results (%) on ImageNet-R and ImageNet-A with ViT-B/16-IN1K (

T{=}40

Method	ImageNet-R		ImageNet-A
Method	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$
L2P	73.04	76.94	40.21	49.02
DualPrompt	69.99	72.24	54.72	62.93
CODA-Prompt	76.63	80.30	42.38	50.21
InfLoRA	76.95	81.81	49.37	63.27
SD-LoRA	77.30	82.30	55.83	65.15

Table 4. Results (%) on ImageNet-R and ImageNet-A with ViT-B/16-IN1K (

T{=}5

Figure 3. Results (%) on OmniBenchmark and Cifar100 across training stages with ViT-B/16-IN1K ( $T{=}20$ ).

Figure 4. Empirical validation of the geometric mechanisms in Geo-LoRA. (a) SPP suppresses shared subspace drift over training epochs. (b) MCBO reduces median-calibrated overlap in deep task-specific LoRA across tasks, indicating controlled reuse of historical directions. (c) ACSA improves cross-task core alignment of shared subspaces. (d) ACSA maintains higher core alignment during training. These results demonstrate that the proposed constraints explicitly regulate subspace drift, overlap, and alignment, jointly enabling controlled geometric evolution of LoRA subspaces.

Method	T(20task)	I(20task)	T(10task)	I(10task)
Baseline	44.37 min	77.84 s	30.83 min	39.61 s
Geo-LoRA	46.85 min	77.60 s	32.27 min	39.54 s

Table 5. Total training time (T) and inference time (I) comparison on ImageNet-A.

Method	20task	10task	5task
Baseline	53.26	57.43	63.39
+ SPP + ACSA	53.84	58.42	63.71
+ SPP + MCBO	54.10	59.04	63.98
+ ACSA + MCBO	53.92	58.97	63.84
Full Geo-LoRA	54.58	59.62	64.25

Table 6. Ablation of geometric objectives on the ImageNet-A dataset with a 1K-pretrained model.

Normalization Choice	$A_{T}$ (%)	$\bar{A}$ (%)
$A_{t-1}(A_{t-1}^{\top}A_{t-1}+\varepsilon I)^{-1}A_{t-1}^{\top}$	79.27	84.77
$A_{t-1}(A_{t-1}^{\top}A_{t-1})^{-1}A_{t-1}^{\top}$	79.62	84.93

Table 7. Comparison of SPP normalization schemes on ImageNet-R (20 tasks).

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 289 papers

Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning
Feb 27, 2026Lingfeng He, De Cheng, Huaijie Wang +3Continual Learning Low-Rank Adaptation
State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi’an, China · School of Electronic Engineering, Xidian University, Xi’an, China · AUTHORERR: Missing \icmlaffiliation.
C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models
Feb 25, 2025Xin Zhang, Liang Bai, Xian YangContinual LoRA Low-Rank Adaptation
Institute of Intelligent Information Processing, Shanxi University, Taiyuan, 030006, China · Alliance Manchester Business School, The University of Manchester, Manchester, M13 9PL, UK
Energy-Structured Low-Rank Adaptation for Continual Learning
May 26, 2026Longhua Li, Lei Qi, Qi Tian +1Continual Learning Low-Rank Adaptation
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shenzhen, China

Feb 27, 2026J/K move · Enter open · S save

Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

Lingfeng He, De Cheng, Huaijie Wang, Xi Yang+2

State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi’an, China · School of Electronic Engineering, Xidian University, Xi’an, China · AUTHORERR: Missing \icmlaffiliation.

Continual Learning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Continual Learning (CL) requires models to sequentially adapt to new tasks without forgetting old knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL. Several LoRA-based CL methods reduce interference across tasks by separating their update spaces, typically building the new space from the estimated null space of past tasks. However, they (i) overlook task-shared directions, which suppresses knowledge transfer, and (ii) fail to capture truly effective task-specific directions since these ``null bases" of old tasks can remain nearly inactive for new task under correlated tasks. To address this, we study LoRA learning capability from a projection energy perspective, and propose Low-rank Decomposition and Adaptation (LoDA). It performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-based objectives, decoupling directions for knowledge sharing and isolation. LoDA fixes LoRA down-projections on two subspaces and learns robust up-projections via a Gradient-Aligned Optimization (GAO) approach. After each task, before integrating the LoRA updates into the backbone, LoDA derives a closed-form recalibration for the general update, approximating a feature-level joint optimum along this task-shared direction. Experiments indicate that LoDA outperforms existing CL methods. Our code is available at https://github.com/HHHLF/LoDA_ICML2026.

Abstract

Continual Learning

Explore similar work

Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models

Energy-Structured Low-Rank Adaptation for Continual Learning