cs.LGNov 17, 2026

Variance Restoration over Covariance Rank: Memory-Budgeted Analytic Continual Learning

Authors: Elena Voss, Pixel Nomand, Marcus Hale, Sofia Reyes

Organizations: University of Wisconsin–Madison · University of Washington

Abstract

Analytic continual learning replaces iterative gradient updates with closed-form ridge regression over frozen pre-trained representations, summarizing all past sessions in two running sufficient statistics: the feature autocorrelation matrix ΣtΣ_t and the class cross-correlation matrix ΓtΓ_t. Lifting backbone features into a wide random code of dimension M≫DM \gg D substantially improves linear separability across tasks, but forces ΣtΣ_t to grow quadratically as O(M2)\mathcal{O}(M^2). Under a fixed persistent memory envelope \Bcal\Bcal, a learner faces a fundamental structural trade-off: store exact pairwise co-activations inside a narrower feature code, or expand to a wide code while compressing cross-coordinate interactions. We observe that the coordinate self-variances vt=\diag(Σt)v_t = \diag(Σ_t) and class statistics ΓtΓ_t grow only linearly with MM, whereas off-diagonal interactions dominate memory. We introduce \method (Coordinate-Restored Analytic Continual Learning), which retains (vt,Γt)(v_t, Γ_t) exactly, maintains off-diagonal co-activations through a streaming rank-qq factor FtF_t, restores exact diagonal variances via \Sighatt=Ft⊤Ft+\Diag(vt−\diag(Ft⊤Ft))\Sighat_t = F_t^\top F_t + \Diag(v_t - \diag(F_t^\top F_t)), and solves the full M×CM\times C normal equations via the Woodbury identity. Across five class-incremental benchmarks on frozen ViT-B/16, \method matches dense full-covariance readouts within 0.090.09 points while cutting persistent state by 90.7%90.7\% (35.935.9 vs.\ 385.3385.3 MiB). Under equal memory envelopes of 1616--8080 MiB, prioritizing expansion width MM over covariance rank qq outperforms exact narrow readouts in 1717 of 1818 configurations by up to +2.19+2.19 points.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Aug 21, 2026

SPARCL: Spectral Partitioned Analytic Continual Learning

Analytic continual learning has emerged as a strong exemplar-free alternative to gradient-based class-incremental learning because it replaces iterative optimization with closed-form ridge updates. Yet the usual forgetting narrative, centered on stochastic gradient overwriting, does not explain why analytic methods still drift on old classes despite exact recursive solvers. We identify the culprit as spectral interference: the joint ridge classifier for all tasks shares the inverse autocorrelation operator (R+λI)−1(R+λI)^{-1}, so incoming task samples that load onto old dominant eigendirections dilute the spectrum and perturb old-class logits even when old labels are never revisited. Based on this view, we propose SPARCL, a spectral partitioned analytic continual learner that decomposes the running autocorrelation into a high-energy core and a residual complement, freezes old-class classifier components in the core subspace, and updates only the residual block through recursive least squares with an optional residual random-projection expansion. This yields a simple closed-form update with a provable invariance guarantee for the core contribution of old logits. Across CIFAR-100, CUB-200, ImageNet-R, and ImageNet-A under a frozen ViT-B/16 protocol, SPARCL closes most of the gap from classical analytic learners to strong representation matchers, while remaining complementary to sparse feature-decorrelation approaches such as Fly-CL.
May 26, 2026

Energy-Structured Low-Rank Adaptation for Continual Learning

While orthogonal subspace methods try to mitigate task interference in Continual Learning (CL), they often suffer from energy diffusion across the basis, hindering knowledge compaction and exhausting capacity for future tasks. We observe that output feature drift induced by parameter updates is inherently low-rank, and theoretically prove that preserving parameters along the principal directions of this drift minimizes the output reconstruction error. Motivated by this, we propose Energy-Concentrated and Energy-Ordered Low-Rank Adaptation (E2^2-LoRA). By explicitly ordering and concentrating knowledge into leading ranks, E2^2-LoRA frees capacity for subsequent tasks. Furthermore, we design a dynamic rank allocation strategy to balance stability and plasticity by jointly optimizing energy retention and model plasticity. Extensive experiments across multiple benchmarks demonstrate that E2^2-LoRA achieves state-of-the-art performance. Code is available at https://github.com/kiddo127/E2-LoRA.
Oct 3, 2026

PaLoRA: Paced Low-Rank Adaptation for Continual Learning

LoRA-based continual learning methods mitigate catastrophic forgetting through various mechanisms, yet nearly all complement these with small learning rates as a heuristic to restrict gradient scaling magnitude. Such fixed heuristics lack theoretical guidance on how the strength of this restriction should evolve as tasks accumulate. We reveal that even under directional constraints such as nullspace projection, finite-precision updates inevitably leak into the subspace of accumulated prior knowledge along multiple directions. While small learning rates attenuate such leakage, they cannot prevent the accumulated forgetting from intensifying as the effective rank of historical knowledge grows. We show that the optimal magnitude restriction should adaptively increase with this effective rank to balance stability and plasticity, i.e., preservation of previous knowledge and acquisition of new task information. Under an anisotropic leakage model, we derive a pacing law s∗=R/cs^*=\sqrt{R/c} that characterizes the optimal scaling of gradient steps, i.e., the magnitude restriction itself, where RR is the effective rank of past updates. Based on this insight, we propose PaLoRA, which compresses historical knowledge via adaptive SVD truncation, projects gradients onto the nullspace of prior tasks, and applies rank-aware adaptive pacing. Experiments demonstrate consistent improvements over prior methods, with particularly strong performance in long-horizon settings, achieving substantial gains of 4% accuracy on challenging 50-task ImageNet-A and ImageNet-R benchmarks.