cs.LGNov 17, 2026

Variance Restoration over Covariance Rank: Memory-Budgeted Analytic Continual Learning

Authors: Elena Voss, Pixel Nomand, Marcus Hale, Sofia Reyes

Organizations: University of Wisconsin–Madison · University of Washington

Abstract

Analytic continual learning replaces iterative gradient updates with closed-form ridge regression over frozen pre-trained representations, summarizing all past sessions in two running sufficient statistics: the feature autocorrelation matrix ΣtΣ_t and the class cross-correlation matrix ΓtΓ_t. Lifting backbone features into a wide random code of dimension M≫DM \gg D substantially improves linear separability across tasks, but forces ΣtΣ_t to grow quadratically as O(M2)\mathcal{O}(M^2). Under a fixed persistent memory envelope \Bcal\Bcal, a learner faces a fundamental structural trade-off: store exact pairwise co-activations inside a narrower feature code, or expand to a wide code while compressing cross-coordinate interactions. We observe that the coordinate self-variances vt=\diag(Σt)v_t = \diag(Σ_t) and class statistics ΓtΓ_t grow only linearly with MM, whereas off-diagonal interactions dominate memory. We introduce \method (Coordinate-Restored Analytic Continual Learning), which retains (vt,Γt)(v_t, Γ_t) exactly, maintains off-diagonal co-activations through a streaming rank-qq factor FtF_t, restores exact diagonal variances via \Sighatt=Ft⊤Ft+\Diag(vt−\diag(Ft⊤Ft))\Sighat_t = F_t^\top F_t + \Diag(v_t - \diag(F_t^\top F_t)), and solves the full M×CM\times C normal equations via the Woodbury identity. Across five class-incremental benchmarks on frozen ViT-B/16, \method matches dense full-covariance readouts within 0.090.09 points while cutting persistent state by 90.7%90.7\% (35.935.9 vs.\ 385.3385.3 MiB). Under equal memory envelopes of 1616--8080 MiB, prioritizing expansion width MM over covariance rank qq outperforms exact narrow readouts in 1717 of 1818 configurations by up to +2.19+2.19 points.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SPARCL: Spectral Partitioned Analytic Continual Learning

    Aug 21, 2026James Hartley, Zeropy Surio, Daniel Whitmore +2Continual LearningClass-Incremental Learning

  2. Energy-Structured Low-Rank Adaptation for Continual Learning

    May 26, 2026Longhua Li, Lei Qi, Qi Tian +1Continual LearningLow-Rank Adaptation

  3. PaLoRA: Paced Low-Rank Adaptation for Continual Learning

    Oct 3, 2026Yuxuan Li, Fanhu Zeng, Hao TangContinual LoRAContinual Learning