Variance Restoration over Covariance Rank: Memory-Budgeted Analytic Continual Learning
Organizations: University of Wisconsin–Madison · University of Washington
Abstract
Analytic continual learning replaces iterative gradient updates with closed-form ridge regression over frozen pre-trained representations, summarizing all past sessions in two running sufficient statistics: the feature autocorrelation matrix and the class cross-correlation matrix . Lifting backbone features into a wide random code of dimension substantially improves linear separability across tasks, but forces to grow quadratically as . Under a fixed persistent memory envelope , a learner faces a fundamental structural trade-off: store exact pairwise co-activations inside a narrower feature code, or expand to a wide code while compressing cross-coordinate interactions. We observe that the coordinate self-variances and class statistics grow only linearly with , whereas off-diagonal interactions dominate memory. We introduce \method (Coordinate-Restored Analytic Continual Learning), which retains exactly, maintains off-diagonal co-activations through a streaming rank- factor , restores exact diagonal variances via , and solves the full normal equations via the Woodbury identity. Across five class-incremental benchmarks on frozen ViT-B/16, \method matches dense full-covariance readouts within points while cutting persistent state by ( vs.\ MiB). Under equal memory envelopes of -- MiB, prioritizing expansion width over covariance rank outperforms exact narrow readouts in of configurations by up to points.
Figures & tables
| CIFAR-100 ( ) | CUB-200 ( ) | VTAB-50 ( ) | ImageNet-R ( ) | ImageNet-A ( ) | State | ||||||
| Method | MiB | ||||||||||
| Prompt-tuning, subspace, and adapter continual learners | |||||||||||
| L2P [ 33 ] | 87.96 | 81.82 | 77.72 | 66.88 | 83.82 | 73.41 | 72.34 | 65.18 | 49.15 | 41.08 | 327.8 |
| DualPrompt [ 32 ] | 88.62 | 83.19 | 79.48 | 69.65 | 84.11 | 75.70 | 71.62 | 64.75 | 53.42 | 45.20 | 328.6 |
| CODA-Prompt [ 24 ] | 90.41 | 85.34 | 82.64 | 74.12 | 87.90 | 80.45 | 75.18 | 68.92 | 56.84 | 48.76 | 331.2 |
| EASE [ 39 ] | 91.74 | 86.91 | 86.92 | 81.62 | 90.68 | 84.94 | 76.08 | 70.14 | 63.85 | 55.92 | 48.3 |
| CIFAR-100 | CUB-200 | VTAB-50 | ||||
| Readout Rule | ||||||
| Exact Narrow ( ) | 91.62 | 0.00 | 91.44 | 0.00 | 93.81 | 0.00 |
| Unrestored LR | 91.69 | 86.12 | 94.21 | |||
| Frequent Directions | 91.81 | 86.34 | 94.02 | |||
| Robust FD (RFD) | 92.22 | 91.88 | 94.56 | |||
| FD + Exact Var. | 92.19 | 91.58 | 94.35 | |||
| CIFAR-100 | CUB-200 | VTAB-50 | ||||
|---|---|---|---|---|---|---|
| State Configuration | ||||||
| Diagonal only | 83.64 | 77.89 | 63.62 | 52.94 | 36.82 | 31.78 |
| Subspace PCR ( ) | 92.19 | 87.76 | 91.42 | 86.48 | 89.98 | 92.82 |
| Unrestored LR ( ) | 92.18 | 87.09 | 91.50 | 87.34 | 95.04 | 92.48 |
| Frequent Directions (FD) | 91.80 | 85.82 | 91.64 | 87.44 | 95.08 | 92.61 |
| Robust FD (RFD) | 92.65 | 88.49 | 91.31 | 86.72 | 95.06 | 92.55 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Method | (%) | (%) | BWT (%) | Persist. MiB | Total MiB |
|---|---|---|---|---|---|---|
| CIFAR-100 | L2P [ 33 ] | 327.8 | 327.8 | |||
| CODA-Prompt [ 24 ] | 331.2 | 331.2 | ||||
| EASE [ 39 ] | 48.3 | 48.3 | ||||
| ACIL [ 44 ] | 518.2 | 566.2 | ||||
| REAL [ 6 ] | 518.2 | 566.2 | ||||
| F-OAL [ 43 ] | 8.4 | 11.7 |
| CIFAR-100 | CUB-200 | VTAB-50 | CIFAR | ||||
|---|---|---|---|---|---|---|---|
| Rank | (%) | (%) | (%) | (%) | (%) | (%) | State (MiB) |
| 128 | 8.7 | ||||||
| 256 | 13.6 | ||||||
| 512 | 23.4 | ||||||
| 840 | 35.9 | ||||||
| 1536 | 62.5 | ||||||
| CIFAR-100 | CUB-200 | VTAB-50 | |||||
|---|---|---|---|---|---|---|---|
| Envelope | Allocation ( / ) | (%) | (%) | (%) | (%) | (%) | (%) |
| 16 MiB | Exact Narrow ( / ) | 91.14 | 86.96 | 89.38 | 84.45 | 92.98 | 89.82 |
| Intermediate ( / ) | 91.92 | 88.12 | 90.76 | 86.04 | 94.12 | 91.45 | |
| Full Width ( / ) | 90.52 | 85.34 | 91.57 | 87.18 | 94.70 | 92.36 | |
| 36 MiB | Exact Narrow ( / ) | 91.48 | 87.52 | 89.94 | 85.08 | 93.66 | 90.84 |
| Intermediate ( / ) | 91.94 | 88.15 | 90.68 | 85.82 | 94.16 | 91.50 | |
| CIFAR-100 | CUB-200 | VTAB-50 | |||||
|---|---|---|---|---|---|---|---|
| Backbone / Horizon | Method | (%) | (%) | (%) | (%) | (%) | (%) |
| ResNet-50 ( ) | Fly-CL (dense ) | 84.15 | 76.74 | 78.31 | 69.58 | 93.31 | 89.90 |
| Exact Narrow ( MiB) | 82.60 | 73.98 | 75.18 | 65.96 | 92.85 | 89.15 | |
| CORE-ACL ( MiB) | 83.26 | 75.82 | 78.56 | 69.66 | 93.28 | 89.80 | |
| DINOv2 ViT-g/14 ( ) | Fly-CL (dense ) | 96.29 | 94.19 | 93.68 | 91.24 | 96.72 | 94.72 |
| Exact Narrow ( MiB) | 95.86 | 93.50 | 92.75 | 89.80 | 96.07 | 93.76 | |
| Hold-Out Validation + Refit | GCV: Packed vs. CORE-ACL | ||||||
|---|---|---|---|---|---|---|---|
| Dataset | Readout Rule | 16 MiB | 36 MiB | 80 MiB | 16 MiB | 36 MiB | 80 MiB |
| CIFAR-100 | Dense Exact Narrow | 91.30 | 91.62 | 91.86 | 91.14 | 91.48 | 91.85 |
| Packed Exact Narrow | 91.58 | 91.84 | 92.05 | 91.52 | 91.81 | 92.06 | |
| Unrestored LR | 91.68 | 91.69 | 92.22 | 89.84 | 91.76 | 92.14 | |
| Robust FD (RFD) | 91.82 | 92.22 | 92.31 | 90.41 | 92.28 | 92.30 | |
| CORE-ACL (standard) | 91.60 | 92.20 | 92.28 | 90.52 | 92.37 | 92.36 | |
| Sparsity Ratio ( ) | Row Non-Zeros ( ) | -Point Fine Grid | |||||||||
| Dataset | LR | Isotropic | CORE-ACL | ||||||||
| CIFAR-100 | 92.44 | 92.81 | 92.76 | 92.70 | 92.14 | 92.66 | 92.81 | 92.74 | 92.16 | 92.18 | 92.23 |
| CUB-200 | 91.18 | 91.52 | 91.46 | 91.35 | 90.88 | 91.39 | 91.52 | 91.48 | 91.56 | 91.88 | 91.84 |
| VTAB-50 | 94.82 | 95.07 | 95.04 | 94.96 | 94.65 | 94.98 | 95.07 | 95.05 | 92.88 | 93.64 | 93.81 |
| Dataset | Dense Solve (ms) | CORE-ACL Solve (ms) | Speedup | Session Fit (s) | Persist. MiB | Peak GPU MiB |
|---|---|---|---|---|---|---|
| CIFAR-100 | ||||||
| CUB-200 | ||||||
| VTAB-50 |
Continual Learning
Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.
See all 324 papers