Score-Preserving Continual Lasso under Task Shift

Jingyi Guo; Runze Li; Lingzhou Xue

stat.MLOct 23, 2026

Score-Preserving Continual Lasso under Task Shift

Authors: Jingyi Guo, Runze Li, Lingzhou Xue

Organizations: Pennsylvania State University

Abstract

Continual learning (CL) aims to learn from sequential tasks while mitigating catastrophic forgetting when historical data are no longer accessible. We study continual learning for high-dimensional sparse linear regression, where $\ell_1$ regularization creates a difficulty absent from continual ridge regression: that recursively reusing an $\ell_1$ -penalized estimator can propagate earlier shrinkage and accumulate regularization bias across tasks. We propose score-preserving continual Lasso (SP-CL), which separates sparse estimation from state transmission by preserving the historical squared-loss score. Under squared loss, its cumulative estimator exactly recovers a full-history Lasso minimizer. When task coefficients coincide, it inherits the full-history sparse rate. For related task-specific coefficients, SP-CL uses a regularized current-task correction that leaves the transmitted state unchanged and attains the offline source-assisted rate under an oracle related-task condition. We also derive Overall/BWT/FWT decompositions that distinguish historical information loss, task conflict, and current-task specialization, and validate the resulting regimes in simulation.

Figures & tables

Work	Sparse	Task-specific	Current-task	Overall/BWT/FWT
		coefficients	correction
Li et al. (2023)	$\times$	$\times$	$\times$	$\times$
Zhao et al. (2024)	$\times$	$\times$	$\times$	$\checkmark$
Zhao et al. (2025)	$\times$	$\times$	$\times$	$\checkmark$
Karpel et al. (2026)	$\times$	$\checkmark$	$\times$	$\times$
SP-CL	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 1: Comparison with representative theoretical work on regularization-based continual linear regression. “Task-specific coefficients” allows different population coefficients across tasks; “Current-task correction” denotes an explicit second-stage correction using the current task.

Figure 1: Signal retained on the true active support (

p=200

s=10

n_{t}=40

T=30

; 50 streams).

Estimator	Raw history	Structure	Error scale	Signed support
Current-only Lasso	No	Sparse	$\displaystyle\frac{s\log p}{n_{t}}$	$\checkmark^{\ddagger}$
Offline pooled Lasso	Yes	Sparse	$\displaystyle\frac{s\log p}{N_{t}}$	$\checkmark^{\ddagger}$
Gen. $\ell_{2}$ CL ( Zhao et al., 2024 )	No	Dense	$\displaystyle\frac{p}{N_{t}}^{\dagger}$	–
SP-CL cumulative	No	Sparse	$\displaystyle\frac{s\log p}{N_{t}}$	$\checkmark^{\ddagger}$

Table 2: Statistical comparison in the shared-coefficient setting. Rates show representative squared-estimation/excess-prediction scales under well-conditioned designs; dense and sparse rows correspond to different parameter classes and are not direct minimax comparisons.

Figure 2: Sparse efficiency (

p=1000

s=10

n_{t}=40

T=20

N_{T}=800<p

; 50 streams). Excess risk and support size; pilot-tuned Ridge versus SP-CL (pooled Lasso matches SP-CL).

Figure 3: Transfer under task shift (50 streams). Mean gains as task mismatch $h$ and history ratio $m=N_{t-1}/n_{t}$ vary: $G_{\mathrm{pool}}=\mathcal{R}_{t}(\widehat{\beta}_{t}^{\mathrm{cur}})-\mathcal{R}_{t}(\widehat{w}_{t})$ , $G_{\mathrm{corr}}=\mathcal{R}_{t}(\widehat{w}_{t})-\mathcal{R}_{t}(\widehat{\beta}_{t})$ , and $G_{\mathrm{adapt}}=G_{\mathrm{pool}}+G_{\mathrm{corr}}$ ; positive values $=$ improvement and hatching $=$ correction rescue (Appendix B.5 ).

Figure 4: Continual performance under task heterogeneity (

p=200

s=10

n_{t}=40

T=20

; 50 streams). Final-task Overall, FWT, and BWT for the cumulative generalist and current-adapted specialist as

h

varies; error bars show one standard error.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Historical score discrepancy

\lVert G_{t}-H_{t}(\cdot)\rVert_{2}/N_{t}

under the same noisy, correlated design as Figure 3.1 , for direct-center (

\widehat{w}_{t}^{\mathrm{dir}}

) and SP-CL (

u_{t}

), log scale.

Figure 6: Final-task (

t=T

) sensitivity to the penalty constant

c_{\lambda}

, same 50 seeds reused across

c_{\lambda}\in\{0.25,0.5,0.75,1,1.25\}

. Panel (A): score discrepancy. Panel (B): active-support magnitude. Panel (C): population MSE.

Figure 7: Squared

\ell_{2}

estimation error in the high-dimensional sparse-versus-dense experiment of Figure 2 . Pooled Ridge uses the pilot-tuned task-specific penalty schedule.

Method	Future MSFE	$R^{2}_{\mathrm{OOS}}$	Hist. MSFE	$\mathrm{MaxF}_{T}^{\mathrm{hist}}$	$\mathrm{MReg}_{T}^{\mathrm{hist}}$	Final support
Zero anomaly	5.256	0.000	–	–	–	–
Persistence	5.902	$-0.123$	–	–	–	–
Current-only Lasso	4.944	0.059	3.709	2.475	1.911	13
Offline pooled Lasso	3.914	0.255	3.551	0.891	0.000	20
Direct center	4.524	0.139	3.980	1.639	1.353	10
SP-CL	3.914	0.255	3.551	0.891	0.000	20

Appendix

Table 3: Climate-stream results with historical test blocks. “Future MSFE” uses the disjoint future block (winters 2025–2026). “Hist. MSFE” is the final-state average over the eight held-out historical test winters.

\mathrm{MaxF}_{T}^{\mathrm{hist}}

and

\mathrm{MReg}_{T}^{\mathrm{hist}}

are the maxima over time of historical forgetting and of historical regret relative to Offline pooled Lasso, respectively.

Figure 8: Climate-stream historical test-block analysis. Top left: raw future-block MSFE for each method, with zero-anomaly and persistence baselines. Top right: historical forgetting $\mathrm{MaxF}_{t}$ . Bottom left: historical regret $\mathrm{MReg}_{t}$ relative to Offline pooled Lasso. Bottom right: pairwise empirical covariance dissimilarities between chronological training tasks. The common penalty is $\lambda_{t}=0.20/\sqrt{t}$ .

Figure 9: Historical forgetting heatmaps and final support comparison. The heatmaps show $[A_{t,j}-A_{j,j}]_{+}$ for SP-CL and Direct center on a common color scale. The map compares final supports: orange points are selected only by Offline pooled Lasso/SP-CL, blue triangles only by Direct center; point size uses a common coefficient-magnitude scale, the dashed line marks the eastern edge of the SLP predictor window at $120^{\circ}$ W, and the star marks Los Angeles immediately east of that window.

Figure 10: Signed final support maps for the climate-stream application. Red points denote positive coefficients, blue points denote negative coefficients, with point size on a common absolute-magnitude scale across panels. These maps separate coefficient sign from the method-overlap comparison in Figure 9 .

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 301 papers

CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework
Sep 15, 2026Yunxiang Fu, Meng Lou, Zicheng Liao +1Class-Incremental Learning Continual Learning
School of Computing and Data Science, The University of Hong Kong · Hong Kong Generative AI Research and Development Center
Recovery Guarantees for Continual Learning of Dependent Tasks: Memory, Data-Dependent Regularization, and Data-Dependent Weights
Apr 19, 2026Liangzu Peng, Uday Kiran Reddy Tadipatri, Ziqing Xu +2Continual Learning Replay-Based Continual Learning
University of Pennsylvania
Optimal Explicit and Implicit Regularization Strengths in High-Dimensional Continual Linear Regression
Oct 14, 2026Gilad Karpel, Daniel SoudryHigh-Dimensional Linear Regression Regularization-Based Continual Learning
Department of Data and Decisions Sciences, Technion · Department of Electrical and Computer Engineering, Technion

Sep 15, 2026J/K move · Enter open · S save

CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework

Yunxiang Fu, Meng Lou, Zicheng Liao, Yizhou Yu

School of Computing and Data Science, The University of Hong Kong · Hong Kong Generative AI Research and Development Center

Class-Incremental Learning Continual Learning Sparse Fine-Tuning Incremental Learning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Continual learning must balance the learning of new knowledge with the retention of previously learned knowledge to incrementally learn tasks from a data stream without catastrophic forgetting. While leveraging pretrained models has significantly advanced continual learning, existing methods exhibit a scalability bottleneck when trained sequentially on many tasks, suffering from performance degradation due to inter-task interference and loss of plasticity. Inspired by evidence that sparse fine-tuning achieves performance comparable to full fine-tuning, this paper presents a novel sparsity-driven continual learning framework. Our continual learning method, termed CLARE, operates in two stages: it first identifies a sparse, task-critical parameter mask via a sparsity-inducing objective, then performs mask-constrained fine-tuning by only optimizing parameters selected by the mask. This two-stage sparse adapter mechanism enables all tasks to be accumulated within a shared adapter space while reducing destructive interference across tasks. Extensive experiments demonstrate the scalability of CLARE. On the long task-sequence benchmark Omnibenchmark-1k, CLARE outperforms strong baselines in final accuracy by a large margin, e.g, improving EASE by 4.64% and 13.34% after learning 100 tasks, respectively.

Abstract

Continual Learning

Explore similar work

CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework

Recovery Guarantees for Continual Learning of Dependent Tasks: Memory, Data-Dependent Regularization, and Data-Dependent Weights

Optimal Explicit and Implicit Regularization Strengths in High-Dimensional Continual Linear Regression