SMART Fine-tuning Factor Augmented Neural Lasso

Jinhang Chai; Jianqing Fan; Cheng Gao; Qishuo Yin

stat.MLApr 14, 2026

SMART Fine-tuning Factor Augmented Neural Lasso

Authors: Jinhang Chai, Jianqing Fan, Cheng Gao, Qishuo Yin

Organizations: Department of Operations Research and Financial Engineering, Princeton University

Abstract

Fine-tuning is a widely used strategy for adapting pre-trained models to new tasks, yet its methodology and theoretical properties in high-dimensional nonparametric settings with variable selection have not yet been developed. We propose a source-model-augmented residual tuning (SMART) framework, which incorporates the pre-trained source model as an augmented feature into the target learner and estimates only the residual target-specific component. The approach is widely applicable, from parametric and sparse models to neural networks and blackbox machine learning models. We focus on the development of fine-tuning factor-augmented neural Lasso, resulting in SMART-FAN-Lasso. This transfer-learning framework for high-dimensional nonparametric regression with variable selection simultaneously handles covariate and posterior shifts. We use a low-rank factor structure to manage high-dimensional dependent covariates and a residual tuning decomposition in which the target function is expressed as a function of source model and other target-specific variables, thereby reducing the effective complexity of the target task. We derive minimax-optimal excess risk bounds, characterizing the precise conditions, in terms of relative sample sizes and function complexities, under which fine-tuning yields statistical acceleration over single-task learning. Extensive numerical experiments across diverse covariate- and posterior-shift scenarios demonstrate that SMART-FAN-Lasso consistently outperforms standard baselines and achieves near-oracle performance even under severe target sample size constraints, empirically validating the derived rates.

Figures & tables

Figure 1: Transfer learning of eigenspaces under covariate shift. The space alignment is measured by $\nu_{\min}(p^{-1}(\widehat{\bm{W}}^{*})^{T}\bm{B}^{Q})$ , a critical quantity for eigenspace learning via diversified projection. The plot is based on 100 simulations. It compares transfer factor estimation $\widehat{\bm{W}}^{\mbox{\scriptsize TL}}$ with $n^{Q}=7$ (solid blue), target-only factor estimation $\widehat{\bm{W}}^{Q}$ (solid yellow for $n^{Q}=7$ , hollow yellow for $n^{Q}=10$ ), and source-only factor estimation $\widehat{\bm{W}}^{P}$ (solid green).

Figure 2: Method Comparison: Target RMSE (with 95% CI) vs. Target Sample Size ( $n_{\text{train}}^{Q}$ ). The plot compares SMART-FAN-Lasso, SMART-Vanilla-NN, source or target-only baseline models (FAST-NN and Vanilla-NN), and the Oracle-NN.

	FAN-Lasso	FT-Vanilla-NN	FAST-NN (Source Only)	Vanilla-NN (Source Only)	Vanilla-NN (Target Only)	FAST-NN (Target Only)
RMSE $\pm$ SE	$\mathbf{0.1333\pm 0.0088}$	$0.1387\pm 0.0104$	$0.1383\pm 0.0088$	$0.1422\pm 0.0101$	$0.1431\pm 0.0111$	$0.1574\pm 0.0104$

Table 1: Real Data Application: Prediction Performance on Communities and Crime Dataset (Lower-Density Communities). Methods are ordered by RMSE. RMSE and standard error (SE) are computed over 10 simulation replicates.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Category	Variables
Demographics & Population (12)	population, householdsize, racepctblack, racePctWhite, racePctAsian, racePctHisp, agePct12t21, agePct12t29, agePct16t24, agePct65up, numbUrban, pctUrban
Socio-Economic (26)	medIncome, pctWWage, pctWFarmSelf, pctWInvInc, pctWSocSec, pctWPubAsst, pctWRetire, medFamInc, perCapInc, whitePerCap, blackPerCap, indianPerCap, AsianPerCap, OtherPerCap, HispPerCap, NumUnderPov, PctPopUnderPov, PctLess9thGrade, PctNotHSGrad, PctBSorMore, PctUnemployed, PctEmploy, PctEmplManu, PctEmplProfServ, PctOccupManu, PctOccupMgmtProf
Family Structure (13)	MalePctDivorce, MalePctNevMarr, FemalePctDiv, TotalPctDiv, PersPerFam, PctFam2Par, PctKids2Par, PctYoungKids2Par, PctTeen2Par, PctWorkMomYoungKids, PctWorkMom, NumIlleg, PctIlleg
Immigration (16)	NumImmig, PctImmigRecent, PctImmigRec5, PctImmigRec8, PctImmigRec10, PctRecentImmig, PctRecImmig5, PctRecImmig8, PctRecImmig10, PctSpeakEnglOnly, PctNotSpeakEnglWell, PctForeignBorn, PctBornSameState, PctSameHouse85, PctSameCity85, PctSameState85
Housing (26)	PctLargHouseFam, PctLargHouseOccup, PersPerOccupHous, PersPerOwnOccHous, PersPerRentOccHous, PctPersOwnOccup, PctPersDenseHous, PctHousLess3BR, MedNumBR, HousVacant, PctHousOccup, PctHousOwnOcc, PctVacantBoarded, PctVacMore6Mos, MedYrHousBuilt, PctHousNoPhone, PctWOFullPlumb, OwnOccLowQuart, OwnOccMedVal, OwnOccHiQuart, RentLowQ, RentMedian, RentHighQ, MedRent, MedRentPctHousInc, MedOwnCostPctInc, MedOwnCostPctIncNoMtg
Law Enforcement & Geography (7)	LemasSwornFT, LemasSwFTPerPop, LemasSwFTFieldOps, LemasSwFTFieldPerPop, LemasTotalReq, LemasPctPolicOnPatr, LemasGangUnitDeploy

Appendix

Table 2: Cleaned Dataset Attributes by Category

Fine-Tuning

Fine-tuning adapts pretrained models to new tasks, domains, and behaviors by updating some or all of their parameters.

See all 250 papers

Apr 22, 2026

SMART: A Spectral Transfer Approach to Multi-Task Learning

Multi-task learning is effective for related applications, but its performance can deteriorate when the target sample size is small. Transfer learning can borrow strength from related studies; yet, many existing methods rely on restrictive bounded-difference assumptions between the source and target models. We propose SMART, a spectral transfer method for multi-task linear regression that instead assumes spectral similarity: the target left and right singular subspaces lie within the corresponding source subspaces and are sparsely aligned with the source singular bases. Such an assumption is natural when studies share latent structures and enables transfer beyond the bounded-difference settings. SMART estimates the target coefficient matrix through structured regularization that incorporates spectral information from a source study. Importantly, it requires only a fitted source model rather than the raw source data, making it useful when data sharing is limited. Although the optimization problem is nonconvex, we develop a practical ADMM-based algorithm. We establish general, non-asymptotic error bounds and a minimax lower bound in the noiseless-source regime. Under additional regularity conditions, these results yield near-minimax Frobenius error rates up to logarithmic factors. Simulations confirm improved estimation accuracy and robustness to negative transfer, and analysis of multi-modal single-cell data demonstrates better predictive performance. The Python implementation of SMART, along with the code to reproduce all experiments in this paper, is publicly available at https://github.com/boxinz17/smart.

Boxin Zhao, Mladen Kolar, Jinchi Lv

University of Chicago · University of Southern California

Multi-Task Learning Transfer Learning Statistical Learning Theory

Ask PDF

Oct 16, 2026

Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning

Pretrained models are often reused across downstream tasks, motivating the question of whether this transfer learning procedure can provide a substantial performance improvement, namely, in terms of the scaling law. We address this question in the prototypical setting of high-dimensional regression, where (i) a pretrained source model is trained via ridge regression, and (ii) a fine-tuned target model is regularized via the

L_2

distance from the source estimator. First, we establish a deterministic equivalent for the risk of this two-step procedure under both model and covariance shift. Next, by specializing the deterministic equivalent to a shared regression function and commuting power-law covariance spectra, we identify a regime where the resulting scaling law achieves the minimax-optimal exponent. Notably, in this regime, the optimal source regularization depends on source data only, indicating its universality across all targets. Furthermore, we show that the same optimal rate is achieved by performing the fine-tuning step via one-pass stochastic gradient descent. We support our theoretical analysis via experiments on both synthetic settings and language models.

Yalda Shabanzadeh, Diyuan Wu, Marco Mondelli

Institute of Science and Technology Austria (ISTA).

Transfer Learning Ridge Regression High-Dimensional Linear Regression Minimax Estimation

Ask PDF

Aug 31, 2025

When is Fine-Tuning Scalable? Certified Recovery from Noisy Data and Linear-Cost Fine-Tuning on Structured Task Shifts

Transformers often transfer across related tasks with little fine-tuning; we study this question in noisy nonparametric regression for continuous maps

f,g:[0,1]^d\to\mathbb{R}^D

, representing the source and shifted tasks, using a single-query, single-head transformer

\mathcal{T}(\cdot)=V\operatorname{Softmax}(-α(\cdot,K))

with

\ell^{\infty}

-based alignment scores

α

. Rather than assume that gradient-based training reaches an empirical risk minimizer, we give a model-specific two-stage procedure certifying source-task recovery and out-of-distribution (OOD) adaptation. With high probability, the resulting attention head recovers

f

to arbitrary accuracy

\varepsilon>0

in every fixed

L^p([0,1]^d)

norm,

1\le p<\infty

, from noisy observations. For Lipschitz targets, its rate is minimax-optimal up to logarithmic factors without assuming oracle access to an asymptotic ERM. If

D=1

and

f

is Pfaffian than the values matrix factors into

AC

with

A

scaling at-most linearly

\mathcal{O}(\varepsilon^{-1})

in the reciprocal approximation error

\varepsilon>0

. Importantly, our training procedure encodes all our identified (potentially) shared geometric structure from

f

only into the values matrix factor

C

and the keys matrix

K

allowing for rapid access, and all

f

-specific structure only into

A

. Our main result concerns the OOD phase; where we identify geometric structures, which if shared by the source

f

and OOD

g

task; then only only the values matrix factor

A

needs to be fine-tuned to guarantee an

L^p

-recovery of

g

. Critically, fine-tuning only

A

is a convex problem which involving an at-most linear number of parameters

\mathcal{O}(\varepsilon^{-1})

; dramatically cutting down on the worst-case scaling of

\mathcal{O}(\varepsilon^{-\mathbf{d}})

unavoidable in the source-training phase.

Anastasis Kratsios, Tin Sum Cheng, Bum Jun Kim +1

Department of Mathematics, McMaster University, Canada · Department of Mathematics and Computer Science, University of Basel, Switzerland · Department of Statistical Sciences, University of Toronto, Canada +1

Fine-Tuning OOD Generalization Transformer Parameter-Efficient Fine-Tuning Nonparametric Regression

Ask PDF

Abstract

Fine-Tuning

Explore similar work

SMART: A Spectral Transfer Approach to Multi-Task Learning

Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning

When is Fine-Tuning Scalable? Certified Recovery from Noisy Data and Linear-Cost Fine-Tuning on Structured Task Shifts