SMART Fine-tuning Factor Augmented Neural Lasso

Jinhang Chai; Jianqing Fan; Cheng Gao; Qishuo Yin

stat.MLApr 14, 2026

SMART Fine-tuning Factor Augmented Neural Lasso

Authors: Jinhang Chai, Jianqing Fan, Cheng Gao, Qishuo Yin

Organizations: Department of Operations Research and Financial Engineering, Princeton University

Abstract

Fine-tuning is a widely used strategy for adapting pre-trained models to new tasks, yet its methodology and theoretical properties in high-dimensional nonparametric settings with variable selection have not yet been developed. We propose a source-model-augmented residual tuning (SMART) framework, which incorporates the pre-trained source model as an augmented feature into the target learner and estimates only the residual target-specific component. The approach is widely applicable, from parametric and sparse models to neural networks and blackbox machine learning models. We focus on the development of fine-tuning factor-augmented neural Lasso, resulting in SMART-FAN-Lasso. This transfer-learning framework for high-dimensional nonparametric regression with variable selection simultaneously handles covariate and posterior shifts. We use a low-rank factor structure to manage high-dimensional dependent covariates and a residual tuning decomposition in which the target function is expressed as a function of source model and other target-specific variables, thereby reducing the effective complexity of the target task. We derive minimax-optimal excess risk bounds, characterizing the precise conditions, in terms of relative sample sizes and function complexities, under which fine-tuning yields statistical acceleration over single-task learning. Extensive numerical experiments across diverse covariate- and posterior-shift scenarios demonstrate that SMART-FAN-Lasso consistently outperforms standard baselines and achieves near-oracle performance even under severe target sample size constraints, empirically validating the derived rates.

Figures & tables

Figure 1: Transfer learning of eigenspaces under covariate shift. The space alignment is measured by $\nu_{\min}(p^{-1}(\widehat{\bm{W}}^{*})^{T}\bm{B}^{Q})$ , a critical quantity for eigenspace learning via diversified projection. The plot is based on 100 simulations. It compares transfer factor estimation $\widehat{\bm{W}}^{\mbox{\scriptsize TL}}$ with $n^{Q}=7$ (solid blue), target-only factor estimation $\widehat{\bm{W}}^{Q}$ (solid yellow for $n^{Q}=7$ , hollow yellow for $n^{Q}=10$ ), and source-only factor estimation $\widehat{\bm{W}}^{P}$ (solid green).

Figure 2: Method Comparison: Target RMSE (with 95% CI) vs. Target Sample Size ( $n_{\text{train}}^{Q}$ ). The plot compares SMART-FAN-Lasso, SMART-Vanilla-NN, source or target-only baseline models (FAST-NN and Vanilla-NN), and the Oracle-NN.

	FAN-Lasso	FT-Vanilla-NN	FAST-NN (Source Only)	Vanilla-NN (Source Only)	Vanilla-NN (Target Only)	FAST-NN (Target Only)
RMSE $\pm$ SE	$\mathbf{0.1333\pm 0.0088}$	$0.1387\pm 0.0104$	$0.1383\pm 0.0088$	$0.1422\pm 0.0101$	$0.1431\pm 0.0111$	$0.1574\pm 0.0104$

Table 1: Real Data Application: Prediction Performance on Communities and Crime Dataset (Lower-Density Communities). Methods are ordered by RMSE. RMSE and standard error (SE) are computed over 10 simulation replicates.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Category	Variables
Demographics & Population (12)	population, householdsize, racepctblack, racePctWhite, racePctAsian, racePctHisp, agePct12t21, agePct12t29, agePct16t24, agePct65up, numbUrban, pctUrban
Socio-Economic (26)	medIncome, pctWWage, pctWFarmSelf, pctWInvInc, pctWSocSec, pctWPubAsst, pctWRetire, medFamInc, perCapInc, whitePerCap, blackPerCap, indianPerCap, AsianPerCap, OtherPerCap, HispPerCap, NumUnderPov, PctPopUnderPov, PctLess9thGrade, PctNotHSGrad, PctBSorMore, PctUnemployed, PctEmploy, PctEmplManu, PctEmplProfServ, PctOccupManu, PctOccupMgmtProf
Family Structure (13)	MalePctDivorce, MalePctNevMarr, FemalePctDiv, TotalPctDiv, PersPerFam, PctFam2Par, PctKids2Par, PctYoungKids2Par, PctTeen2Par, PctWorkMomYoungKids, PctWorkMom, NumIlleg, PctIlleg
Immigration (16)	NumImmig, PctImmigRecent, PctImmigRec5, PctImmigRec8, PctImmigRec10, PctRecentImmig, PctRecImmig5, PctRecImmig8, PctRecImmig10, PctSpeakEnglOnly, PctNotSpeakEnglWell, PctForeignBorn, PctBornSameState, PctSameHouse85, PctSameCity85, PctSameState85
Housing (26)	PctLargHouseFam, PctLargHouseOccup, PersPerOccupHous, PersPerOwnOccHous, PersPerRentOccHous, PctPersOwnOccup, PctPersDenseHous, PctHousLess3BR, MedNumBR, HousVacant, PctHousOccup, PctHousOwnOcc, PctVacantBoarded, PctVacMore6Mos, MedYrHousBuilt, PctHousNoPhone, PctWOFullPlumb, OwnOccLowQuart, OwnOccMedVal, OwnOccHiQuart, RentLowQ, RentMedian, RentHighQ, MedRent, MedRentPctHousInc, MedOwnCostPctInc, MedOwnCostPctIncNoMtg
Law Enforcement & Geography (7)	LemasSwornFT, LemasSwFTPerPop, LemasSwFTFieldOps, LemasSwFTFieldPerPop, LemasTotalReq, LemasPctPolicOnPatr, LemasGangUnitDeploy

Appendix

Table 2: Cleaned Dataset Attributes by Category

Fine-Tuning

Fine-tuning adapts pretrained models to new tasks, domains, and behaviors by updating some or all of their parameters.

See all 250 papers

SMART: A Spectral Transfer Approach to Multi-Task Learning
Apr 22, 2026Boxin Zhao, Mladen Kolar, Jinchi LvMulti-Task Learning Transfer Learning
University of Chicago · University of Southern California
Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning
Oct 16, 2026Yalda Shabanzadeh, Diyuan Wu, Marco MondelliTransfer Learning Ridge Regression
Institute of Science and Technology Austria (ISTA).
When is Fine-Tuning Scalable? Certified Recovery from Noisy Data and Linear-Cost Fine-Tuning on Structured Task Shifts
Aug 31, 2025Anastasis Kratsios, Tin Sum Cheng, Bum Jun Kim +1Fine-Tuning OOD Generalization
Department of Mathematics, McMaster University, Canada · Department of Mathematics and Computer Science, University of Basel, Switzerland · Department of Statistical Sciences, University of Toronto, Canada +1

Apr 22, 2026J/K move · Enter open · S save

SMART: A Spectral Transfer Approach to Multi-Task Learning

Boxin Zhao, Mladen Kolar, Jinchi Lv

University of Chicago · University of Southern California

Multi-Task Learning Transfer Learning Statistical Learning Theory

Open paper AskPDF arXiv

Multi-task learning is effective for related applications, but its performance can deteriorate when the target sample size is small. Transfer learning can borrow strength from related studies; yet, many existing methods rely on restrictive bounded-difference assumptions between the source and target models. We propose SMART, a spectral transfer method for multi-task linear regression that instead assumes spectral similarity: the target left and right singular subspaces lie within the corresponding source subspaces and are sparsely aligned with the source singular bases. Such an assumption is natural when studies share latent structures and enables transfer beyond the bounded-difference settings. SMART estimates the target coefficient matrix through structured regularization that incorporates spectral information from a source study. Importantly, it requires only a fitted source model rather than the raw source data, making it useful when data sharing is limited. Although the optimization problem is nonconvex, we develop a practical ADMM-based algorithm. We establish general, non-asymptotic error bounds and a minimax lower bound in the noiseless-source regime. Under additional regularity conditions, these results yield near-minimax Frobenius error rates up to logarithmic factors. Simulations confirm improved estimation accuracy and robustness to negative transfer, and analysis of multi-modal single-cell data demonstrates better predictive performance. The Python implementation of SMART, along with the code to reproduce all experiments in this paper, is publicly available at https://github.com/boxinz17/smart.

Abstract

Fine-Tuning

Explore similar work

SMART: A Spectral Transfer Approach to Multi-Task Learning

Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning

When is Fine-Tuning Scalable? Certified Recovery from Noisy Data and Linear-Cost Fine-Tuning on Structured Task Shifts