Sparse Linear Regression

Sparse linear regression aims to identify a small subset of relevant features from a high-dimensional dataset to predict a response variable, prioritizing both accuracy and interpretability. Current research emphasizes developing computationally efficient algorithms, such as those based on iterative reweighted least squares, LASSO variations (including contextual and rescaled versions), and techniques leveraging singular value decomposition or other matrix factorization methods, to handle high-dimensionality, correlated features, and outliers. These advancements are crucial for various applications, including model discovery in dynamical systems, biomedical data analysis, and improving the efficiency and robustness of machine learning models. The field also actively explores theoretical limits and computational-statistical tradeoffs, seeking to understand the fundamental barriers to efficient sparse regression.

Papers

November 7, 2024

Statistical-Computational Trade-offs for Greedy Recursive Partitioning Estimators
Yan Shuo Tan, Jason M. Klusowski, Krishnakumar Balasubramanian
Stochastic Gradient Descent Empirical Risk Minimization Linear Estimator Trade Offs Sparse Linear Regression Greedy Learning

October 10, 2024

Theoretical limits of descending $\ell_0$ sparse-regression ML algorithms
Mihailo Stojnic
Maximum Likelihood Optimization Algorithm Deterministic Algorithm Sparse Linear Regression Sparse Optimization Fundamental Limit Random Duality Theory

August 8, 2024

How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regression
Xingwu Chen, Lei Zhao, Difan Zou
Transformer Megatron Decepticons Case Study Context Learning Human Attention Transformer Based Model Multi Head Attention Sparse Linear Regression Head Transformer Context Learner

August 2, 2024

Sparse Linear Regression when Noises and Covariates are Heavy-Tailed and Contaminated by Outliers
Takeyuki Sasai, Hironori Fujisawa
Linear Regression Relevant Covariates Linear Estimator Sparse Linear Regression Real World Noise

June 18, 2024

A variational Bayes approach to debiased inference for low-dimensional parameters in high-dimensional linear regression
Ismaël Castillo, Alice L'Huillier, Kolyan Ray, Luke Travis
High Dimensional Scientific Inference Low Rank Mean Field Sparse Linear Regression Parameter Space Variational Bayes Scalable Variational Mean Field Variational

May 14, 2024

Scalable Sparse Regression for Model Discovery: The Fast Lane to Insight
Matthew Golden
Dynamical System DCU Insight AQ Singular Value Decomposition Sparse Linear Regression Sparse Regression Model Discovery Agnostic Learning Algorithm

February 23, 2024

Lasso with Latents: Efficient Estimation, Covariate Rescaling, and Computational-Statistical Gaps
Jonathan Kelner, Frederic Koehler, Raghu Meka, Dhruv Rohatgi
Sparse Linear Regression Sparse Principal Component Analysis Standard LASSO Efficient Estimation Situated Latents Computational Gap Word Level Rescaling Asset Correlation Localized Sparse Incomplete Sparse Parity

February 22, 2024

Sparse Linear Regression and Lattice Problems
Aparna Gupte, Neekon Vafa, Vinod Vaikuntanathan
Sparse Linear Regression Computational Gap Basis Pursuit Gaussian Design

February 21, 2024

Computational-Statistical Gaps for Improper Learning in Sparse Linear Regression
Rares-Darius Buhai, Jingqiu Ding, Stefan Tiegel
Sparse Linear Regression Sparse Principal Component Analysis Computational Gap Improper Learning

January 9, 2024

Linear Recursive Feature Machines provably recover low-rank matrices
Adityanarayanan Radhakrishnan, Mikhail Belkin, Dmitriy Drusvyatskiy
Feature Learning Auto Encoder Deep Linear Low Rank Matrix Feature Vector Sparse Linear Regression

October 11, 2023

Improved Analysis of Sparse Linear Regression in Local Differential Privacy Model
Liyang Zhu, Meng Ding, Vaneet Aggarwal, Jinhui Xu, Di Wang
Local Differential Privacy Sparse Linear Regression Sparse Optimization Improved Analysis

August 13, 2023

Weighted Sparse Partial Least Squares for Joint Sample and Feature Selection
Wenwen Min, Taosheng Xu, Chris Ding
Multi View Feature Selection Many Sparse Sparse Linear Regression Joint Distribution Partial Least Square

July 29, 2023

Multi-view Sparse Laplacian Eigenmaps for nonlinear Spectral Feature Selection
Gaurav Srivastava, Mahesh Jangid
Feature Selection Graph Laplacian Sparse Linear Regression Laplacian Matrix Laplacian Eigenmaps

June 25, 2023

Joint Learning of Network Topology and Opinion Dynamics Based on Bandit Algorithms
Yu Xing, Xudong Sun, Karl H. Johansson
Multi Armed Bandit Joint Learning Network Topology Opinion Dynamic Sparse Linear Regression

June 22, 2023

Outlier-robust Estimation of a Sparse Linear Model Using Invexity
Adarsh Barik, Jean Honorio
Sparse Linear Regression Standard LASSO Lasso Problem Outlier Sample Sparse Linear Outlier Robust

June 12, 2023

Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization
Jiuling Zhang, Zhiming Ding
Sparse Linear Regression Differentiable Architecture Search Supernet Training Explicit Regularization Information Leakage Generalized Gradient

May 27, 2023

Structured model selection via $\ell_1-\ell_2$ optimization
Xiaofan Lu, Linan Zhang, Hongjin He
Optimization Purpose Time Series Model Selection Sparse Linear Regression Sparse Optimization

May 26, 2023

April 13, 2023

OKRidge: Scalable Optimal k-Sparse Ridge Regression
Jiachang Liu, Sam Rosen, Chudi Zhong, Cynthia Rudin
Nonlinear Dynamic Saddle Point Sparse Linear Regression ADMM Algorithm