Sparsity Regularization

Sparsity regularization aims to improve machine learning models by encouraging solutions with only a few non-zero parameters, enhancing efficiency and generalizability. Current research focuses on developing novel regularization techniques, including those based on tree structures, dropout mechanisms, Gaussian mixtures, and adaptive regularization within iterative algorithms like iterative hard thresholding. These advancements are applied across diverse areas, such as causal inference, natural language processing, inverse problems, and recommendation systems, leading to improved model performance and reduced computational costs. The resulting sparse models offer benefits in terms of interpretability, reduced memory footprint, and faster inference.

Papers

November 4, 2024

A General Recipe for Contractive Graph Neural Networks -- Technical Report
Maya Bechler-Speicher, Moshe Eliasof
Graph Neural Network Technical Report Graph Structured Data Complete Recipe Graph Based Learning Sparsity Regularization

October 29, 2024

A Fresh Look at Generalized Category Discovery through Non-negative Matrix Factorization
Zhong Ji, Shuo Yang, Jingren Liu, Yanwei Pang, Jungong Han
Contrastive Learning Matrix Factorization Non Negative Matrix Factorization Generalized Category Discovery Sparsity Regularization

October 8, 2024

Network Topology Inference from Smooth Signals Under Partial Observability
Chuansen Peng, Hanning Tang, Zhiguo Wang, Xiaojing Shen
Graph Learning Partial Observability Network Structure Large Scale Network First Order Algorithm Smooth Signal Sparsity Regularization

May 30, 2024

Sparsity regularization via tree-structured environments for disentangled representations
Elliot Layne, Jason Hartford, Sébastien Lachapelle, Mathieu Blanchette, Dhanya Sridhar
Latent Variable Disentangled Representation Causal Representation Learning Latent Causal Gene Expression Data Structured Environment Sparsity Regularization

April 15, 2024

LoRA Dropout as a Sparsity Regularizer for Overfitting Control
Yang Lin, Xinyu Ma, Xu Chu, Yujie Jin, Zhibang Yang, Yasha Wang, Hong Mei
Model Overfitting Safe LoRA Sparsity Regularization LoRA Based

January 29, 2024

Learning a Gaussian Mixture for Sparsity Regularization in Inverse Problems
Giovanni S. Alberti, Luca Ratti, Matteo Santacesaria, Silvia Sciutto
LeArning Abstract Inverse Problem Sparsity Increase Group Lasso Sparsity Regularization

September 23, 2023

Gaining the Sparse Rewards by Exploring Lottery Tickets in Spiking Neural Network
Hao Cheng, Jiahang Cao, Erjia Xiao, Mengshu Sun, Renjing Xu
Spiking Neural Network Sparse Reward Lottery Ticket Lottery Ticket Hypothesis Energy Efficient Neural Network Sparsity Regularization

July 14, 2023

Learning Sparse Neural Networks with Identity Layers
Mingjian Ni, Guangyao Chen, Xiawu Zheng, Peixi Peng, Li Yuan, Yonghong Tian
Sparse Network Sparse Neural Network Network Sparsity Sparsity Regularization Sparse Network Training

June 27, 2022

AdaSparse: Learning Adaptively Sparse Structures for Multi-Domain Click-Through Rate Prediction
Xuanhua Yang, Xiaoyu Peng, Penghui Wei, Shaoguo Liu, Liang Wang, Bo Zheng
Click Through Rate Prediction Sparse Structure Extensible Framework Sparsity Regularization

April 11, 2022

Iterative Hard Thresholding with Adaptive Regularization: Sparser Solutions Without Sacrificing Runtime
Kyriakos Axiotis, Maxim Sviridenko
Sparsity Increase Runtime Analysis Adaptive Regularization Sparse Solution Sparsity Level Sparsity Regularization Hard Thresholding

February 20, 2022

Alternative design of DeepPDNet in the context of image restoration
Mingyuan Jiu, Nelly Pustelnik
Context Information Deep Network Image Restoration Primal Dual Lloyd Algorithm Novel Design Sparsity Regularization

February 15, 2022

Accelerating Non-Negative and Bounded-Variable Linear Regression Algorithms with Safe Screening
Cassio F. Dantas, Emmanuel Soubies, Cédric Févotte
Machine Learning Regression Problem Regression Algorithm Sparsity Regularization Safe Screening

January 26, 2022

Sparsity Regularization For Cold-Start Recommendation
Aksheshkumar Ajaykumar Shah, Hemanth Venkateswara
Generative Adversarial Network Collaborative Filtering User Behavior Cold Start Recommendation Sparsity Regularization

December 14, 2021

Unsupervised feature selection via self-paced learning and low-redundant regularization
Weiyi Li, Hongmei Chen, Tianrui Li, Jihong Wan, Binbin Sang
Discriminative Feature Subspace Learning Regularization Term Unsupervised Feature Selection Self Paced Learning Sparsity Regularization

Sparsity Regularization

Papers

A General Recipe for Contractive Graph Neural Networks -- Technical Report

A Fresh Look at Generalized Category Discovery through Non-negative Matrix Factorization

Network Topology Inference from Smooth Signals Under Partial Observability

Sparsity regularization via tree-structured environments for disentangled representations

LoRA Dropout as a Sparsity Regularizer for Overfitting Control

Learning a Gaussian Mixture for Sparsity Regularization in Inverse Problems

Gaining the Sparse Rewards by Exploring Lottery Tickets in Spiking Neural Network

Learning Sparse Neural Networks with Identity Layers

AdaSparse: Learning Adaptively Sparse Structures for Multi-Domain Click-Through Rate Prediction

Iterative Hard Thresholding with Adaptive Regularization: Sparser Solutions Without Sacrificing Runtime

Alternative design of DeepPDNet in the context of image restoration

Accelerating Non-Negative and Bounded-Variable Linear Regression Algorithms with Safe Screening

Sparsity Regularization For Cold-Start Recommendation

Unsupervised feature selection via self-paced learning and low-redundant regularization