Adaptive Sparsity

Adaptive sparsity focuses on developing methods that automatically adjust the number of active parameters or features in a model during training or operation, aiming for improved efficiency and performance without sacrificing accuracy. Current research emphasizes the application of adaptive sparsity within various model architectures, including transformers, neural networks, and graph generation models, often employing techniques like pruning, low-rank approximations, and automatic relevance determination. This approach is proving valuable across diverse fields, enhancing the efficiency of time series forecasting, portfolio optimization, and anomaly detection in complex systems like IoT networks, while also improving the interpretability and robustness of models.

Papers

November 4, 2024

Adaptive Sparse Allocation with Mutual Choice & Feature Choice Sparse Autoencoders
Kola Ayonrinde
Supervised Autoencoder Sparse Autoencoders Sparse Feature Parameter Efficient Sparsity Adaptive Sparsity Sparsity Tradeoff

September 10, 2024

SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation
Teng Hu, Jiangning Zhang, Ran Yi, Hongrui Huang, Yabiao Wang, Lizhuang Ma
Diffusion Model Pre Trained Model Adaptation Concern Pre Trained Diffusion Model Large Pre Trained Model Efficient Fine Tuning Adaptive Sparsity

May 13, 2024

Autonomous Sparse Mean-CVaR Portfolio Optimization
Yizun Lin, Yangyu Zhang, Zhao-Rong Lai, Cheng Li
Portfolio Optimization Optimal Portfolio Adaptive Sparsity Mean CVaR

February 27, 2024

CGGM: A conditional graph generation model with adaptive sparsity for node anomaly detection in IoT networks
Xianshi Su, Munan Li, Tongbang Jiang, Hao Long
Generative Model Dynamic Graph Graph Generation Internet of Thing Network Feature Generation Anomaly Node Adaptive Sparsity Conditional Graph Generation

May 29, 2023

HySST: A Stable Sparse Rapidly-Exploring Random Trees Optimal Motion Planning Algorithm for Hybrid Dynamical Systems
Nan Wang, Ricardo G. Sanfelice
Local Path Planning Hybrid System Hybrid Dynamical System Optimal Motion Planning Tensegrity Robot Random Tree Adaptive Sparsity

May 28, 2023

Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers
Zahra Atashgahi, Mykola Pechenizkiy, Raymond Veldhuis, Decebal Constantin Mocanu
Training Data Transformer Megatron Decepticons Sparse Neural Network Sparse Training Efficient Time Series Forecasting Forecasting Analysis Sparsity Tradeoff Adaptive Sparsity

January 31, 2023

Adaptive sparseness for correntropy-based robust regression via automatic relevance determination
Yuanhao Li, Badong Chen, Okito Yamashita, Natsue Yoshimura, Yasuharu Koike
Robust Regression Relevance Modeling Adaptive Sparsity Regularization Hyperparameters

October 27, 2022

Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost
Sungjun Cho, Seonwoo Min, Jinwoo Kim, Moontae Lee, Honglak Lee, Seunghoon Hong
Transformer Megatron Decepticons Self Attention Human Attention Hidden CoST Sparse Attention Stochastic Block Model Attention Mask Strip Transformer Adaptive Sparsity Mixed Membership Stochastic Block Model

September 30, 2022

Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition
Chendong Zhao, Jianzong Wang, Wen qi Wei, Xiaoyang Qu, Haoqian Wang, Jing Xiao
Sparse Attention Multi Head Attention Mechanism Transformer Based Automatic Speech Recognition Attention Regularization Position Aware Attention Adaptive Sparsity

July 24, 2022

AMS-Net: Adaptive Multiscale Sparse Neural Network with Interpretable Basis Expansion for Multiphase Flow Problems
Yating Wang, Wing Tat Leung, Guang Lin
Sparsity Increase Adaptive Learning Sparse Representation Multiphase Flow Sparse Solution Adaptive Sparsity Basis Expansion

March 10, 2022

Cascaded Sparse Feature Propagation Network for Interactive Segmentation
Chuyu Zhang, Chuanyang Hu, Hui Ren, Yongfei Liu, Xuming He
Accurate Segmentation Interactive Segmentation Feature Propagation Adaptive Sparsity

Adaptive Sparsity

Papers

Adaptive Sparse Allocation with Mutual Choice & Feature Choice Sparse Autoencoders

SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation

Autonomous Sparse Mean-CVaR Portfolio Optimization

CGGM: A conditional graph generation model with adaptive sparsity for node anomaly detection in IoT networks

HySST: A Stable Sparse Rapidly-Exploring Random Trees Optimal Motion Planning Algorithm for Hybrid Dynamical Systems

Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers

Adaptive sparseness for correntropy-based robust regression via automatic relevance determination

Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost

Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition

AMS-Net: Adaptive Multiscale Sparse Neural Network with Interpretable Basis Expansion for Multiphase Flow Problems

Cascaded Sparse Feature Propagation Network for Interactive Segmentation