Sparsity Aware

Sparsity-aware techniques aim to improve the efficiency and scalability of machine learning models by leveraging the inherent sparsity present in data or model parameters. Current research focuses on developing algorithms and architectures that exploit sparsity in various contexts, including vision transformers, large language models, and mixture-of-experts models, often employing techniques like pruning, matrix factorization, and query-aware selection to achieve significant speedups and memory savings. These advancements are crucial for deploying large-scale models on resource-constrained devices and accelerating inference times for computationally intensive tasks, impacting fields ranging from computer vision and natural language processing to distributed optimization and signal processing.

Papers

September 15, 2024

ELSA: Exploiting Layer-wise N:M Sparsity for Vision Transformer Acceleration
Ning-Chi Huang, Chi-Chih Chang, Wei-Cheng Lin, Endri Taka, Diana Marculescu, Kai-Chiang Wu
Layer Wise Sparse Matrix Multiplication Sparsity Level Sparsity Aware Sparse Accelerator

June 16, 2024

Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han
Self Attention Inference Latency Long Context Large Language Model KV Cache Long Context LLM QueST Sparsity Aware

April 23, 2024

Estimation Network Design framework for efficient distributed optimization
Mattia Bianchi, Sergio Grammatico
Optimization Purpose Sparsity Increase Estimation Problem Sparsity Aware Estimation Network

October 29, 2023

SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models
Zhixu Du, Shiyu Li, Yuhao Wu, Xiangyu Jiang, Jingwei Sun, Qilin Zheng, Yongkai Wu, Ang Li, Hai "Helen" Li, Yiran Chen
High Efficiency Mixture of Expert Potential Scalability Efficient Inference Efficient Model Model Capacity Sparsity Aware

September 15, 2023

Sparsity-Aware Distributed Learning for Gaussian Processes with Linear Multiple Kernel
Richard Cornelius Suwandi, Zhidi Lin, Feng Yin, Zhiguo Wang, Sergios Theodoridis
LeArning Abstract Gaussian Process Hyperparameter Optimization Kernel Learning Sparsity Aware Linear Kernel Spectral Mixture

June 25, 2023

Low-Rank Prune-And-Factorize for Language Model Compression
Siyu Ren, Kenny Q. Zhu
Fine Tuning Matrix Factorization PLM Based Factorization Based Sparsity Aware Language Model Compression

April 3, 2023

Learning Sparsity of Representations with Discrete Latent Variables
Zhao Xu, Daniel Onoro Rubio, Giuseppe Serra, Mathias Niepert
Meaningful Representation Sparse Representation Sparse Structure Discrete Latent Variable Sparsity Aware Sparse Embeddings

December 15, 2022

Learning Sparsity and Randomness for Data-driven Low Rank Approximation
Tiejin Chen, Yicheng Tao
Low Rank Approximation Inherent Randomness Sparsity Pattern Sparsity Aware

November 14, 2022

SNIPER Training: Single-Shot Sparse Training for Text-to-Speech
Perry Lam, Huayun Zhang, Nancy F. Chen, Berrak Sisman, Dorien Herremans
Training Data Text to Speech Speech to Text Sparse Training High Sparsity Dense Training Sparsity Aware

May 28, 2022

Rethinking Bayesian Learning for Data Analysis: The Art of Prior and Inference in Sparsity-Aware Modeling
Lei Cheng, Feng Yin, Sergios Theodoridis, Sotirios Chatzis, Tsung-Hui Chang
Scientific Inference Variational Inference Bayesian Approach Bayesian Learning Prior Correction Sparsity Aware Bayesian Tensor

April 16, 2022

Robust PCA Unrolling Network for Super-resolution Vessel Extraction in X-ray Coronary Angiography
Binjie Qin, Haohao Mao, Yiming Liu, Jun Zhao, Yisong Lv, Yueqi Zhu, Song Ding, Xu Chen
Super Resolution Robust Principal Component Analysis Deep Unrolled Sparsity Aware X Ray Coronary Angiography Image