Sparsity Pattern

Sparsity patterns, the non-random distributions of zero and non-zero elements in data structures like matrices or neural network activations, are a central focus in modern machine learning and data analysis. Current research emphasizes developing algorithms and architectures that efficiently leverage sparsity for improved computational efficiency, reduced memory footprint, and enhanced model performance, particularly in high-dimensional settings. This includes work on distributed algorithms for sparse model fitting, techniques for inducing and exploiting sparsity in large language models and vision transformers, and methods for optimizing hardware and software to handle diverse sparsity patterns. Understanding and harnessing sparsity patterns is crucial for advancing the scalability and efficiency of machine learning across various applications.

Papers

November 1, 2024

Inducing Semi-Structured Sparsity by Masking for Efficient Model Inference in Convolutional Networks
David A. Danhofer
Convolutional Neural Network Deep Convolutional Neural Network Scientific Inference Structured Sparsity Convolution Kernel Efficient Model Sparsity Pattern

August 30, 2024

Minimax and Communication-Efficient Distributed Best Subset Selection with Oracle Property
Jingguo Lan, Hongmei Lin, Xueqin Wang
Test Oracle Minimax Optimality Sparse Regression Sparsity Pattern Surrogate Likelihood Best Subset Sparse Estimation

August 27, 2024

Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations
Yize Zhao, Tina Behnia, Vala Vakilian, Christos Thrampoulidis
Large Language Model Sparse Label Sparsity Pattern Model Representation Next Token Context Embeddings

July 10, 2024

Uncovering Layer-Dependent Activation Sparsity Patterns in ReLU Transformers
Cody Wild, Jesper Anderson
Multi Layer Sparsity Increase ReLU Layer Activation Sparsity Sparsity Pattern Token Sparsification

June 24, 2024

ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models
Yash Akhauri, Ahmed F AbouElhamayed, Jordan Dotzel, Zhiru Zhang, Alexander M Rush, Safeen Huda, Mohamed S Abdelfattah
Large Language Model Sparsity Pattern Contextual Sparsity

June 15, 2024

SparseRadNet: Sparse Perception Neural Network on Subsampled Radar Data
Jialong Wu, Mirko Meuter, Markus Schoeler, Matthias Rottmann
Radar Point Cloud Radar Data Sparsity Pattern Radar Based Perception Sparse Perception

June 14, 2024

Misam: Using ML in Dataflow Selection of Sparse-Sparse Matrix Multiplication
Sanjali Yadav, Bahar Asgari
Machine Learning Hardware Accelerator Sparse Matrix Dataflow Architecture Sparsity Pattern Sparse Matrix Multiplication

September 30, 2023

The Sparsity Roofline: Understanding the Hardware Limits of Sparse Neural Networks
Cameron Shinn, Collin McCarthy, Saurav Muralidharan, Muhammad Osama, John D. Owens
Sparsity Increase Sparse Neural Network Sparsity Pattern Sparse Kernel Hardware Constraint

June 5, 2023

Does a sparse ReLU network training problem always admit an optimum?
Quoc-Tung Le, Elisa Riccietti, Rémi Gribonval
Near Optimal Shallow ReLU Sparsity Pattern

June 1, 2023

Attribute-Efficient PAC Learning of Low-Degree Polynomial Threshold Functions with Nasty Noise
Shiwei Zeng, Jie Shen
Industrial Disturbing Noise PAC Learning Sparsity Pattern Feature Sparsity Threshold Function

January 13, 2023

GOHSP: A Unified Framework of Graph and Optimization-based Heterogeneous Structured Pruning for Vision Transformer
Miao Yin, Burak Uzkent, Yilin Shen, Hongxia Jin, Bo Yuan
Vision Transformer Graph Drawing Unified Framework Vision Task Structured Pruning Sparsity Pattern Structured Pruning Method

January 3, 2023

A Theory of I/O-Efficient Sparse Neural Network Inference
Niels Gleinig, Tal Ben-Nun, Torsten Hoefler
Machine Learning Model Theoretical Understanding Hardware Design Neural Network Inference Sparsity Pattern High Bandwidth Memory

December 15, 2022

Learning Sparsity and Randomness for Data-driven Low Rank Approximation
Tiejin Chen, Yicheng Tao
Low Rank Approximation Inherent Randomness Sparsity Pattern Sparsity Aware

November 21, 2022

Multiresolution kernel matrix algebra
H. Harbrecht, M. Multerer, O. Schenk, Ch. Schwab
Many Sparse Sparse Matrix Kernel Matrix Sparsity Pattern

September 17, 2022

Improved Generalization Bound and Learning of Sparsity Patterns for Data-Driven Low-Rank Approximation
Shinsaku Sakaue, Taihei Oki
LeArning Abstract Data Driven Average Approximation Generalization Bound Low Rank Approximation Sparsity Pattern Rank $K$ Approximation

June 30, 2022

Sparse Periodic Systolic Dataflow for Lowering Latency and Power Dissipation of Convolutional Neural Network Accelerators
Jung Hwan Heo, Arash Fayyazi, Amirhossein Esmaili, Massoud Pedram
CNN Accelerator Sparsity Pattern Multi Accelerator Deep Learning Compiler Efficient Compilation Systolic Array Systolic Complex Power Dissipation

June 14, 2022

Learning the Structure of Large Networked Systems Obeying Conservation Laws
Anirudh Rayas, Rajasekhar Anguluri, Gautam Dasarathy
LeArning Abstract Inner Structure Opinion Dynamic Conservation Law Network Dynamic Sparsity Pattern Networked System Sparsity Structure

June 10, 2022

Provable Guarantees for Sparsity Recovery with Deterministic Missing Data Patterns
Chuyang Ke, Jean Honorio
Provable Guarantee Missingness Pattern Sparsity Pattern Imputation Strategy

May 16, 2022

DDAC-SpAM: A Distributed Algorithm for Fitting High-dimensional Sparse Additive Models with Feature Division and Decorrelation
Yifan He, Ruiyang Wu, Yong Zhou, Yang Feng
Practical Algorithm High Dimensional Actor Critic Content Based Feature Additive Model Feature Decorrelation Sparsity Pattern

Sparsity Pattern

Papers

Inducing Semi-Structured Sparsity by Masking for Efficient Model Inference in Convolutional Networks

Minimax and Communication-Efficient Distributed Best Subset Selection with Oracle Property

Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations

Uncovering Layer-Dependent Activation Sparsity Patterns in ReLU Transformers

ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models

SparseRadNet: Sparse Perception Neural Network on Subsampled Radar Data

Misam: Using ML in Dataflow Selection of Sparse-Sparse Matrix Multiplication

The Sparsity Roofline: Understanding the Hardware Limits of Sparse Neural Networks

Does a sparse ReLU network training problem always admit an optimum?

Attribute-Efficient PAC Learning of Low-Degree Polynomial Threshold Functions with Nasty Noise

GOHSP: A Unified Framework of Graph and Optimization-based Heterogeneous Structured Pruning for Vision Transformer

A Theory of I/O-Efficient Sparse Neural Network Inference

Learning Sparsity and Randomness for Data-driven Low Rank Approximation

Multiresolution kernel matrix algebra

Improved Generalization Bound and Learning of Sparsity Patterns for Data-Driven Low-Rank Approximation

Sparse Periodic Systolic Dataflow for Lowering Latency and Power Dissipation of Convolutional Neural Network Accelerators

Learning the Structure of Large Networked Systems Obeying Conservation Laws

Provable Guarantees for Sparsity Recovery with Deterministic Missing Data Patterns

DDAC-SpAM: A Distributed Algorithm for Fitting High-dimensional Sparse Additive Models with Feature Division and Decorrelation