Sparse Computation

Sparse computation aims to improve the efficiency of deep learning models by selectively performing computations only on essential parts of the network, leveraging the inherent sparsity in data or model parameters. Current research focuses on developing efficient sparse algorithms and architectures, including sparse Mixture-of-Experts models and optimized activation functions like ReLU², as well as specialized libraries like Scorch for seamless integration into existing deep learning frameworks. This approach offers significant potential for reducing computational costs and energy consumption in various applications, from large language models and automatic speech recognition to graph neural networks and reinforcement learning, while maintaining high accuracy.

Papers

August 29, 2024

PACiM: A Sparsity-Centric Hybrid Compute-in-Memory Architecture via Probabilistic Approximation
Wenlun Zhang, Shimpei Ando, Yung-Chin Chen, Satomi Miyagi, Shinya Takamaeda-Yamazaki, Kentaro Yoshioka
Stochastic Approximation Computational Power Approximate Computing Sparse Architecture Sparse Computation

May 27, 2024

Scorch: A Library for Sparse Deep Learning
Bobby Yan, Alexander J. Root, Trevor Gale, David Broman, Fredrik Kjolstad
Pytorch Model Easy to Use Library Sparse Deep Sparse Computation

February 27, 2024

XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection
Yuanhang Yang, Shiyi Qi, Wenchao Gu, Chaozheng Wang, Cuiyun Gao, Zenglin Xu
Fine Grained Transformer Model Sparse Model Dense Model Expert Selection Sparse Computation

February 6, 2024

ReLU$^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs
Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song, Zhiyuan Liu, Zeyu Mi, Maosong Sun
Large Language Model Activation Function Sparse Activation ReLU Neural Network Sparse Computation

October 19, 2022

RSC: Accelerating Graph Neural Networks Training via Randomized Sparse Computations
Zirui Liu, Shengyuan Chen, Kaixiong Zhou, Daochen Zha, Xiao Huang, Xia Hu
Sparse Operation Sparse Computation

July 5, 2022

Compute Cost Amortized Transformer for Streaming ASR
Yi Xie, Jonathan Macoskey, Martin Radfar, Feng-Ju Chang, Brian King, Ariya Rastrow, Athanasios Mouchtaris, Grant P. Strimel
Automatic Speech Recognition Speech Recognition Neural Network Inference Sparse Computation

January 7, 2022

Neural Network Optimization for Reinforcement Learning Tasks Using Sparse Computations
Dmitry Ivanov, Mikhail Kiselev, Denis Larionov
Reinforcement Learning Neural Network Reinforcement Learning Task Neural State Sparse Computation