Neural Network Compression

Neural network compression aims to reduce the size and computational cost of deep learning models without significant performance loss. Current research focuses on techniques like pruning (removing less important connections), quantization (reducing the precision of weights), knowledge distillation (transferring knowledge from a larger model), and tensor decomposition (factorizing weight matrices), often applied to convolutional neural networks, recurrent neural networks, and transformers. These methods are crucial for deploying large models on resource-constrained devices like mobile phones and embedded systems, enabling broader applications in areas such as real-time image processing, autonomous driving, and medical image analysis. The development of efficient compression algorithms is driving progress in both the theoretical understanding of deep learning and its practical deployment across diverse fields.

Papers

November 28, 2022

AcceRL: Policy Acceleration Framework for Deep Reinforcement Learning
Hongjie Zhang
Reinforcement Learning Deep Reinforcement Learning Policy Learning Neural Network Compression Parallel Training

November 17, 2022

VeriCompress: A Tool to Streamline the Synthesis of Verified Robust Compressed Neural Networks from Scratch
Sawinder Kaur, Yi Xiao, Asif Salekin
Critical Synthesis BRIO Tool Scratch Project Neural Network Compression Robustness Guarantee Safety Critical Application

October 17, 2022

Approximating Continuous Convolutions for Deep Network Compression
Theo W. Costain, Victor Adrian Prisacariu
Convolutional Neural Network Neural Network Compression Convolutional Filter Continuous Convolution Traditional Convolution

October 14, 2022

Neural Network Compression by Joint Sparsity Promotion and Redundancy Reduction
Tariq M. Khan, Syed S. Naqvi, Antonio Robles-Kelly, Erik Meijering
Convolutional Neural Network Model Neural Network Compression Sparse Structure Feature Redundancy Sparsity Constraint Redundancy Reduction

August 20, 2022

Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks
Rajiv Movva, Jinhao Lei, Shayne Longpre, Ajay Gupta, Chris DuBois
Knowledge Distillation Multiplicative Size Scaling Neural Network Compression Model Size Natural Language Task Magnitude Pruning BERT Architecture Functional Compression

July 22, 2022

Quantized Sparse Weight Decomposition for Neural Network Compression
Andrey Kuzmin, Mart van Baalen, Markus Nagel, Arash Behboodi
Vector Quantization Neural Network Compression Sparse Principal Component Analysis Sparse Factorization Weight Decomposition

July 4, 2022

Quantum Neural Network Compression
Zhirui Hu, Peiyan Dong, Zhepeng Wang, Youzuo Lin, Yanzhi Wang, Weiwen Jiang
Quantum Computer Variational Quantum Circuit Neural Network Compression Near Term Quantum Noisy Quantum

July 3, 2022

FasterAI: A Lightweight Library for Creating Sparse Neural Networks
Nathan Hubens
Pytorch Model AI Model Neural Network Compression Sparse Neural Network

June 15, 2022

Hardening DNNs against Transfer Attacks during Network Compression using Greedy Adversarial Pruning
Jonah O'Brien Weiss, Tiago Alves, Sandip Kundu
Adversarial Robustness DNN Framework Adversarial Input Neural Network Compression Network Compression Transfer Attack Adversarial Pruning

June 11, 2022

A Theoretical Understanding of Neural Network Compression from Sparse Linear Approximation
Wenjing Yang, Ganghua Wang, Jie Ding, Yuhong Yang
Theoretical Understanding Model Compression Linear Compression Neural Network Compression Sparse Function Sparsity Search

June 7, 2022

Neural Network Compression via Effective Filter Analysis and Hierarchical Pruning
Ziqi Zhou, Li Lian, Yilong Yin, Ze Wang
Deep Network Neural Network Compression Network Pruning Network Compression Hierarchical Pruning

May 24, 2022

Compressing Deep Graph Neural Networks via Adversarial Knowledge Distillation
Huarui He, Jie Wang, Zhanqiu Zhang, Feng Wu
Knowledge Distillation Neural Network Compression Deep Graph Adversarial Knowledge Distillation Deep Graph Model

May 17, 2022

Do Neural Networks Compress Manifolds Optimally?
Sourbh Bhadane, Aaron B. Wagner, Johannes Ballé
Low Dimensional Manifold Neural Network Compression

May 11, 2022

Revisiting Random Channel Pruning for Neural Network Compression
Yawei Li, Kamil Adamczewski, Wen Li, Shuhang Gu, Radu Timofte, Luc Van Gool
Neural Network Compression Channel Pruning Channel Selection Random Pruning Channel Importance

February 2, 2022

Approximate Bisimulation Relations for Neural Networks and Application to Assured Neural Network Compression
Weiming Xiang, Zhongzhu Shao
Neural Network Application Proficiency Neural Network Compression Bisimulation Metric

January 17, 2022

Neural Network Compression of ACAS Xu Early Prototype is Unsafe: Closed-Loop Verification through Quantized State Backreachability
Stanley Bak, Hoang-Dung Tran
Collision Avoidance Closed Loop Visual Prototype Neural Network Verification Neural Network Compression

November 19, 2021

Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression
Yuezhou Sun, Wenlong Zhao, Lijun Zhang, Xiao Liu, Hui Guan, Matei Zaharia
Deep Neural Network Joint Representation Neural Network Compression Deep Neural Network Parameter Parametric Representation