Neural Network Compression

Follow this topic Ask about this topic

Neural network compression studies how to represent, train, or run models with fewer parameters and less memory or computation, making them practical on devices with limited resources. Common approaches include pruning weights or neurons, quantization, low-rank and tensor factorization, parameter sharing, and knowledge distillation; some methods also compress activations exchanged during distributed training. Work spans small edge models and microcontrollers as well as large language, vision, and diffusion models, evaluated on tasks such as image classification, language modeling, and video coding. A central challenge is that smaller parameter counts or fewer theoretical operations do not always mean faster inference, and aggressive compression can harm accuracy, robustness, or task-specific behavior.

Latest papers

218 papers28 papers in the last four weeks, up 8% on the four weeks before. 0.2% of all new papers.

Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
Jun 2, 2025Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham +2Communication-Efficient Distributed Training Decentralized Learning
Pluralis Research
Gibbs randomness-compression proposition
May 29, 2025M. SüzenNeural Network Compression
Tensorization is a powerful but underexplored tool for compression and interpretability of neural networks
May 26, 2025Safa Hamreras, Sukhbinder Singh, Román OrúsTensor Networks Neural Network Interpretability
Donostia International Physics Center, Paseo Manuel de Lardizabal 4, E-20018 San Sebastián, Spain · Multiverse Computing, Spadina Ave., Toronto, ON M5T 2C2, Canada · Multiverse Computing, Paseo de Miramón 170, E-20014 San Sebastián, Spain
Neural Stereo Video Compression with Hybrid Disparity Compensation
Apr 29, 2025Shiyin Jiang, Zhenghao Chen, Minghao Han +1Stereo Vision Neural Video Compression
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan 611731, China · School of Computer and Information Sciences, The University of Newcastle, Newcastle, NSW 2300, Australia
Explainable Bayesian deep learning through input-skip Latent Binary Bayesian Neural Networks
Mar 13, 2025Eirik Høyheim, Lars Skaaret-Lund, Solve Sæbø +1Bayesian Neural Networks Uncertainty Quantification
Bioinformatics and applied statistics (BIAS), Norwegian University of Life Sciences, Ås, Norway
Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies
Feb 26, 2025Shaibal Saha, Lanyu XuEfficient ViTs Neural Network Compression
Department of Computer Science and Engineering, Oakland University, , Rochester Hills, 48309, Michigan, USA
TropNNC: Structured Neural Network Compression Using Tropical Geometry
Sep 5, 2024Konstantinos Fotopoulos, Petros Maragos, Panagiotis MisiakosNeural Network Compression Neural Network Pruning
School of ECE, National Technical University of Athens, Greece · Robotics Institute, Athena Research Center, Maroussi, Greece · HERON - Hellenic Robotics Center of Excellence, Athens, Greece +1
Evolutionary Deep Nets for Non-Intrusive Load Monitoring
Mar 6, 2023Jinsong Wang, Kenneth A. LoparoNon-Intrusive Load Monitoring Sparse Neural Networks

Jun 2, 2025J/K move · Enter open · S save

Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism

Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham, Yan Zuo+1

Pluralis Research

Communication-Efficient Distributed Training Decentralized Learning Activation Compression Distributed Training Neural Network Compression

Open paper AskPDF arXiv

Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks. While existing compression techniques are effective in data-parallel, they do not extend to model parallelism. Unlike data-parallel training, where weight gradients are exchanged, model-parallel requires compressing activations and activation gradients as they propagate through layers, accumulating compression errors. We propose a novel compression algorithm that compresses both forward and backward passes, enabling up to 99% compression with no convergence degradation with negligible memory/compute overhead. By leveraging a recursive structure in transformer networks, we predefine a low-dimensional subspace to confine the activations and gradients, allowing full reconstruction in subsequent layers. Our method achieves up to 100x improvement in communication efficiency and enables training billion-parameter-scale models over low-end GPUs connected via consumer-grade internet speeds as low as 80Mbps, matching the convergence of centralized datacenter systems with 100Gbps connections with model parallel.