Neural Network Compression

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 128

All topics

Cross-Paradigm Knowledge Distillation: A Comprehensive Study of Bidirectional Transfer Between Random Forests and Deep Neural Networks for Big Data Applications
May 19, 2026Mahdi Naser MoghadasiMulti-Teacher Knowledge Distillation Knowledge Distillation
BrightMind AI Research Seattle, Washington, USA
Quantized Machine Learning Models for Medical Imaging in Low-Resource Healthcare Settings
May 19, 2026Sumanth Meenan Kanneti, Aryan ShahEfficient Neural Network Inference Medical Image Classification
Georgia State University · University of Southern California
SparseSAM: Structured Sparsification of Activations in Segment Anything Models
May 17, 2026Hoai-Chau Tran, Chi H. Nguyen, Duy M. H. Nguyen +3Efficient Transformer Inference Image Segmentation
University of Illinois at Urbana-Champaign · VinUni-Illinois Smart Health Center, VinUniversity · College of Engineering & Computer Science, VinUniversity +3
Perforated Neural Networks for Keyword Spotting
May 15, 2026Vishy Gopal, Aris Ilias Goutis, Ralph Crewe +2Keyword Spotting Edge ML
2Purdue University · 3Renesas Electronics · 1Perforated AI
IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression
May 15, 2026Ali Abbasi, Chayne Thrash, Haoran Qin +2LLM Compression Low-Rank Matrix Decomposition
Vanderbilt University · University of California, Davis
Two-Valued Symmetric Circulant Matrices: Applications in Deep Learning
May 15, 2026Jayakrishna Amathi, Venkata Prasanth Yanambaka, Saraju P. Mohanty +1Structured Sparsity Edge ML
Department of Computer Science and Engineering University of North Texas Denton, TX, USA. · Division of Computer Science Texas Woman’s University Denton, TX, USA.
Spectral structural distortion reveals redundant neurons in neural networks
May 14, 2026Yongyu WangLLM Pruning Neural Network Compression
Neural Video Compression with Domain Transfer
May 13, 2026Tiange Zhang, Rongqun Lin, Xiandong Meng +4Test-Time Adaptation Cross-Domain Transfer Learning
Shenzhen Graduate School, Peking University, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · School of Computer Science, Peking University, Beijing, China
Robust Basis Spline Decoupling for the Compression of Transformer Models
May 11, 2026Joppe De Jonghe, Van Tien Pham, Mariya IshtevaModel Compression Tensor Decomposition
aNUMA, Department of Computer Science, KU Leuven, Kleinhoefstraat 4, Geel, 2440, Antwerpen, België · bUniversité Paris-Saclay, CEA, List, Palaiseau, F-91120, France
Evolving Knowledge Distillation for Lightweight Neural Machine Translation
May 11, 2026Xuewen Zhang, Haixiao Zhang, Xinlong HuangMulti-Teacher Knowledge Distillation Knowledge Distillation
Department of Content Generation Li Auto Beijing, China
Selection Plateau and a Sparsity-Dependent Hierarchy of Pruning Features
May 10, 2026Guangqi Li, Yongxin LiSparse Neural Networks Neural Network Compression
Zaozhuang University, Zaozhuang, Shandong, China
Compact SO(3) Equivariant Atomistic Foundation Models via Structural Pruning
May 9, 2026Chen Wang, Siyu Hu, Guangming Tan +1Equivariant GNNs Structured Pruning
State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
May 9, 2026Shengkun Tang, Zekun Wang, Bo Zheng +7Mixture-of-Experts Pruning LLM Pruning
Qwen Team, Alibaba Inc. · MBZUAI · KAUST
DiBA: Diagonal and Binary Matrix Approximation for Neural Network Weight Compression
May 7, 2026Nobutaka OnoEfficient Neural Network Inference Matrix Optimization
Tokyo Metropolitan University, 6–6 Asahigaoka, Hino-shi, Tokyo 191–0065, Japan.
Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Count
May 4, 2026Lurong PanNeural Network Optimization Feedforward Neural Networks
Elastic Spiking Transformers for Efficient Gesture Understanding
May 4, 2026Alberto Ancilotto, Gianluca Amprimo, Stefano Di Carlo +1Efficient Neural Network Inference Spiking Transformers
Fondazione Bruno Kessler · Politecnico di Torino
Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm
May 2, 2026Wen-Da Wei, Han-Bin Fang, Yang-Di Liu +3LLM Compression Gradient Compression
Nanjing University, Nanjing, China · Tsinghua University, Beijing, China · Huazhong University of Science and Technology, Wuhan, China +1
Auto-FlexSwitch: Efficient Dynamic Model Merging via Learnable Task Vector Compression
Apr 30, 2026Junqi Gao, Dazhi Zhang, Zhichang Guo +3Task Vectors Task Vector Merging
School of Mathematics, Harbin Institute of Technology, Harbin, P. R. China · Shanghai Artificial Intelligence Laboratory, Shanghai, P. R. China
Parameter-Efficient Architectural Modifications for Translation-Invariant CNNs
Apr 30, 2026Nuria Alabau-Bosque, Jorge Vila-Tomas, Paula Dauden-Oliver +2Neural Network Robustness Convolutional Neural Networks
Image Processing Lab, Universitat de València, Carrer del Catedr´atic Jos´e Beltr´an Martinez, Paterna, 46980, Spain.
Random Cloud: Finding Minimal Neural Architectures Without Training
Apr 29, 2026Javier Gil BlázquezNeural Architecture Search Feedforward Neural Networks
QYOLO: Lightweight Object Detection via Quantum Inspired Shared Channel Mixing
Apr 29, 2026Garvit Kumar Mittal, Sahil Tomar, Sandeep KumarEfficient Neural Network Inference Quantum-Inspired ML
Central Research Laboratory Bharat Electronics Limited Ghaziabad, India
Hierarchical Spatio-Channel Clustering for Efficient Model Compression in Medical Image Analysis
Apr 25, 2026Sisipho Hamlomo, Marcellin Atemkeng, Habte Tadesse Likassa +5Low-Rank Compression Low-Rank Approximation
Department of Mathematics, Rhodes University, Makhanda 6140, South Africa · Department of Statistics, Rhodes University, Makhanda 6140, South Africa · National Institute for Theoretical and Computational Sciences, Stellenbosch 7600, South Africa +8
Efficient VQ-QAT and Mixed Vector/Linear quantized Neural Networks
Apr 25, 2026Terry Gou, Puneet GuptaQuantization-Aware Training Vector Quantization
LTBs-KAN: Linear-Time B-splines Kolmogorov-Arnold Networks
Apr 23, 2026Eduardo Said Merin-Martinez, Andres Mendez-Vazquez, Eduardo Rodriguez-TelloEfficient Neural Network Inference Kolmogorov-Arnold Networks
Cinvestav, Unidad Guadalajara · Av. del Bosque 1145, El Bajio · Zapopan, Jalisco, 45017, Mexico +3
Leveraging Kernel Symmetry for Joint Compression and Error Mitigation in Edge Model Transfer
Apr 19, 2026Anis Hamadouche, Mathini SellathuraiNeural Network Compression
School of Engineering & Physical Sciences Heriot-Watt University Edinburgh EH14 4AS, UK
Towards Joint Quantization and Token Pruning of Vision-Language Models
Apr 19, 2026Xinqing Li, Xin He, Xindong Zhang +3Vision-Language Models VLM Quantization
VCIP, College of Computer Science, Nankai University · School of Computer Science and Engineering, Tianjin University of Technology · OPPO Research Institute +3
LASER: Low-Rank Activation SVD for Efficient Recursion
Apr 19, 2026Ege Çakar, Ketan Ali Raghu, Lia ZhengLow-Rank Matrix Decomposition Recurrent Neural Networks
Harvard University Cambridge, MA, USA
A Comparative Study of CNN Optimization Methods for Edge AI: Exploring the Role of Early Exits
Apr 16, 2026Nekane Fernandez, Ivan Valdes, Steven Van Vaerenbergh +2Efficient Inference Edge Inference
Nekane Fernandez, Ivan Valdes, Idoia de la Iglesia and Julen Arratibel are with the Dept. of Connected and Distributed Intelligence, Ikerlan Technology Research Centre, Arrasate, Spain · Nekane Fernandez and Steven Van Vaerenbergh are with the Dept. of Mathematics, Statistics and Computation, Universidad de Cantabria, Santander, Spain
Exploiting Correlations in Federated Learning: Opportunities and Practical Limitations
Apr 16, 2026Adrian Edin, Michel Kieffer, Mikael Johansson +1Communication-Efficient Distributed Training Gradient Compression
Department of Electrical Engineering (ISY), Linköping University, SE-581 83 Linköping, Sweden · CentraleSupélec, CNRS, L2S, Université Paris-Saclay, Gif-sur-Yvette, FR-91192, France · Division of Decision and Control Systems, School of EECS, KTH Royal Institute of Technology, SE-100 44 Stockholm, Sweden
Causal Mechanism Reduction: Mechanism Replacement for Neural Network Pruning and Abstraction
Feb 27, 2026Amir AsiaeeStructural Causal Models Causal Abstraction
Vanderbilt University Medical Center Nashville, TN

May 19, 2026, cs.LGJ/K move · Enter open · S save

Cross-Paradigm Knowledge Distillation: A Comprehensive Study of Bidirectional Transfer Between Random Forests and Deep Neural Networks for Big Data Applications

Mahdi Naser Moghadasi

BrightMind AI Research Seattle, Washington, USA

Multi-Teacher Knowledge Distillation Knowledge Distillation Random Forest Neural Network Compression

Open paper AskPDF arXiv

The exponential growth of big data has intensified the need for efficient and interpretable machine learning models that can handle diverse data characteristics while maintaining computational efficiency. Knowledge distillation has primarily focused on neural network-to-neural network transfer, leaving cross-paradigm knowledge transfer largely unexplored. This paper presents the first comprehensive study of bidirectional knowledge distillation between Random Forests (RF) and Deep Neural Networks (DNN), addressing critical gaps in ensemble learning and model compression for big data applications. We propose novel methodologies including progressive multi-stage distillation, multi-teacher ensemble distillation from diverse tree models, and uncertainty-aware cross-paradigm transfer mechanisms. Through 144 comprehensive experiments across 6 diverse datasets encompassing classification and regression tasks, we demonstrate that bidirectional RF-DL distillation achieves competitive performance while providing complementary benefits: interpretability from tree models and expressiveness from neural networks. Our results show that multi-teacher ensemble distillation consistently outperforms traditional approaches, with NN-COMPACT achieving 98.13% classification accuracy and NN-WIDE reaching 92.6% R^2 score in regression tasks. The proposed framework enables deployment flexibility in big data environments, allowing optimal model selection based on computational constraints and interpretability requirements. This work establishes a new research direction in cross-paradigm knowledge transfer with significant implications for interpretable AI and scalable model deployment in resource-constrained big data systems.

Momentum

Related topics

Latest papers 128