Neural Network Compression

Ask about this topic

Overview

Neural network compression studies how to represent, train, or run models with fewer parameters and less memory or computation, making them practical on devices with limited resources. Common approaches include pruning weights or neurons, quantization, low-rank and tensor factorization, parameter sharing, and knowledge distillation; some methods also compress activations exchanged during distributed training. Work spans small edge models and microcontrollers as well as large language, vision, and diffusion models, evaluated on tasks such as image classification, language modeling, and video coding. A central challenge is that smaller parameter counts or fewer theoretical operations do not always mean faster inference, and aggressive compression can harm accuracy, robustness, or task-specific behavior.

Latest papers

CardsList

218 papers28 papers in the last four weeks, up 8% on the four weeks before. 0.2% of all new papers.

May 2, 2026

Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm

Training large language models (LLMs) is highly memory-intensive, as training must store not only weights and optimizer states but also intermediate activations for backpropagation. While existing memory-efficient methods largely focus on gradients and optimizer states, activation compression is less well established due to the lack of LLM-tailored theory and guarantees. In this work, we develop a theoretical framework showing that activation compression is safe for linear operators when activation compression is unbiased, but problematic for nonlinear ones. We further derive gradient variance bound and establish convergence guarantees for applying activation compression to all linear operators under the standard LL-smoothness assumption, showing that it does not change the convergence rate. Guided by the theory, we propose an activation-gradient co-compression method that reuses low-rank activation factors to compress linear-layer gradients without extra computation or additional gradient error. We conduct extensive experiments on Qwen and LLaMA models using a pretraining benchmark and multiple fine-tuning benchmarks to validate our theory and demonstrate competitive performance of our method in both accuracy and compression efficiency. We provide our code in the supplementary material for reproducibility.

Wen-Da Wei, Han-Bin Fang, Yang-Di Liu +3

Nanjing University, Nanjing, China · Tsinghua University, Beijing, China · Huazhong University of Science and Technology, Wuhan, China +1

LLM CompressionGradient CompressionLow-Rank CompressionActivation CompressionLLM TrainingNeural Network Compression+1
AskPDF
Apr 30, 2026

Auto-FlexSwitch: Efficient Dynamic Model Merging via Learnable Task Vector Compression

Model merging has attracted attention as an effective path toward multi-task adaptation by integrating knowledge from multiple task-specific models. Among existing approaches, dynamic merging mitigates performance degradation caused by conflicting parameter updates across tasks by flexibly combining task-specific parameters at inference time, thereby maintaining high performance. However, these methods require storing independent parameters for each task, resulting in prohibitive storage overhead. To address this issue, we first experimentally demonstrate that the fine-tuned weight increments (referred to as task vectors) exhibit an impulse-like activation pattern and high robustness to low-bit representations. Driven by this insight, we propose T-Switch, which decomposes task vectors into three compact components: a binary sparse mask, a sign vector, and a scalar scaling factor, achieving high-fidelity approximation at high compression ratios. We then introduce Auto-Switch, a training-free merging scheme that automatically composes task vectors via feature similarity retrieval. Building on this, we develop Auto-Switch, a training-free merging scheme that automatically assembles task vectors through feature similarity retrieval. Furthermore, to transform task vector sparsification and quantization from static rules to adaptive learning, we propose FlexSwitch, a learnable framework which jointly optimizes the compression strategy for each model unit via Learnable Gating Sparsification (LGS) and Bit-width Adaptive Selection (BAS), while employing the Sparsity-Aware Storage Strategy (SASS) to select the optimal storage encoding structure. Finally, by incorporating a K-Nearest Neighbor (KNN) inference scheme with a learnable low-rank metric, we present Auto-FlexSwitch, a dynamic model merging approach that supports highly efficient task vector compression.

Junqi Gao, Dazhi Zhang, Zhichang Guo +3

School of Mathematics, Harbin Institute of Technology, Harbin, P. R. China · Shanghai Artificial Intelligence Laboratory, Shanghai, P. R. China

Task VectorsTask Vector MergingModel MergingNeural Network Compression
AskPDF
Apr 30, 2026

Parameter-Efficient Architectural Modifications for Translation-Invariant CNNs

Convolutional Neural Networks (CNNs) are widely assumed to be translation-invariant, yet standard architectures exhibit a startling fragility: even a single-pixel shift can drastically degrade performance due to their reliance on spatially dependent fully connected layers. In this work, we resolve this vulnerability by proposing a lightweight 'Online Architecture' strategy. By strategically inserting Global Average Pooling (GAP) layers at various network depths, we effectively decouple feature recognition from spatial location. Using VGG-16 as a primary case study, we demonstrate that this architectural modification achieves a massive 98% reduction in trainable parameters (from 5.2M to just 82K) and a 90% reduction in total network size (138M to 14M). Despite this drastic pruning, our variants maintain competitive Top-1 accuracy on ImageNet (66.4%) while doubling translational robustness, reducing average relative loss from 0.09 to 0.05. Furthermore, our analysis identifies a fundamental limit to invariance: while GAP resolves macroscopic sensitivity, discrete pooling operations introduce a residual periodic aliasing that prevents perfect pixel-level stability. Finally, we extend these findings to Perceptual Image Quality Assessment (IQA) by integrating our invariant backbones into the LPIPS framework. The resulting metric significantly outperforms the retrained baseline in generalization across the KADID-10k dataset (Spearman 0.89 vs. 0.75) and achieves a near-perfect alignment with human psychophysical response curves on the RAID dataset (Spearman 0.95). These results confirm that enforcing architectural invariance is a far more efficient and biologically plausible path to robustness than traditional data augmentation. Data and code are publicly available. The data and code are publicly available to facilitate validation and further research.

Nuria Alabau-Bosque, Jorge Vila-Tomas, Paula Dauden-Oliver +2

Image Processing Lab, Universitat de València, Carrer del Catedr´atic Jos´e Beltr´an Martinez, Paterna, 46980, Spain.

Neural Network RobustnessConvolutional Neural NetworksInvariant Representation LearningNeural Network Compression
AskPDF
Apr 29, 2026

Random Cloud: Finding Minimal Neural Architectures Without Training

I propose the Random Cloud method, a training-free approach to neural architecture search that discovers minimal feedforward network topologies through stochastic exploration and progressive structural reduction. Unlike post-training pruning methods that require a full train-prune-retrain cycle, this method evaluates randomly initialized networks without backpropagation, progressively reduces their topology, and only trains the best minimal candidate at the end. I evaluate on 7 classification benchmarks against magnitude pruning and random pruning baselines. The Random Cloud matches or outperforms both baselines in 6 of 7 datasets, achieving statistically significant improvements on Sonar (+4.9+4.9pp accuracy, p=0.017p{=}0.017 vs magnitude pruning) with 87% parameter reduction. Crucially, the method is faster than both pruning baselines in 4 of 5 datasets (0.67--0.94×\times the cost of full training), since it avoids training the full-size network entirely.

Javier Gil Blázquez

Neural Architecture SearchFeedforward Neural NetworksNeural Network CompressionLightweight Neural Networks
AskPDF
Apr 29, 2026

QYOLO: Lightweight Object Detection via Quantum Inspired Shared Channel Mixing

The rapid advancement of object detection architectures has positioned single stage detectors as the dominant solution for real-time visual perception. A primary source of computational overhead in these models lies in the deep backbone stages, where C2f bottleneck modules at high stride levels accumulate a disproportionate share of parameters due to quadratic scaling with channel width. This work introduces QYOLO, a quantum-inspired channel mixing framework that achieves genuine architectural compression by replacing the two deepest backbone C2f modules at P4/16 (512 channels) and P5/32 (1024 channels) with a compact QMixBlock. The proposed block performs global channel recalibration through a sinusoidal mixing mechanism with shared learnable parameters across both backbone stages, enforcing consistent channel importance without requiring independent per-stage parameter sets. The neck and detection head remain fully classical and unchanged. Evaluation on the VisDrone2019 benchmark demonstrates that QYOLOv8n achieves a 20.2% reduction in parameter count (3.01M to 2.40M) and 12.3% GFLOPs reduction with only 0.4 pp mAP@50 degradation. QYOLOv8s achieves 21.8% reduction with 0.1 pp degradation. When combined with knowledge distillation, full accuracy parity is recovered at no cost to compression. An expanded backbone plus neck variant achieved 38 to 41% reduction at the cost of greater accuracy degradation, motivating the backbone-only final design.

Garvit Kumar Mittal, Sahil Tomar, Sandeep Kumar

Central Research Laboratory Bharat Electronics Limited Ghaziabad, India

Efficient Neural Network InferenceQuantum-Inspired MLObject DetectionNeural Network Compression
AskPDF
Apr 25, 2026

Hierarchical Spatio-Channel Clustering for Efficient Model Compression in Medical Image Analysis

Convolutional neural networks (CNNs) have become increasingly difficult to deploy in resource-constrained environments due to their large memory and computational requirements. Although low-rank compression methods can reduce this burden, most existing approaches compress spatial and channel redundancy independently and therefore do not fully exploit the localised structure within convolutional feature maps. This paper proposes a hierarchical spatio-channel low-rank compression framework for CNNs that exploits redundancy across spatial regions and channel activations. Unlike conventional methods, which apply a uniform decomposition across an entire layer, the proposed approach first partitions feature maps into spatial regions, then groups channels according to their co-activation patterns within each region, and finally applies rank-adaptive SVD to each resulting spatio-channel cluster. The method is evaluated on an AlexNet-based brain tumour MRI classification model and compared with Global SVD and Tucker decomposition under 3×3\times and 6×6\times compression budgets. Our method outperforms both baselines, reducing FLOPs from 8.21 G8.21\,\mathrm{G} to 1.55 G1.55\,\mathrm{G} (81.1%81.1\% reduction), achieving a 1.38×1.38\times inference speed-up, and increasing classification accuracy from 87.76%87.76\% to 89.80%89.80\%. The method also improves the macro F1F_1-score and performance on challenging classes such as meningioma. A hyper-parameter trade-off analysis demonstrates that the framework provides Pareto-optimal configurations, enabling control over the balance between compression and predictive performance. Moderate clustering with adaptive rank selection yields strong results. Bootstrap standard errors are reported for all classification metrics.

Sisipho Hamlomo, Marcellin Atemkeng, Habte Tadesse Likassa +5

Department of Mathematics, Rhodes University, Makhanda 6140, South Africa · Department of Statistics, Rhodes University, Makhanda 6140, South Africa · National Institute for Theoretical and Computational Sciences, Stellenbosch 7600, South Africa +8

Low-Rank CompressionLow-Rank ApproximationConvolutional Neural NetworksNeural Network Compression
AskPDF
Apr 25, 2026

Efficient VQ-QAT and Mixed Vector/Linear quantized Neural Networks

In this work, we developed and tested 3 techniques for vector quantization (VQ) based model weight compression. To mitigate codebook collapse and enable end-to-end training, we adopted cosine similarity-based assignment. Building on ideas from attention-based formulations in Differentiable K-Means (DKM), we further improved this approach by using cosine similarity for assignment combined with top-1 sampling and a straight-through estimator, thereby eliminating the need for weighted-average reconstruction. Finally, we investigated the use of differentiable neural architecture search (NAS) to adaptively select layer-wise quantization configurations, further optimizing the compression process. Although our method does not consistently outperform existing approaches across all quantization levels, it provides useful insights into the design trade-offs and behaviors of VQ-based model compression methods.

Terry Gou, Puneet Gupta

Quantization-Aware TrainingVector QuantizationNeural Network QuantizationNeural Network Compression
AskPDF
Apr 23, 2026

LTBs-KAN: Linear-Time B-splines Kolmogorov-Arnold Networks

Kolmogorov-Arnold Networks (KANs) are a recent neural network architecture offering an alternative to Multilayer Perceptrons (MLPs) with improved explainability and expressibility. However, KANs are significantly slower than MLPs due to the recursive nature of B-spline function computations, limiting their application. This work addresses these issues by proposing a novel base-spline Linear-Time B-splines Kolmogorov-Arnold Network (LTBs-KAN) with linear complexity. Unlike previous methods that rely on the Boor-Mansfield-Cox spline algorithm or other computationally intensive mathematical functions, our approach significantly reduces the computational burden. Additionally, we further reduce model's parameter through product-of-sums matrix factorization in the forward pass without sacrificing performance. Experiments on MNIST, Fashion-MNIST and CIFAR-10 demonstrate that LTBs-KAN achieves good time complexity and parameter reduction, when used as building architectural blocks, compared to other KAN implementations.

Eduardo Said Merin-Martinez, Andres Mendez-Vazquez, Eduardo Rodriguez-Tello

Cinvestav, Unidad Guadalajara · Av. del Bosque 1145, El Bajio · Zapopan, Jalisco, 45017, Mexico +3

Efficient Neural Network InferenceKolmogorov-Arnold NetworksNeural Network Compression
AskPDF
Apr 19, 2026

Leveraging Kernel Symmetry for Joint Compression and Error Mitigation in Edge Model Transfer

This paper investigates communication-efficient neural network transmission by exploiting structured symmetry constraints in convolutional kernels. Instead of transmitting all model parameters, we propose a degrees-of-freedom (DoF) based codec that sends only the unique coefficients implied by a chosen symmetry group, enabling deterministic reconstruction of the full weight tensor at the receiver. The proposed framework is evaluated under quantization and noisy channel conditions across multiple symmetry patterns, signal-to-noise ratios, and bit-widths. To improve robustness against transmission impairments, a projection step is further applied at the receiver to enforce consistency with the symmetry-invariant subspace, effectively denoising corrupted parameters. Experimental results on MNIST and CIFAR-10 using a DeepCNN architecture demonstrate that DoF-based transmission achieves substantial bandwidth reduction while preserving significantly higher accuracy than pruning-based baselines, which often suffer catastrophic degradation. Among the tested symmetries, central-skew symmetry consistently provides the best accuracy-compression tradeoff, confirming that structured redundancy can be leveraged for reliable and efficient neural model delivery over constrained links.

Anis Hamadouche, Mathini Sellathurai

School of Engineering & Physical Sciences Heriot-Watt University Edinburgh EH14 4AS, UK

Neural Network Compression
AskPDF
Apr 19, 2026

Towards Joint Quantization and Token Pruning of Vision-Language Models

Deploying Vision-Language Models (VLMs) under aggressive low-bit inference remains challenging because inference cost is dominated by the long visual-token prefix during prefill and the growing KV cache during autoregressive decoding. Token pruning and low-bit quantization are complementary for reducing these costs, yet naive stage-wise combinations are often brittle due to a mismatch between quantization calibration and pruning execution. We present a collaborative quantization-and-pruning framework that unifies low-bit inference and deterministic visual-token pruning in a single deployable pipeline. The framework introduces the Quantization Unified Offline Token Allocator (QUOTA), which converts low-bit calibration signals into a layer-wise token allocation schedule and materializes it as a pruning recipe. Token importance is evaluated under deployed W4A4 operators with a quantized KV cache by combining activation magnitude, attention cues, and an explicit low-bit risk signal, enabling consistent budgeted top-kk selection. Experiments on standard VLM benchmarks show improved robustness over stage-wise baselines under the same low-bit regime, achieving 95.65% average retention while retaining only 30% of visual tokens, compared with about 94.3% retention for representative stage-wise combinations. The code will be released.

Xinqing Li, Xin He, Xindong Zhang +3

VCIP, College of Computer Science, Nankai University · School of Computer Science and Engineering, Tianjin University of Technology · OPPO Research Institute +3

Vision-Language ModelsVLM QuantizationEfficient VLM InferenceLow-Bit QuantizationVisual Token PruningNeural Network Compression+1
AskPDF
Apr 19, 2026

LASER: Low-Rank Activation SVD for Efficient Recursion

Recursive architectures such as Tiny Recursive Models (TRMs) perform implicit reasoning through iterative latent computation, yet the geometric structure of these reasoning trajectories remains poorly understood. We investigate the activation manifold of TRMs during recursive unrolling and find that activations occupy an effectively linear, low-dimensional subspace whose principal directions can be tracked dynamically with cheap power iterations. This suggests that weight-sharing concentrates iterative computation along a small number of dominant eigendirections, and we find that this concentration varies sharply across computational sites. We exploit this structure through LASER (Low-Rank Activation SVD for Efficient Recursion), a dynamic compression framework that maintains an evolving low-rank basis via matrix-free subspace tracking with a fidelity-triggered reset mechanism, achieving ∼60%{\sim}60\% activation memory savings with no statistically significant accuracy degradation. Our analysis raises questions about how recursive architectures allocate representational capacity during implicit reasoning, and whether this concentration can be exploited to improve the efficiency and stability of latent computation.

Ege Çakar, Ketan Ali Raghu, Lia Zheng

Harvard University Cambridge, MA, USA

Low-Rank Matrix DecompositionRecurrent Neural NetworksLow-Rank CompressionLow-Rank ApproximationNeural Network Compression
AskPDF
Apr 16, 2026

A Comparative Study of CNN Optimization Methods for Edge AI: Exploring the Role of Early Exits

Deploying deep neural networks on edge devices requires balancing accuracy, latency, and resource constraints under realistic execution conditions. To fit models within these constraints, two broad strategies have emerged: static compression techniques such as pruning and quantization, which permanently reduce model size, and dynamic approaches such as early-exit mechanisms, which adapt computational cost at runtime. While both families are widely studied in isolation, they are rarely compared under identical conditions on physical hardware. This paper presents a unified deployment-oriented comparison of static compression and dynamic early-exit mechanisms, evaluated on real edge devices using ONNX based inference pipelines. Our results show that static and dynamic techniques offer fundamentally different trade-offs for edge deployment. While pruning and quantization deliver consistent memory footprint reduction, early-exit mechanisms enable input-adaptive computation savings that static methods cannot match. Their combination proves highly effective, simultaneously reducing inference latency and memory usage with minimal accuracy loss, expanding what is achievable at the edge.

Nekane Fernandez, Ivan Valdes, Steven Van Vaerenbergh +2

Nekane Fernandez, Ivan Valdes, Idoia de la Iglesia and Julen Arratibel are with the Dept. of Connected and Distributed Intelligence, Ikerlan Technology Research Centre, Arrasate, Spain · Nekane Fernandez and Steven Van Vaerenbergh are with the Dept. of Mathematics, Statistics and Computation, Universidad de Cantabria, Santander, Spain

Efficient InferenceEdge InferenceNeural Network QuantizationNeural Network CompressionEarly-Exit Neural NetworksNeural Network Pruning
AskPDF
Apr 16, 2026

Exploiting Correlations in Federated Learning: Opportunities and Practical Limitations

The communication bottleneck in federated learning (FL) has spurred extensive research into techniques to reduce the volume of data exchanged between client devices and the central parameter server. In this paper, we systematically classify gradient and model compression schemes into three categories based on the type of correlations they exploit: structural, temporal, and spatial. We examine the sources of such correlations, propose quantitative metrics for measuring their magnitude, and reinterpret existing compression methods through this unified correlation-based framework. Our experimental studies demonstrate that the degrees of structural, temporal, and spatial correlations vary significantly depending on task complexity, model architecture, and algorithmic configurations. These findings suggest that algorithm designers should carefully evaluate correlation assumptions under specific deployment scenarios rather than assuming that they are always present. Motivated by these findings, we propose two adaptive compression designs that actively switch between different compression modes based on the measured correlation strength, and we evaluate their performance gains relative to conventional non-adaptive approaches. In summary, our unified taxonomy provides a clean and principled foundation for developing more effective and application-specific compression techniques for FL systems.

Adrian Edin, Michel Kieffer, Mikael Johansson +1

Department of Electrical Engineering (ISY), Linköping University, SE-581 83 Linköping, Sweden · CentraleSupélec, CNRS, L2S, Université Paris-Saclay, Gif-sur-Yvette, FR-91192, France · Division of Decision and Control Systems, School of EECS, KTH Royal Institute of Technology, SE-100 44 Stockholm, Sweden

Communication-Efficient Distributed TrainingGradient CompressionNeural Network CompressionFederated Learning
AskPDF
Apr 14, 2026

OFA-Diffusion Compression: Compressing Diffusion Model in One-Shot Manner

The Diffusion Probabilistic Model (DPM) achieves remarkable performance in image generation, while its increasing parameter size and computational overhead hinder its deployment in practical applications. To improve this, the existing literature focuses on obtaining a smaller model with a fixed architecture through model compression. However, in practice, DPMs usually need to be deployed on various devices with different resource constraints, which leads to multiple compression processes, incurring significant overhead for repeated training. To obviate this, we propose a once-for-all (OFA) compression framework for DPMs that yields different subnetworks with various computations in a one-shot training manner. The existing OFA framework typically involves massive subnetworks with different parameter sizes, while such a huge candidate space slows the optimization. Thus, we propose to restrict the candidate subnetworks with a certain set of parameter sizes, where each size corresponds to a specific subnetwork. Specifically, to construct each subnetwork with a given size, we gradually allocate the maintained channels by their importance. Furthermore, we propose a reweighting strategy to balance the optimization process of different subnetworks. Experimental results show that our approach can produce compressed DPMs for various sizes with significantly lower training overhead while achieving satisfactory performance.

Haoyang Jiang, Zekun Wang, Mingyang Yi +6

Renmin University of China Beijing, China · Alibaba Inc. Hangzhou, China · Independent researcher +1

Diffusion Model CompressionModel CompressionNeural Network CompressionDiffusion Models
AskPDF
Apr 13, 2026

ResBM: Residual Bottleneck Models for Low-Bandwidth Pipeline Parallelism

Unlocking large-scale low-bandwidth decentralized training has the potential to utilize otherwise untapped compute resources. In centralized settings, large-scale multi-node training is primarily enabled by data and pipeline parallelism, two techniques that require ultra-high-bandwidth communication. While efficient methods now exist for decentralized data parallelism, pipeline parallelism remains the primary challenge. Recent efforts, such as Subspace Models (SM), have claimed up to 100x activation compression but rely on complex constrained optimization and diverge from true end-to-end training. In this paper, we propose a different approach, based on an architecture designed from the ground up to be native to low-bandwidth communication environments while still applicable to any standard transformer-based architecture. We call this architecture the Residual Bottleneck Model or ResBM, it introduces a residual encoder-decoder bottleneck module across pipeline boundaries that can be trained end-to-end as part of the model's parameters while preserving an explicit low-rank identity path. We show that ResBMs achieve state-of-the-art 128x activation compression without significant loss in convergence rates and without significant memory or compute overhead.

Alan Aboudib, Rodrigo Lopez Portillo A., Kalei Brady +1

Macrocosmos AI

Activation CompressionCommunication-Efficient Distributed TrainingPipeline ParallelismNeural Network CompressionLow-Rank Compression
AskPDF
Apr 13, 2026

Saccade Attention Networks: Using Transfer Learning of Attention to Reduce Network Sizes

One of the limitations of transformer networks is the sequence length due to the quadratic nature of the attention matrix. Classical self attention uses the entire sequence length, however, the actual attention being used is sparse. Humans use a form of sparse attention when analyzing an image or scene called saccades. Focusing on key features greatly reduces computation time. By using a network (Saccade Attention Network) to learn where to attend from a large pre-trained model, we can use it to pre-process images and greatly reduce network size by reducing the input sequence length to just the key features being attended to. Our results indicate that you can reduce calculations by close to 80% and produce similar results.

Marc Estafanous

Johns Hopkins University · Neurobaby Corporation

Efficient ViTsNeural Network CompressionSparse AttentionVisual AttentionVision Transformer
AskPDF
Apr 11, 2026

Engineering Resource-constrained Software Systems with DNN Components: a Concept-based Pruning Approach

Deep Neural Networks (DNNs) are widely used by engineers to solve difficult problems that require predictive modeling from data. However, these models are often massive, with millions or billions of parameters, and require substantial computational power, RAM, and storage. This becomes a limitation in practical scenarios where strict size and resource constraints must be respected. In this paper, we present a novel concept-based pruning technique for DNNs that guides pruning decisions using human-interpretable concepts, such as features, colors, and classes. This is particularly important in a software engineering context, as DNNs are integrated into systems and must be pruned according to specific system requirements. Our concept-based pruning solution analyzes neuron activations to identify important neurons from a system requirements viewpoint and uses this information to guide the DNN pruning. We assess our solution using the VGG-19 network and a dataset of 26'384 RGB images, focusing on its ability to produce small, effective pruned DNNs and on the computational complexity and performance of these pruned DNNs. We also analyzed the pruning efficiency of our solution and compared alternative configurations. Our results show that concept-based pruning efficiently generates much smaller, effective pruned DNNs. Pruning greatly improves the computational efficiency and performance of DNNs, properties that are particularly useful for practical applications with stringent memory and computational time constraints. Finally, alternative configuration options enable engineers to identify trade-offs adapted to different practical situations.

Federico Formica, Andrea Rota, Aurora Francesca Zanenga +4

University of Bergamo, Bergamo, Italy

Neural Network PruningNeural Network CompressionEfficient Neural Network InferenceSoftware Engineering
AskPDF
Apr 10, 2026

TinyNeRV: Compact Neural Video Representations via Capacity Scaling, Distillation, and Low-Precision Inference

Implicit neural video representations encode entire video sequences within the parameters of a neural network and enable constant time frame reconstruction. Recent work on Neural Representations for Videos (NeRV) has demonstrated competitive reconstruction performance while avoiding the sequential decoding process of conventional video codecs. However, most existing studies focus on moderate or high capacity models, leaving the behavior of extremely compact configurations required for constrained environments insufficiently explored. This paper presents a systematic study of tiny NeRV architectures designed for efficient deployment. Two lightweight configurations, NeRV-T and NeRV-T+, are introduced and evaluated across multiple video datasets in order to analyze how aggressive capacity reduction affects reconstruction quality, computational complexity, and decoding throughput. Beyond architectural scaling, the work investigates strategies for improving the performance of compact models without increasing inference cost. Knowledge distillation with frequency-aware focal supervision is explored to enhance reconstruction fidelity in low-capacity networks. In addition, the impact of lowprecision inference is examined through both post training quantization and quantization aware training to study the robustness of tiny models under reduced numerical precision. Experimental results demonstrate that carefully designed tiny NeRV variants can achieve favorable quality efficiency trade offs while substantially reducing parameter count, computational cost, and memory requirements. These findings provide insight into the practical limits of compact neural video representations and offer guidance for deploying NeRV style models in resource constrained and real-time environments. The official implementation is available at https: //github.com/HannanAkhtar/TinyNeRV-Implementation.

Muhammad Hannan Akhtar, Ihab Amer, Tamer Shanableh

Department of Computer Science & Engineering, American University of Sharjah, Sharjah, United Arab Emirates

Neural Video CompressionNeural Network CompressionEfficient Neural Network Inference
AskPDF
Apr 9, 2026

SOLAR: Communication-Efficient Model Adaptation via Subspace-Oriented Latent Adapter Reparametrization

Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, enable scalable adaptation of foundation models by injecting low-rank adapters. However, their communication and storage costs remain a major bottleneck in resource-constrained settings. We propose SOLAR (Subspace-Oriented Latent Adapter Reparameterization), a post-training compression framework that substantially reduces the communication cost (i.e., the number of parameters to transmit or store) of PEFT adapters. SOLAR expresses each PEFT update as a linear combination of basis vectors formed from the foundation model's singular vectors with controlled random perturbations. By exploiting the subspace similarity (the alignment of principal directions) between the foundation model and task-specific fine-tuned updates, SOLAR decouples the adapter size from PEFT structure and ensures compact yet expressive representations. It is model-agnostic and compatible with existing PEFT methods, including LoRA, AdaLoRA, and other adapter modules. We theoretically establish a bound on the reconstruction error. Experiments on language and vision tasks using LLaMA, GPT, and ViT models demonstrate that SOLAR preserves task performance while significantly reducing model representation sizes, offering an effective and communication-efficient solution for deployment in distributed systems and edge devices.

Seyed Mahmoud Sajjadi Mohammadabadi, Xiaolong Ma, Lei Yang +2

University of Nevada, Reno, Reno, NV, USA · Argonne National Laboratory, Lemont, IL, USA · University of Houston, Houston, TX, USA +1

Model CompressionNeural Network CompressionParameter-Efficient Fine-TuningFine-TuningLow-Rank Adaptation
AskPDF
Apr 8, 2026

Learning is Forgetting: LLM Training As Lossy Compression

Despite the increasing prevalence of large language models (LLMs), we still have a limited understanding of how their representational spaces are structured. This limits our ability to interpret how and what they learn or relate them to learning in humans. We argue LLMs are best seen as an instance of lossy compression, where over training they learn by retaining only information in their training data relevant to their objective(s). We show pre-training results in models that are optimally compressed for next-sequence prediction, approaching the Information Bottleneck bound on compression. Across an array of open weights models, each compresses differently, likely due to differences in the data and training recipes used. However even across different families of LLMs the optimality of a model's compression, and the information present in it, can predict downstream performance on across a wide array of benchmarks, letting us directly link representational structure to actionable insights about model performance. In the general case the work presented here offers a unified Information-Theoretic framing for how these models learn that is deployable at scale.

Henry C. Conklin, Tom Hosking, Tan Yi-Chern +5

Princeton University · Cohere

Language Model PretrainingLLM CompressionInformation BottleneckRepresentation Geometry in Language ModelsNeural Network Compression
AskPDF
Apr 8, 2026

SL-FAC: A Communication-Efficient Split Learning Framework with Frequency-Aware Compression

The growing complexity of neural networks hinders the deployment of distributed machine learning on resource-constrained devices. Split learning (SL) offers a promising solution by partitioning the large model and offloading the primary training workload from edge devices to an edge server. However, the increasing number of participating devices and model complexity leads to significant communication overhead from the transmission of smashed data (e.g., activations and gradients), which constitutes a critical bottleneck for SL. To tackle this challenge, we propose SL-FAC, a communication-efficient SL framework comprising two key components: adaptive frequency decomposition (AFD) and frequency-based quantization compression (FQC). AFD first transforms the smashed data into the frequency domain and decomposes it into spectral components with distinct information. FQC then applies customized quantization bit widths to each component based on its spectral energy distribution. This collaborative approach enables SL-FAC to achieve significant communication reduction while strategically preserving the information most crucial for model convergence. Extensive experiments confirm the superior performance of SL-FAC for improving the training efficiency.

Zehang Lin, Miao Yang, Haihan Zhu +9

School of Computer and Information Engineering, Xiamen University of Technology, Xiamen, 361000, China · School of Artificial Intelligence, Hebei University of Technology · Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong +5

Activation CompressionCommunication-Efficient Distributed TrainingDistributed TrainingActivation QuantizationNeural Network Compression
AskPDF
Apr 8, 2026

The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression

We decompose the spectral edge -- the dominant direction of the Gram matrix of parameter updates -- into its gradient and weight-decay components during grokking in two sequence tasks (Dyck-1 and SCAN). We find a sharp two-phase lifecycle: before grokking the edge is gradient-driven and functionally active; at grokking, gradient and weight decay align, and the edge becomes a compression axis that is perturbation-flat yet ablation-critical (>4000x more impactful than random directions). Three universality classes emerge (functional, mixed, compression), predicted by the gap flow equation. Nonlinear probes show information is re-encoded, not lost (MLP R2=0.99R^2=0.99 where linear R2=0.86R^2=0.86), and removing weight decay post-grok reverses compression while preserving the algorithm.

Yongzhong Xu

Neural Network Training DynamicsGrokkingWeight DecayNeural Network CompressionSpectral Methods
AskPDF
Apr 6, 2026

Training Transformers in Cosine Coefficient Space

Linear layers hold most of a transformer's parameters. We replace each linear layer with one that stores KK out of mnmn two-dimensional DCT coefficients per weight matrix and reconstructs the full matrix through an inverse DCT at every forward pass; the KK coefficients are the trainable parameters. A 4-layer, 128-dim transformer trained from scratch on character-level Shakespeare reaches validation loss 1.6041.604 at K=mn/2K = mn/2, against 1.5801.580 for a standard dense baseline -- a gap of +0.024+0.024 at half the trainable parameter count, within the terminal-epoch variation of the dense run. A rank-48 LoRA factorization at the same trainable parameter count reaches only 1.8011.801 (+0.221+0.221). The structural advantage of sparse-coefficient over low-rank parameterizations at matched KK is qualitative. We identify rank flexibility as the mechanism. A random orthonormal basis matches the DCT within noise at K=mn/2K = mn/2, and a compression sweep through K=mn/10K = mn/10 and K=mn/20K = mn/20 shows that subspaces that can host high-rank matrices keep the loss low, while subspaces that flatten into a low-rank block (zigzag-selection variants) converge onto the observed stable rank and the loss line of the rank-48 LoRA reference in lock-step. Among these orthonormal bases, the DCT is preferred because its separable fast transform admits a fused reconstruction kernel: the materialized weight matrix never leaves on-chip memory, so the parameter saving translates into a bandwidth saving as well.

Mohamed Amine Bergach

Illumina, San Diego, CA, USA

TransformerNeural Network Compression
AskPDF
Apr 5, 2026

Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression

Modern deployment often requires trading accuracy for efficiency under tight CPU and memory constraints, yet common compression proxies such as parameter count or FLOPs do not reliably predict wall-clock inference time. In particular, unstructured sparsity can reduce model storage while failing to accelerate (and sometimes slightly slowing down) standard CPU execution due to irregular memory access and sparse kernel overhead. Motivated by this gap between compression and acceleration, we study a practical, ordered pipeline that targets measured latency by combining three widely used techniques: unstructured pruning, INT8 quantization-aware training (QAT), and knowledge distillation (KD). Empirically, INT8 QAT provides the dominant runtime benefit, while pruning mainly acts as a capacity-reduction pre-conditioner that improves the robustness of subsequent low-precision optimization; KD, applied last, recovers accuracy within the already constrained sparse INT8 regime without changing the deployment form. We evaluate on CIFAR-10/100 using three backbones (ResNet-18, WRN-28-10, and VGG-16-BN). Across all settings, the ordered pipeline achieves a stronger accuracy-size-latency frontier than any single technique alone, reaching 0.99-1.42 ms CPU latency with competitive accuracy and compact checkpoints. Controlled ordering ablations with a fixed 20/40/40 epoch allocation further confirm that stage order is consequential, with the proposed ordering generally performing best among the tested permutations. Overall, our results provide a simple guideline for edge deployment: evaluate compression choices in the joint accuracy-size-latency space using measured runtime, rather than proxy metrics alone.

Longsheng Zhou, Yu Shen

Institute of Advanced Technology University of Science and Technology of China Hefei, China · Supercomputing Center University of Science and Technology of China Hefei, China

Neural Network CompressionEfficient Neural Network InferenceNeural Network Pruning
AskPDF
Apr 4, 2026

AutoCompress: Critical Layer Isolation for Efficient Transformer Compression

We present AutoCompress, a transformer compression method motivated by an empirical finding: in small transformers, Layer 0 carries disproportionately high task-critical information, with an NTK-based importance score of 3.6 compared to a maximum of 0.054 for all other layers -- a gap of over 60x. Based on this finding, we propose Critical Layer Isolation (CLI), an architecture that protects Layer 0 at full dimensionality, compresses all intermediate layers through a learned bottleneck, and restores the full dimension at the final layer. Applied to GPT-2 Medium (354.8M parameters), CLI-GPT2 achieves 204.5 perplexity on WikiText-103 with only 143.8M parameters -- a 2.47x compression ratio and 59.5% parameter reduction. Crucially, an ablation study demonstrates that a uniform bottleneck baseline of comparable size achieves only 571.8 perplexity under identical training conditions, confirming that the architectural decision to protect Layer 0 -- rather than simply reducing model size -- is the primary driver of performance. Code and checkpoints are publicly available.

Archit Thorat

Independent Researcher

Neural Network CompressionLLM Compression
AskPDF
Apr 2, 2026

Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression

The deployment of Large Language Models is constrained by the memory and bandwidth demands of static weights and dynamic Key-Value cache. SVD-based compression provides a hardware-friendly solution to reduce these costs. However, existing methods suffer from two key limitations: some are suboptimal in reconstruction error, while others are theoretically optimal but practically inefficient. In this paper, we propose Swift-SVD, an activation-aware, closed-form compression framework that simultaneously guarantees theoretical optimum, practical efficiency and numerical stability. Swift-SVD incrementally aggregates covariance of output activations given a batch of inputs and performs a single eigenvalue decomposition after aggregation, enabling training-free, fast, and optimal layer-wise low-rank approximation. We employ effective rank to analyze local layer-wise compressibility and design a dynamic rank allocation strategy that jointly accounts for local reconstruction loss and end-to-end layer importance. Extensive experiments across six LLMs and eight datasets demonstrate that Swift-SVD outperforms state-of-the-art baselines, achieving optimal compression accuracy while delivering 3-70X speedups in end-to-end compression time. Our code is available at https://github.com/hiahei/Swift-SVD.

Ruoling Qi, Yirui Liu, Xuaner Wu +6

Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Jiao Tong University · University of Maryland +2

LLM CompressionLow-Rank CompressionNeural Network CompressionLow-Rank ApproximationSingular Value Decomposition
AskPDF
Apr 1, 2026

A Hierarchical Importance-Guided Multi-objective Evolutionary Framework for Deep Neural Network Pruning

The optimization of over-parameterized deep neural networks represents a large-scale, high-dimensional, and strongly non-convex decision problem that challenges existing optimization frameworks. Current evolutionary and gradient-based pruning methods often struggle to scale to such dimensionalities, as they rely on flat search spaces, scalarized objectives, or repeated retraining, leading to premature convergence and prohibitive computational cost. This paper introduces a hierarchical importance-guided evolutionary framework that reformulates convolutional network pruning as a tractable large-scale multi-objective optimization problem. In the first phase, a continuous evolutionary search performs coarse exploration of weight-wise pruning thresholds to shrink the search space and identify promising regions of the Pareto set. The second phase applies a fine-grained binary evolutionary optimization constrained to the surviving weights, where importance-aware sampling and adaptive variation operators refine local search in the sparse region of the Pareto set. This hierarchical design combines global exploration and localized exploitation to achieve a well-distributed Pareto set of networks balancing compactness and accuracy. Empirical results on CIFAR-10 and CIFAR-100 using ResNet-56 and ResNet-110 confirm the method's effectiveness compared to existing evolutionary approaches: pruning achieves up to 51.9% and 38.9% parameter reductions with almost no accuracy loss compared to state-of-the-art evolutionary DNN pruning methods. The proposed method contributes a scalable evolutionary approach for solving very-large-scale multi-objective optimization problems, offering a general paradigm extendable to other domains where the decision space is exponentially large, objective functions are conflicting, and efficient trade-off discovery is essential.

Zak Khan, Azam Asilian Bidgoli

Department of Computer Science and Physics, Wilfrid Laurier University, Waterloo, N2L 3C5, Ontario, Canada

Neural Network PruningNeural Network CompressionMulti-Objective Evolutionary Optimization
AskPDF
Mar 31, 2026

Big2Small: A Unifying Neural Network Framework for Model Compression

With the development of foundational models, model compression has become a critical requirement. Various model compression approaches have been proposed such as low-rank decomposition, pruning, quantization, ergodic dynamic systems, and knowledge distillation, which are based on different heuristics. To elevate the field from fragmentation to a principled discipline, we construct a unifying mathematical framework for model compression grounded in measure theory. We further demonstrate that each model compression technique is mathematically equivalent to a neural network subject to a regularization. Building upon this mathematical and structural equivalence, we propose an experimentally-verified data-free model compression framework, termed Big2Small, which translates Implicit Neural Representations (INRs) from data domain to the domain of network parameters. Big2Small trains compact INRs to encode the weights of larger models and reconstruct the weights during inference. To enhance reconstruction fidelity, we introduce Outlier-Aware Preprocessing to handle extreme weight values and a Frequency-Aware Loss function to preserve high-frequency details. Experiments on image classification and segmentation demonstrate that Big2Small achieves competitive accuracy and compression ratios compared to state-of-the-art baselines.

Jing-Xiao Liao, Haoran Wang, Tao Li +4

Frontier of Artificial Networks (FAN) Lab, Department of Data Science, City University of Hong Kong, Kowloon, Hong Kong SAR, China · Hong Kong Applied Science and Technology Research Institute (ASTRI) · Sichuan Engineering Research Center for Big Data Visual Analytics, Sichuan University of Science and Engineering, Zigong, China +3

Model CompressionNeural Network Compression
AskPDF
Mar 30, 2026

OneComp: One-Line Revolution for Generative AI Model Compression

Deploying foundation models is increasingly constrained by memory footprint, latency, and hardware costs. Post-training compression can mitigate these bottlenecks by reducing the precision of model parameters without significantly degrading performance; however, its practical implementation remains challenging as practitioners navigate a fragmented landscape of quantization algorithms, precision budgets, data-driven calibration strategies, and hardware-dependent execution regimes. We present OneComp, an open-source compression framework that transforms this expert workflow into a reproducible, resource-adaptive pipeline. Given a model identifier and available hardware, OneComp automatically inspects the model, plans mixed-precision assignments, and executes progressive quantization stages, ranging from layer-wise compression to block-wise refinement and global refinement. A key architectural choice is treating the first quantized checkpoint as a deployable pivot, ensuring that each subsequent stage improves the same model and that quality increases as more compute is invested. By converting state-of-the-art compression research into an extensible, open-source, hardware-aware pipeline, OneComp bridges the gap between algorithmic innovation and production-grade model deployment.

Yuma Ichikawa, Keiji Kimura, Akihiro Yoshida +11

Fujitsu Limited · RIKEN Center for AIP · Institute of Science Tokyo +2

Hardware-Aware QuantizationModel CompressionNeural Network CompressionPost-Training QuantizationNeural Network QuantizationMixed-Precision Quantization
AskPDF
Mar 28, 2026

Decompose, Mix, Adapt: A Unified Framework for Parameter-Efficient Neural Network Recombination and Compression

Parameter Recombination (PR) methods aim to efficiently compose the weights of a neural network for applications like Parameter-Efficient FineTuning (PEFT) and Model Compression (MC), among others. Most methods typically focus on one application of PR, which can make composing them challenging. For example, when deploying a large model you may wish to compress the model and also quickly adapt to new settings. However, PEFT methods often can still contain millions of parameters. This may be small compared to the original model size, but can be problematic in resource constrained deployments like edge devices, where they take a larger portion of the compressed model's parameters. To address this, we present Coefficient-gated weight Recombination by Interpolated Shared basis Projections (CRISP), a general approach that seamlessly integrates multiple PR tasks within the same framework. CRISP accomplishes this by factorizing pretrained weights into basis matrices and their component mixing projections. Sharing basis matrices across layers and adjusting its size enables us to perform MC, whereas the mixer weight's small size (fewer than 200 in some experiments) enables CRISP to support PEFT. Experiments show CRISP outperforms methods from prior work capable of dual-task applications by 4-5% while also outperforming the state-of-the-art in PEFT by 1.5% and PEFT+MC combinations by 1%. Our code is available on the repository: https://github.com/appledora/CRISP-CVPR26.

Nazia Tasnim, Shrimai Prabhumoye, Bryan A. Plummer

Boston University · NVIDIA, Boston University

Neural Network CompressionParameter-Efficient Fine-TuningModel Merging
AskPDF