cs.LGFeb 20, 2026

Cut Less, Fold More: Model Compression through the Lens of Projection Geometry

Authors: Olga Saukh, Dong Wang, Haris Šikić, Yun Cheng, Lothar Thiele

Organizations: Graz University of Technology · Complexity Science Hub · Swiss Data Science Center · ETH Zurich

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Compressing neural networks without retraining is vital for deployment at scale. We study calibration-free compression through the lens of projection geometry: structured pruning is an axis-aligned projection, whereas model folding performs a low-rank projection via weight clustering. We formalize both as orthogonal operators and show that, within a rank distance of one, folding provably yields smaller parameter reconstruction error, and under mild smoothness assumptions, smaller functional perturbations than pruning. At scale, we evaluate >1000 checkpoints spanning ResNet18, PreActResNet18, ViT-B/32, and CLIP ViT-B/32 on CIFAR-10 and ImageNet-1K, covering diverse training hyperparameters (optimizers, learning rates, augmentations, regularization, sharpness-aware training), as well as multiple LLaMA-family 60M and 130M parameter models trained on C4. We show that folding typically achieves higher post-compression accuracy, with the largest gains at moderate-high compression. The gap narrows and occasionally reverses at specific training setups. Our results position folding as a geometry-aware, calibration-free alternative to pruning that is often superior in practice and principled in theory.

Figures & tables

Figure 1: Folding outperforms magnitude pruning across diverse training regimes. Top row: ResNet18 and PreActResNet18 on CIFAR-10. ResNet18 checkpoints were trained from scratch with Adam using different hyperparameter configurations. PreActResNet18 checkpoints are from Andriushchenko et al. (2023) . Bottom row: ViT-B/32 on CIFAR-10 from ( Andriushchenko et al., 2023 ) and CLIP ViT-B/32 on ImageNet-1K from ( Wortsman et al., 2022b ) . See Appendix D for details. In these plots, we use checkpoints that were trained without L1 regularization. Scatter plots show post-compression accuracy for magnitude pruning (L1 criterion) versus folding at uniform per-layer compression ratios (color-coded by layer-wise compression ratio). Bar plots depict the accuracy gain by folding, computed as Δ=Acc(FOLD )−Acc(MAG1 )\Delta=\mathrm{Acc}{(\text{{FOLD} })}-\mathrm{Acc}{(\text{{MAG1} })} , as a function of layer-wise compression ratio. Folding yields the largest improvements at moderate to high compression, confirming its robustness across architectures and datasets. Fig. 8 shows the results for magnitude pruning with L2 criterion.
Figure 2: MAG1 versus FOLD on ViTs after LayerNorm-only fine-tuning for ViT-B/32 on CIFAR-10 and CLIP ViT-B/32 on ImageNet-1K. In the scatter plots, points are checkpoints, color encodes layer-wise compression. Bar plots depict the accuracy gain Δ=Acc(FOLD )−Acc(MAG1 )\Delta=\mathrm{Acc}{(\text{{FOLD} })}-\mathrm{Acc}{(\text{{MAG1} })} , which remains positive and typically grows with compression, indicating that even under lightweight LayerNorm adaptation FOLD retains a consistent advantage over pruning.
Figure 3: Folded models retain their accuracy advantage after fine-tuning. Results for ResNet18 trained by Adam on CIFAR-10 ( top row ) and CLIP-ViT-B/32 trained on ImageNet-1K ( bottom row ): (a,d) compares post-compression accuracy of magnitude pruning ( MAG1 ) versus folding ( FOLD ) after 1 and 5 epochs of fine-tuning. (b,e) show the accuracy gap between folding and pruning as a function of fine-tuning epochs, demonstrating that folding maintains a consistent lead, i.e. , the FOLD accuracy delta is positive. (c,f) illustrate accuracy trajectories before and after 5 epochs of fine-tuning for both methods, highlighting that folded models recover accuracy faster. Further results in Appendix E .
weight_decaywarmup_stepsmax_lrPPL ↓\downarrow 0% sparsityPPL ↓\downarrow MAG2 (20%)PPL ↓\downarrow FOLD (20%)PPL ↓\downarrow MAG2 (50%)PPL ↓\downarrow FOLD (50%)
0.018800.00132.1154.5147.17398.62221.32
0.0111000.00132.1450.1146.75220.54172.57
0.0122000.00132.2046.5747.54174.58216.36
08800.00132.1751.1448.23220.33223.86
011000.00132.2150.0347.47231.41204.47
022000.00132.4046.3846.92177.48185.27
Table 1: Evaluation of FOLD and MAG2 on LLaMA-60M. We train and evaluate 18 LLaMA-family models with 60M parameters on C4 by varying max_lr, warmup steps and weight decay. Columns 4–8 show perplexity of the trained model before compression and after pruning / folding using layer-wise pruning ratio of 20%20\% and 50%50\% . We prune only FFN blocks. Except for low learning rates with long warmup schedules, FOLD outperforms MAG2 (highlighted in bold).
Figure 4: Optimizer effect evaluated on ResNet18 checkpoints trained on CIFAR-10 with SGD (no L1 regularization). The figure complements Fig. 1 (a).
Figure 5: Learning rate modulates folding’s edge. Post-compression accuracy of FOLD and MAG1 across learning rates: ResNet18 with Adam (a) and SGD (b) , PreActResNet18 (c) , and ViT-B/32 (d) . FOLD leads at moderate–low rates. With Adam, the gap shrinks or reverses at very high rates, and closes again at extremely small rates. SGD shows weaker or opposite dependence.
Figure 6: SAM ( Foret et al., 2021 ) can boost model compression. Post-compression accuracy under training with / without SAM. (a) ResNet18 (Adam), no L1. (b) ResNet18 (Adam), L1 =10−5=10^{-5} . (c) PreActResNet18 (SGD), no L1. (d) ViT - B/32, no L1. SAM improves both FOLD and MAG1 , but the uplift is consistently larger for FOLD , especially with Adam. Light L1 regularization helps MAG1 at low compression, yet FOLD retains a clear advantage at moderate–high compression.
Figure 7: Augmentations have a generally positive effect on the post-compression accuracy. Post-compression accuracy without / with random augmentations for (a) ResNet18 (Adam), (b) ResNet18 (SGD), (c) PreActResNet18, and (d) ViT-B/32. Augmentations boost both FOLD and MAG1 . On ResNet18 they also narrow FOLD ’s advantage—most noticeably at moderate compression—consistent with added invariances making axis-aligned removals less damaging. In ViT-B/32, augmentations are essential for folding 2 2 2 Note that the base accuracy of ViT-B/32 checkpoints trained without RAUG is lower than with RAUG. .
Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

ParameterValues
Optimizeradam , sgd
Learning Rateadam : 0.1, 0.01, 0.001
 sgd : 0.1, 0.05, 0.01, 0.001
Momentumadam : 0.0
 sgd : 0.9, 0.99
Weight Decay0.0, 0.0005, 0.001
Appendix
Table 2: Hyperparameter combinations used for ResNet18 training on CIFAR-10.
Figure 8: Folding outperforms magnitude pruning across diverse training regimes. The same setup as in Fig. 1 , but compared to the L2 magnitude pruning criterion. Top row: ResNet18 and PreActResNet18 on CIFAR-10. ResNet18 checkpoints were trained from scratch with Adam using different hyperparameter configurations. Bottom row: ViT-B/32 on CIFAR-10 and CLIP ViT-B/32 on ImageNet-1K. Scatter plots show post-compression accuracy for folding versus magnitude pruning (L2 criterion) at uniform per-layer compression ratios. Bar plots depict the accuracy gain by folding, computed as Δ=Acc(FOLD )−Acc(MAG2 )\Delta=\mathrm{Acc}{(\text{{FOLD} })}-\mathrm{Acc}{(\text{{MAG2} })} , as a function of layer-wise compression ratio. Folding yields the largest improvements at moderate to high compression, confirming its robustness across architectures and datasets.
Figure 9: FOLD versus MAG2 on ViTs after LayerNorm-only fine-tuning for ViT-B/32 on CIFAR-10 and CLIP ViT-B/32 on ImageNet-1K. In the scatter plots, points are checkpoints, color encodes layer-wise compression. Bar plots depict the accuracy gain Δ=Acc(FOLD )−Acc(MAG1 )\Delta=\mathrm{Acc}{(\text{{FOLD} })}-\mathrm{Acc}{(\text{{MAG1} })} , which remains positive and typically grows with compression, indicating that even under lightweight LayerNorm adaptation FOLD retains a consistent advantage over pruning. The figure follows the same setup as Fig. 2 in the main paper, but for MAG2 .
ParameterValues
Optimizersgd
Max / Base Learning Rate ( lr_max )from 0.0504 to 4.9759
SAM Strength ( sam_rho )0.0, 0.05, 0.1
Standard Augmentation ( augm )True, False
RandAugment ( randaug )True, False
Appendix
Table 3: Fixed and varying parameters for PreActResNet18 training on CIFAR-10.
Figure 10: FOLD outperforms MAG1 after full fine-tuning for 1–5 epochs on PreActResNet18 and ViT-B/32 on CIFAR-10. Results for PreActResNet18 ( top ) and ViT-B/32 ( bottom ). (a,d) accuracy of MAG1 vs. FOLD after 1 and 5 epochs of fine-tuning. (b,e) accuracy gap Δ\Delta over epochs, remaining positive. (c,f) accuracy trajectories from post-compression through 5 epochs, showing faster recovery and higher final accuracy for FOLD . The figure extends Fig. 3 in the main paper to PreActResNet18 and ViT-B/32 architectures where FOLD is benchmarked against MAG1 .
ParameterValues
Optimizersgd
Max / Base Learning Rate ( lr_max )from 0.005087 to 0.492936
SAM Strength ( sam_rho )0.0, 0.05, 0.1
Standard Augmentation ( augm )True, False
RandAugment ( randaug )True, False
Appendix
Table 4: Fixed and varying parameters for ViT-B/32 Base training on CIFAR-10.
ParamsHiddenIntermediateHeadsLayersStepsData (Tokens)
60M51213768811K1.3B
130M7682048121222K2.6B
Appendix
Table 5: Training hyperparameters of LLaMA-60M architecture.
Figure 11: Folded models retain their accuracy advantage after fine-tuning. Results for ResNet18 trained by Adam ( top row ) and PreActResNet18 trained by SGD on CIFAR-10 ( bottom row ): (a,d) compares post-compression accuracy of magnitude pruning with L2 criterion ( MAG2 ) versus folding ( FOLD ) after 1 and 5 epochs of fine-tuning. (b,e) show the accuracy gap between folding and pruning as a function of fine-tuning epochs, demonstrating that folding maintains a consistent lead, i.e. , the FOLD accuracy delta is positive. (c,f) illustrate accuracy trajectories before and after 5 epochs of fine-tuning for both methods, highlighting that folded models recover accuracy faster and reach higher final performance than pruned models. The figure extends Fig. 3 in the main paper and Fig. 10 in the appendix to MAG2 .
Figure 12: FOLD outperforms MAG2 after full fine-tuning for 1–5 epochs on ViT-B/32 and CLIP ViT-B/32. Results for ViT-B/32 on CIFAR-10 ( top ) and CLIP ViT-B/32 on ImageNet-1K ( bottom ). (a,d) accuracy of MAG2 vs. FOLD after 1 and 5 epochs of fine-tuning. (b,e) accuracy gap Δ\Delta over epochs, remaining positive. (c,f) accuracy trajectories from post-compression through 5 epochs, showing faster recovery and higher final accuracy for FOLD . The figure extends Fig. 3 in the main paper and Fig. 10 in the appendix to MAG2 .
Figure 13: Learning rate modulates folding’s edge. Post-compression accuracy of MAG2 and FOLD across learning rates: ResNet18 with Adam (a) and SGD (b) , PreActResNet18 (c) , and ViT-B/32 (d) . FOLD typically leads at moderate–low rates; the gap shrinks or reverses at very high rates, and closes again at extremely small rates. The same setup as in Fig. 5 in the main paper, but for MAG2 .
Figure 14: SAM can boost model compression. Post-compression accuracy under training with / without SAM. (a) ResNet18 (Adam), no L1. (b) ResNet18 (Adam), L1 =10−5=10^{-5} . (c) PreActResNet18 (SGD), no L1. (d) ViT - B/32, no L1. The figure extends the results in Fig. 6 to MAG2 .
Figure 15: Random augmentations narrow the folding–pruning gap. Post-compression accuracy on ResNet18 (CIFAR-10) trained without vs. with random augmentations: (a) Adam, (b) SGD, (c) PreActResNet, (d) ViT-B/32. The figure extends Fig. 2 to MAG2 .
Figure 16: ResNet18: Weight Decay. Test accuracy of ResNet18 checkpoints trained with varying weight decay values. Weight decay does not diminish the advantage of FOLD compared to MAG2 , especially for SGD-trained models.
weight_decaywarmup_stepsmax_lrPPL ↓\downarrow 0% sparsityPPL ↓\downarrow MAG2 (20%)PPL ↓\downarrow FOLD (20%)PPL ↓\downarrow MAG2 (50%)PPL ↓\downarrow FOLD (50%)
0.0111000.00123.9039.8839.48236.16308.77
0.0122000.00123.9938.7539.61259.79469.25
0.0133000.00124.0838.5439.10289.67451.27
0.011000.00124.0142.3942.27270.31477.70
0.022000.00124.1240.0141.53239.25489.48
0.033000.00124.1938.7240.31277.10531.09
Appendix
Table 6: Evaluation of FOLD and MAG2 on LLaMA-130M (in addition to LLaMA-60M evaluations in Tab. 1 ). We train and evaluate 18 LLaMA-family models with 130M parameters on C4 while varying max_lr, warmup steps, and weight decay. Columns show perplexity of the pretrained model (0% sparsity) and perplexity after structured magnitude pruning and folding with 20% and 50% sparsity in FFN blocks. For higher learning rates, especially for the settings with the best achieved performance in each sparsity category (underlined), FOLD consistently outperforms MAG2 (bold).
Figure 17: Folding vs. magnitude pruning before REPAIR. The same setup as in Fig. 1 and Fig. 8 for CNNs (ResNet18 and PreActResNet18 on CIFAR-10), but the performance is compared for both pruning and folding before REPAIR. Top row: MAG1 , bottom row: MAG2 . In both cases, folding shows stronger performance already before data-based REPAIR is applied.
Figure 18: Uncompressed model accuracy vs. performance difference ΔAccuracy=Acc(FOLD )−Acc(MAG )\Delta\text{Accuracy}=\mathrm{Acc}{(\text{{FOLD} })}-\mathrm{Acc}{(\text{{MAG} })} . The same setup as in Fig. 1 and Fig. 8 . Top row: MAG1 , Bottom row: MAG2 . Model folding shows strong performance on models of different quality, with amplified effect on high-performing models (especially on ResNet18, SGD and ViT-B/32).
Figure 19: Worst-case ℓ∞\ell_{\infty} sharpness as a function of compression ratio across architectures and pruning methods. Each subplot reports the sharpness distribution for independently trained models at three perturbation radii (ρ=10−4, 5×10−4, 10−3)(\rho=10^{-4},\,5\times 10^{-4},\,10^{-3}) . Panels (a)–(d) show FOLD sharpness for ResNet18 trained with Adam and SGD, PreActResNet18, and CLIP ViT-B/32, respectively. Panels (e)–(h) show the corresponding results for MAG1 . Observed trends: (i) Sharpness generally increases with compression ratio up to moderate levels before flattening or dropping at extreme compression. (ii) FOLD produces on-average lower sharpness than MAG1 . (iii) Transformer models (CLIP ViT-B/32) experience substantially sharper solutions under compression compared to residual networks. These patterns indicate that FOLD maintains flatter loss landscapes across a wide range of settings, whereas MAG more often drives models toward sharper and less stable minima.
Figure 20: Correlation between Δ\Delta -sharpness and Δ\Delta -accuracy across architectures and pruning baselines. Each column shows the relationship between pruning–induced differences in worst-case ℓ∞\ell_{\infty} sharpness ( Δ\Delta sharpness = FOLD −- mag ) and differences in test accuracy ( Δ\Delta accuracy = FOLD −- mag ), for the pair of pruning methods indicated below each plot. Color encodes the layer-wise compression ratio. The tables underneath each subplot report Pearson and Spearman correlations at every compression ratio, quantifying how predictive the sharpness difference is of the accuracy difference. Results are shown for FOLD vs MAG1 for ResNet18 trained with Adam and SGD, PreActResNet18 and CLIP ViT-B/32. Statistics are computed over 200 independently trained ResNets18 and 50 PreActResNet18 on CIFAR-10, and 72 CLIP ViT-B/32 models on ImageNet-1K. Correlations use the sharpness value at ρ=5×10−4\rho=5\times 10^{-4} . Results at ρ=10−4\rho=10^{-4} and ρ=10−3\rho=10^{-3} are qualitatively very close.
Figure 21: Sharpness-accuracy trade-off between FOLD and MAG1 for ResNet18 trained with Adam. Each column corresponds to a layer-wise compression ratio (0.1–0.9), and the three rows group models by learning rate, SAM configuration (including ρ\rho ), and RandAug usage. Points show Δ\Delta worst-case ℓ∞\ell_{\infty} sharpness ( FOLD −- MAG1 ) vs. Δ\Delta test accuracy ( FOLD −- MAG1 ). Observations: (1) The difference in model sharpness strongly predicts the difference in performance between FOLD and MAG1 across almost all pruning ratios. (2) Learning rate: higher learning rates lead to more dispersed sharpness changes. For models trained with Adam using high learning rates, MAG1 moves the model along a less sharp path compared to FOLD , which struggles to catch up. However, the behavior flips for moderate and low learning rates. (3) SAM/ ρ\rho : using SAM reduces the variability in the sharpness shift between methods, especially for larger ρ\rho . Δ\Delta sharpness gets closer to zero. (4) RandAug: augmentation show little specific visible trend.
Figure 22: Sharpness-accuracy trade-off between FOLD and MAG1 for ResNet18 trained with SGD. Each column corresponds to a layer-wise compression ratio (0.1–0.9), and the three rows group models by learning rate, SAM configuration (including ρ\rho ), and RandAug usage. Points show Δ\Delta worst-case ℓ∞\ell_{\infty} sharpness ( FOLD −- MAG1 ) vs. Δ\Delta test accuracy ( FOLD −- MAG1 ). Observations: (1) For SGD, the relationship between Δ\Delta sharpness and Δ\Delta accuracy is visibly weaker and more scattered than for Adam, reflecting the flatter and more anisotropic minima found by SGD. (2) Learning rate: Unlike Adam, higher SGD learning rates do not systematically increase the sharpness gap between the methods. For most ratios, FOLD tends to remain less sharp than MAG1 , producing positive Δ\Delta accuracy even when Δ\Delta sharpness is near zero. (3) SAM/ ρ\rho : SAM has a strong flattening effect under SGD— Δ\Delta sharpness clusters tightly around zero, and the accuracy advantage of FOLD becomes more stable as ρ\rho increases. (4) RandAug: Augmentation increases robustness to pruning under SGD, reducing the spread in Δ\Delta accuracy and further weakening the sharpness–accuracy link. Overall, SGD-trained models exhibit a regime where FOLD consistently follows a gentler sharpness trajectory than MAG1 , leading to a clearer accuracy advantage at moderate pruning ratios.
MethodParamsComp. time [s]Comp. peak mem [MB]Lat. [ms/batch]Lat. [ms/img]FLOPs [MFLOPs/img]Fwd peak mem [MB]
Original11,172,170––3.690.0288557.65214.30
Fold4,008,3469.48157.473.170.0248199.05170.53
Mag4,008,3461.77115.223.150.0246199.05169.82
Appendix
Table 7: Runtime characteristics of PreActResNet18 before and after compression ( 64.1%64.1\% parameter reduction). Latency is measured for a full batch. FLOPs are reported per image. Comp. time and comp. peak mem refer to the overhead of running the compression method once.
MethodParamsComp. time [s]Comp. peak mem [MB]Lat. [ms/batch]Lat. [ms/img]FLOPs [MFLOPs/img]Fwd peak mem [MB]
Original151,790,313––19.8510.62032946.76684.18
Fold140,447,25392.833681.2317.3430.54202379.98636.97
Mag140,447,2532.627625.6117.3720.54292379.98637.88
Appendix
Table 8: Runtime characteristics of CLIP ViT-B/32 before and after compression ( 7.47%7.47\% parameter reduction). Latency is measured for a full batch. FLOPs are reported per image. Comp. time and Comp. peak mem refer to the one-off overhead of running the compression method.
LayerParams_foldParams_magΔp\Delta pFLOPs_foldFLOPs_magΔF\Delta FAct_foldAct_magΔa\Delta aNZ_foldNZ_magΔnz\Delta nz
conv11026102601050624105062403891238912038912389120
layer1.0.conv112996129960133079041330790403891238912038912389120
layer1.0.conv212996129960133079041330790403891238912038912389120
layer1.1.conv112996129960133079041330790403891238912038912389120
layer1.1.conv212996129960133079041330790403891238912038912389120
layer2.0.conv1259922599206653952665395201945619456019456194560
Appendix
Table 9: Per-layer comparison of PreActResNet18 after FOLD and MAG2 at compression ratio 0.40.4 . For each convolutional and linear layer we report parameters, per-image FLOPs, activation size, and the number of non-zero activations (effective activations). All per-layer differences are zero, confirming that parameters, FLOPs, activations, and effective activations are exactly matched between the two compressed models.
LayerParams_foldParams_magΔp\Delta pFLOPs_foldFLOPs_magΔF\Delta FAct_foldAct_magΔa\Delta aNZ_foldNZ_magΔnz\Delta nz
classification_head5130005130000000000000
transformer.resblocks.0.attn.out_proj2626562626560000000000
transformer.resblocks.0.mlp.c_fc105062410506240000000000
transformer.resblocks.0.mlp.c_proj104908810490880000000000
transformer.resblocks.1.attn.out_proj2626562626560000000000
transformer.resblocks.1.mlp.c_fc105062410506240000000000
Appendix
Table 10: Per-layer comparison of CLIP ViT-B/32 after FOLD and MAG2 at compression ratio 0.20.2 (global parameter reduction 7.47%7.47\% ). The table reports all convolutional and linear layers in the vision transformer and classification head, including their parameters, per-image FLOPs, activation sizes, and effective activations. The transformer.resblocks.* modules belong to CLIP’s text encoder. Because the ImageNet-1k fine-tuned variant evaluates only the vision encoder and classification head, the text encoder is not part of the forward graph. THOP therefore records zero FLOPs and zero activations for these layers, while their parameters remain included in the model. Note that the per-layer totals ( ≈8.36×107\approx 8.36\times 10^{7} parameters) are smaller than the full model parameter count ( ≈1.40×108\approx 1.40\times 10^{8} ) because this table excludes components without FLOPs, such as token embeddings, positional embeddings, and LayerNorm parameters, which are included in the global counts but not part of the per-layer FLOP/activation analysis. Several projection layers inside the attention blocks show zero FLOPs because CLIP implements attention using fused operations; these operations are profiled at the block level by THOP rather than attributed to the individual Linear submodules. All per-layer differences are zero, showing that FOLD and MAG2 produce structurally identical compressed models on every layer affected by compression.
Figure 23: Effect of increasing the retained rank by one is significantly lower than changing the compression method from MAG to FOLD . Comparison of (i) the effect of increasing the retained rank by one (blue curves) and (ii) the effect of switching from MAG2 to FOLD at the same nominal rank (orange curves). Each panel corresponds to a single weight matrix W\mathbf{W} in ResNet18 convolutional layers ( top ) and ViT-B/32 FFN layers ( bottom ) and shows the relative Frobenius error difference Δ\Delta as a function of the retained rank kk .
Figure 24: Relative reconstruction error of pruning, special folding Wf′\mathbf{W}_{f}^{\prime} (from the proof of Theorem 2.1 ), and optimal folding Wf∗\mathbf{W}_{f}^{*} for all layers of ResNet18 (top) and ViT-B/32 (bottom). For each layer, we report the normalized squared Frobenius error ∥W−W∙∥F2/∥W∥F2\|\mathbf{W}-\mathbf{W}_{\bullet}\|_{F}^{2}/\|\mathbf{W}\|_{F}^{2} at several keep ratios kp/mk_{p}/m , where ∙∈{MAG2,singleton,FOLD}\bullet\in\{\texttt{MAG2},\text{singleton},\texttt{FOLD}\} . MAG2 (blue) denotes structured magnitude pruning with kpk_{p} retained rows. The special fold Wf′\mathbf{W}_{f}^{\prime} (orange) merges all pruned rows into a single extra cluster ( kf=kp+1k_{f}=k_{p}+1 ). The optimal fold Wf∗\mathbf{W}_{f}^{*} (green) is the kk -means solution with kfk_{f} clusters. Across all layers, Wf′\mathbf{W}_{f}^{\prime} consistently outperforms pruning, and Wf∗\mathbf{W}_{f}^{*} yields the smallest error, empirically validating error(Wp)≥error(Wf′)≥error(Wf∗)\mathrm{error}(\mathbf{W}_{p})\geq\mathrm{error}(\mathbf{W}_{f}^{\prime})\geq\mathrm{error}(\mathbf{W}_{f}^{*}) .
More in this topic

Model Compression

Model compression reduces the storage, computation, or latency of machine-learning models while preserving task performance.

See all 118 papers

Explore similar work

CardsList
Feb 27, 2026

GRAIL: Post-hoc Compensation by Linear Reconstruction for Compressed Networks

Structured deep model compression methods are hardware-friendly and substantially reduce memory and inference costs. However, under aggressive compression, the resulting accuracy degradation often necessitates post-compression finetuning, which can be impractical due to missing labeled data or high training cost. We propose post-hoc blockwise compensation, called GRAIL, a simple zero-finetuning step applied after model compression that restores each block's input-output behavior using a small calibration set. The method summarizes hidden activations via a Gram matrix and applies ridge regression to linearly reconstruct the original hidden representation from the reduced one. The resulting reconstruction map is absorbed into the downstream projection weights, while the upstream layer is compressed. The approach is selector-agnostic (Magnitude, Wanda, Gram-based selection, or folding), data-aware (requiring only a few forward passes without gradients or labels), and recovers classic pruning or folding when the Gram matrix is near identity, indicating weak inter-channel correlations. Across ResNets, ViTs, and decoder-only LLMs, GRAIL consistently improves accuracy or perplexity over data-free and data-aware pruning or folding baselines in practical compression regimes, with manageable overhead and no backpropagation. The code is available at https://github.com/TWWinde/GRAIL_Compensation.

Wenwu Tang, Dong Wang, Lothar Thiele +1

Graz University of Technology · ETH Zurich, Switzerland · Complexity Science Hub, Austria

Neural Network CompressionNeural Network Pruning
AskPDF
Sep 14, 2026

Optimal Pruning for Neural Architectures using Fisher Information Distances

A new scheme for parameter pruning is introduced, derived from the differential-geometric distance in model space. Pruning a parameter sets its value to zero, representing a displacement of the model to the hypersurface on which that parameter vanishes. The minimal distance from the unpruned model to this hypersurface is naturally computed via the geodesic distance in the model space as determined by the Fisher information metric. This distance determines the true change in the model, and its performance, under pruning. By analysing progressively more faithful approximations of this geodesic distance a natural hierarchy of optimality for pruning methods is determined. This starts with the traditional magnitude pruning, then develops into new more sophisticated and effective pruning schemes. The method is demonstrated for both fully-connected networks and vision transformers, on MNIST and CIFAR-10, over the complete 00-100%100\% pruning range and across five random seeds. It outperforms pruning by parameter magnitude and by the local Fisher information alone in every architecture and dataset combination considered, on both accuracy and the Matthews correlation coefficient. Additionally, analysis of different levels of geodesic approximation produces intermediate pruning schemes that are computationally efficient and maintain near-optimal performance. This geometric picture supplies not only a state-of-the-art pruning methodology for AI models, but also a verified and mathematically-motivated justification for pruning schemes.

David S. Berman, Yen-Yu Fu, Edward Hirst +1

Centre for Theoretical Physics, Queen Mary University of London, Mile End Road, London E1 4NS, UK · Institute of Mathematics, Statistics and Scientific Computing, University of Campinas (Unicamp), 13083-859, Brazil

Neural Network PruningNeural Network Compression
AskPDF
Oct 13, 2026

Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too

Post-training pruning enables efficient deployment, but conventional methods do not explicitly preserve internal-gradient alignment between sparse and dense models. We study novel connections between downstream performance of compressed models and the ability to retain gradient information via influence-function preservation. Central to this connection is a curvature-aware objective that accounts for both layer activations and backpropagated gradients. Despite its promise, this objective remains underexplored in pruning due to a lack of efficient solvers. We introduce CACTUS\texttt{CACTUS}, a novel ADMM-based optimizer supporting both pruning and sparse-plus-low-rank decompositions of pre-trained weights. Compared to LLM-Surgeon which targets the same objective, CACTUS\texttt{CACTUS} can achieve over an order-of-magnitude speedup. At 50%50\% pruning of Qwen3-4B, CACTUS\texttt{CACTUS} reduces the C4 perplexity gap to the dense model by 45%45\%, improves compressed-to-dense per-sample gradient cosine similarity on WikiText-2 from 0.6700.670 to 0.7550.755, and runs over 28×28\times faster than LLM-Surgeon on substantially weaker hardware. By preserving influence functions, CACTUS\texttt{CACTUS} supports effective data selection. On Llama-3.2-3B, fine-tuning a fresh dense model on the 5%5\% of data selected by CACTUS\texttt{CACTUS} achieves 49.44%49.44\% accuracy on BIG-Bench Hard, compared with 47.22%47.22\% using ALPS and 49.54%49.54\% using dense-model selection. Finally, integrating CACTUS\texttt{CACTUS} with LoGRA, a recent gradient-projection strategy, on Qwen3-4B with MLP layers pruned to 2:4 sparsity and a rank-64 correction retains a 0.7110.711 cosine alignment with dense LoGRA-projected C4 gradients.

Mehdi Makni, Natalia Ponomareva, Rahul Mazumder

Operations Research Center, Massachusetts Institute of Technology · Google Research

LLM PruningNeural Network PruningTransformer PruningInfluence FunctionsStructured Sparsity
AskPDF