GRAIL: Post-hoc Compensation by Linear Reconstruction for Compressed Networks

Wenwu Tang; Dong Wang; Lothar Thiele; Olga Saukh

cs.LGFeb 27, 2026

GRAIL: Post-hoc Compensation by Linear Reconstruction for Compressed Networks

Authors: Wenwu Tang, Dong Wang, Lothar Thiele, Olga Saukh

Organizations: Graz University of Technology · ETH Zurich, Switzerland · Complexity Science Hub, Austria

Abstract

Structured deep model compression methods are hardware-friendly and substantially reduce memory and inference costs. However, under aggressive compression, the resulting accuracy degradation often necessitates post-compression finetuning, which can be impractical due to missing labeled data or high training cost. We propose post-hoc blockwise compensation, called GRAIL, a simple zero-finetuning step applied after model compression that restores each block's input-output behavior using a small calibration set. The method summarizes hidden activations via a Gram matrix and applies ridge regression to linearly reconstruct the original hidden representation from the reduced one. The resulting reconstruction map is absorbed into the downstream projection weights, while the upstream layer is compressed. The approach is selector-agnostic (Magnitude, Wanda, Gram-based selection, or folding), data-aware (requiring only a few forward passes without gradients or labels), and recovers classic pruning or folding when the Gram matrix is near identity, indicating weak inter-channel correlations. Across ResNets, ViTs, and decoder-only LLMs, GRAIL consistently improves accuracy or perplexity over data-free and data-aware pruning or folding baselines in practical compression regimes, with manageable overhead and no backpropagation. The code is available at https://github.com/TWWinde/GRAIL_Compensation.

Figures & tables

Figure 1: GRAIL : GRAm-Integrated Linear compression workflow. After a structured compression decision (pruning / folding) narrows the producer’s hidden width, we run a small, unlabeled calibration set through the upstream blocks and collect activations at consumer input. From these activations, GRAIL forms the Gram matrix of second-order statistics and solves a ridge regression that reconstructs the original hidden representation from the reduced one. The resulting linear compensation is merged into the consumer projection weights, while the producer is narrowed by selection or clustering. This one-shot, training-free step restores each block’s input–output behavior and applies uniformly to pruning and folding across CNNs, ViTs, and LLMs.

Panel of Figure 2 — Figure 2: GRAIL on 576 SGD-trained ResNet-18 models on CIFAR-10. Using checkpoints from Saukh et al. [22] , we find that GRAIL consistently improves both pruning (most notably at low compression ratios) and folding (at moderate to high ratios), as shown in panels (a) and (c). It also outperforms REPAIR [ 34 ] , further narrowing the gap to models fine-tuned for 5 epochs (panel (b)).

Panel of Figure 3 — Figure 3: GRAIL on 72 CLIP ViT-B/32 checkpoints on ImageNet-1K. Using checkpoints from Wortsman et al. [36] , we find that GRAIL consistently improves both pruning and folding (panels (a) and (c)), with pruning benefiting more strongly for all compression ratios. Consequently, GRAIL -compensated folded models trail their GRAIL -compensated pruned counterparts.

Dataset	Method	10%	20%	30%	40%	50%	60%	70%
C4	ZipLM	8.26	9.86	12.26	16.78	21.28	35.76	2291.52
	Wanda	8.25	9.88	12.49	20.80	131.54	155.41	508.51
	Wanda + GRAIL	8.05	9.28	11.26	15.50	25.97	213.92	2127.03
	Wanda++	8.07	9.32	11.74	15.89	26.72	77.35	132.58
	Wanda++ + GRAIL	8.05	9.28	11.26	15.05	23.75	187.21	1688.64
	SlimGPT	11.41	18.80	23.51	57.75	235.56	1288.51	4049.08

Table 1: Perplexity (

\downarrow

) comparison on LLaMA-2-7B under different sparsity levels across three datasets (C4, PTB, WikiText2) with 2048 sequence length and 128 calibration samples.

Sparsity	Method	ARC-C	ARC-E	HellaSwag	PIQA	BoolQ	Winogrande
20%	ZipLM	0.3763	0.7083	0.5116	0.7388	0.5697	0.6685
	Wanda	0.3865	0.7096	0.5452	0.7633	0.7141	0.6417
	Wanda + GRAIL	0.4164	0.7382	0.5348	0.7655	0.7416	0.6843
	Wanda++	0.3959	0.7256	0.5436	0.7688	0.7083	0.6827
	Wanda++ + GRAIL	0.4104	0.7357	0.5357	0.7644	0.7446	0.6772
	SlimGPT	0.3200	0.5829	0.4781	0.7133	0.6642	0.5359

Table 2: Zero-shot accuracy (

\uparrow

) comparison on LLaMA-2-7B for different model compression methods with and without GRAIL compensation calibrated on C4 dataset across six benchmarks.

Model	Overhead Time (s)		Overhead Memory (MB)
	Calibration	Compensation	Calibration	Compensation
ResNet	0.19	0.10	162.02	22.00
ViT	0.20	0.04	161.62	5.76
CLIP	0.95	0.16	300.00	139.54
LLaMA-2-7B	58.077	3.16	1137.12	3367.00

Table 3: Overhead of GRAIL compensation on vision and language models. The calibration step collect activation statistics, while the compensation step comprises GRAIL computations.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: GRAIL on 125 ViT-B/32 models on CIFAR-10. Using checkpoints from Andriushchenko et al. [35] , we find that GRAIL consistently improves pruning and in most cases also folding, as shown in panels (a) and (c). However, similarly to CLIP ViT-B/32 results reported in the main paper, GRAIL -compensated folded models lag behind GRAIL -compensated pruned models.

Panel of Figure 7 — Figure 7: GRAIL on ResNet-18 and ViT-B/32 under random folding and pruning. Across both architectures, GRAIL consistently improves the accuracy of compressed models, as seen in the before/after scatter plots (left) and the accuracy gains across compression ratios (right) for all four settings: ResNet-18 folding (a), ResNet-18 pruning (b), ViT-B/32 folding (c), and ViT-B/32 pruning (d).

Panel of Figure 8 — Figure 8: GRAIL across pruning and folding compression for ResNet-18, ViT-B/32, and CLIP ViT-B/32. From left to right: Folding, Mag-L1, Mag-L2, and Wanda. Across all architectures and reduction methods, GRAIL produces a consistent upward shift in accuracy, demonstrating robust post-compression recovery.

Neural Network Compression

Neural network compression reduces model size, memory, communication, or computation while preserving useful accuracy.

See all 209 papers

Cut Less, Fold More: Model Compression through the Lens of Projection Geometry
Feb 20, 2026Olga Saukh, Dong Wang, Haris Šikić +2Model Compression Neural Network Compression
Graz University of Technology · Complexity Science Hub · Swiss Data Science Center +1
IrekoGPT: Turning Structured Pruning into Post-Hoc Slimmable LLMs
Sep 30, 2026Pietro Moriello, Pietro Buzzega, Angelo Porrello +1LLM Compression LLM Inference Acceleration
AImageLab, University of Modena and Reggio Emilia, Italy
Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too
Oct 13, 2026Mehdi Makni, Natalia Ponomareva, Rahul MazumderLLM Pruning Neural Network Pruning
Operations Research Center, Massachusetts Institute of Technology · Google Research

Feb 20, 2026J/K move · Enter open · S save

Cut Less, Fold More: Model Compression through the Lens of Projection Geometry

Olga Saukh, Dong Wang, Haris Šikić, Yun Cheng+1

Graz University of Technology · Complexity Science Hub · Swiss Data Science Center+1

Model Compression Neural Network Compression Neural Network Pruning

Open paper AskPDF arXiv

Compressing neural networks without retraining is vital for deployment at scale. We study calibration-free compression through the lens of projection geometry: structured pruning is an axis-aligned projection, whereas model folding performs a low-rank projection via weight clustering. We formalize both as orthogonal operators and show that, within a rank distance of one, folding provably yields smaller parameter reconstruction error, and under mild smoothness assumptions, smaller functional perturbations than pruning. At scale, we evaluate >1000 checkpoints spanning ResNet18, PreActResNet18, ViT-B/32, and CLIP ViT-B/32 on CIFAR-10 and ImageNet-1K, covering diverse training hyperparameters (optimizers, learning rates, augmentations, regularization, sharpness-aware training), as well as multiple LLaMA-family 60M and 130M parameter models trained on C4. We show that folding typically achieves higher post-compression accuracy, with the largest gains at moderate-high compression. The gap narrows and occasionally reverses at specific training setups. Our results position folding as a geometry-aware, calibration-free alternative to pruning that is often superior in practice and principled in theory.

Abstract

Neural Network Compression

Explore similar work

Cut Less, Fold More: Model Compression through the Lens of Projection Geometry

IrekoGPT: Turning Structured Pruning into Post-Hoc Slimmable LLMs

Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too