CORP: Closed-Form One-shot Representation-Preserving Structured Pruning
Organizations: Purdue University West Lafayette, IN 47907, USA
Abstract
Transformers achieve strong accuracy but incur high compute and memory cost. Structured pruning reduces inference cost, but most methods rely on retraining or multi-stage optimization, which limits post-training deployment. We propose CORP, a closed-form one-shot structured pruning method that removes MLP dimensions and attention substructures using only unlabeled calibration data without gradients or fine-tuning. CORP formulates structured pruning as a representation recovery problem. It models removed components as affine functions of retained components and derives closed-form ridge regression solutions that fold compensation into model weights. This minimizes a layer-local affine/logit reconstruction objective under the calibration distribution. Experiments on ImageNet with DeiT reveal strong redundancy in both MLP and attention representations. With CORP, models retain high accuracy under aggressive sparsity. On DeiT-Huge, CORP achieves 83.24% Top-1 accuracy after pruning 50% of both MLP and attention structures.
Figures & tables
| Method | Pruning Scope | Selection Signal | Recovery | Finetune Free |
|---|---|---|---|---|
| SparseGPT | Weight/N:M weight | Hessian approx. | Second-order weight update | ✓ |
| OBC | Weight/quant. group | Gauss–Newton / OBS | Second-order weight update | ✓ |
| Wanda | Weight | Act. Mag. | None | ✓ |
| OPTIN | MLP channel/attn head/token | Trajectory score | None | ✓ |
| Olica | MLP channel / attn head dim. | Act. Mag. (FFN) / PCA (attn) | Low-rank ridge residual branch (FFN only) | ✓ |
| KCM | MLP channel | Output contribution | Scaling compensation | ✓ |
| Base | MLP | Attn | Both | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Top1 FLOPs P | Top1 | FLOPs | P | Top1 | FLOPs | P | Top1 | FLOPs | P |
| Tiny | 72.02 1.4 5.7 | 55.43 0.05 | 1.1 24.3% | 3.9 | 64.33 0.14 | 1.1 21.6% | 5.3 | 41.69 0.24 | 0.8 45.9% | 3.5 |
| Small | 79.72 5.0 22.1 | 69.35 0.12 | 3.6 28.1% | 15.0 | 72.52 0.06 | 4.2 16.0% | 20.3 | 58.47 0.26 | 2.8 44.1% | 13.2 |
| Base | 81.74 18.3 86.6 | 72.20 0.16 | 12.7 30.5% | 58.2 | 80.76 0.04 | 16.0 12.5% | 79.5 | 70.17 0.14 | 10.4 43.0% | 51.2 |
| Large | 84.58 63.5 304.4 | 82.06 0.06 | 43.7 31.2% | 203.7 | 83.62 0.04 | 56.2 11.6% | 279.2 | 80.34 0.06 | 36.3 42.8% | 178.5 |
| Huge | 84.97 173.0 632.1 | 84.09 0.04 | 119.0 31.2% | 422.3 | 84.22 0.03 | 153.0 11.7% | 579.7 | 83.24 0.05 | 98.7 42.9% | 369.9 |
| Model | Sparsity | Top-1 | Param | FLOPs | Lat | TP | Param | FLOPs | TP |
|---|---|---|---|---|---|---|---|---|---|
| (%) | (M) | (G) | (ms) | (fps) | (%) | (%) | (×) | ||
| DeiT-H | 0.0 | 84.97 | 632.1 | 172.8 | 30.84 | 43 | 0.0 | 0.0 | 1.00 |
| 0.1 | 84.96 | 579.7 | 153.7 | 27.70 | 48 | 8.3 | 11.1 | 1.12 | |
| 0.2 | 84.91 | 527.2 | 139.9 | 24.82 | 52 | 16.6 | 19.0 | 1.21 | |
| 0.3 | 84.66 | 474.8 | 126.2 | 22.73 | 57 | 24.9 | 27.0 | 1.32 | |
| 0.4 | 84.23 | 422.3 | 112.4 | 20.70 | 63 | 33.2 | 35.0 | 1.46 |
| Model | Params (M) | RMSE | mIoU | |
|---|---|---|---|---|
| Small | 22.1 14.6 | 0.3992 0.4459 | 0.9192 0.8750 | 0.4198 0.2549 |
| Base | 86.6 56.9 | 0.3730 0.4201 | 0.9374 0.9118 | 0.4368 0.3527 |
| Large | 304.4 198.9 | 0.3492 0.3445 | 0.9466 0.9413 | 0.4313 0.4011 |
| Huge | 1.1B 0.74B | 0.3308 0.3167 | 0.9529 0.9514 | 0.4428 0.4367 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Layer | Dim | Eff. Rank | Rank Ratio | Ratio | Act. Sparsity | |
|---|---|---|---|---|---|---|
| blocks.0.mlp.act | 3072 | 164.2 | 0.053 | 542 | 0.176 | 0.23 |
| blocks.1.mlp.act | 3072 | 136.4 | 0.044 | 336 | 0.109 | 0.30 |
| blocks.2.mlp.act | 3072 | 294.2 | 0.096 | 1118 | 0.364 | 0.16 |
| blocks.3.mlp.act | 3072 | 341.0 | 0.111 | 1577 | 0.513 | 0.08 |
| blocks.4.mlp.act | 3072 | 353.8 | 0.115 | 1804 | 0.587 | 0.06 |
| blocks.5.mlp.act | 3072 | 239.6 | 0.078 | 1783 | 0.580 | 0.07 |
| Protocol | Module | Pooled | Mean std. | Min | Max |
|---|---|---|---|---|---|
| Calibration | MLP (24 layers) | 0.4893 | 0.3944 0.1707 | 0.1126 | 0.7816 |
| Evaluation (oracle) | MLP | 0.4825 | 0.3909 0.1698 | 0.1133 | 0.7766 |
| Transfer (CORP) | MLP | 0.4736 | 0.3755 0.1789 | 0.0619 | 0.7735 |
| Calibration | Attention (384 heads) | 0.7825 | 0.4018 0.2336 | 0.1045 | 0.9995 |
| Evaluation (oracle) | Attention | 0.7775 | 0.3993 0.2323 | 0.1018 | 0.9995 |
| Transfer (CORP) | Attention | 0.7768 | 0.3970 0.2335 | 0.1008 | 0.9995 |
| Split | Module | Oracle (eval) | Transfer | Gap | Transfer mean std. |
|---|---|---|---|---|---|
| Class-disjoint (500/500 classes) | MLP | 0.4771 | 0.4538 | 0.0233 | 0.3437 0.1810 |
| Class-disjoint (500/500 classes) | Attention | 0.7795 | 0.7779 | 0.0016 | 0.3942 0.2352 |
| Same-distribution control | MLP | 0.4866 | 0.4687 | 0.0179 | 0.3674 0.1838 |
| Same-distribution control | Attention | 0.7779 | 0.7767 | 0.0012 | 0.3973 0.2339 |
| Joint sparsity | (default) | Mean std. ( – ) | |||
|---|---|---|---|---|---|
| 50% | 80.34 | 80.33 | 80.30 | 80.06 | 80.32 0.02 |
| 60% | 75.95 | 75.96 | 75.92 | 75.57 | 75.94 0.02 |
| 70% | 64.49 | 64.49 | 64.40 | 63.74 | 64.46 0.05 |
| Model | Sparsity | Top-1 | Param | FLOPs | Lat | TP | Param | FLOPs | TP |
|---|---|---|---|---|---|---|---|---|---|
| (%) | (M) | (G) | (ms) | (fps) | (%) | (%) | (×) | ||
| DeiT-T | 0.0 | 72.02 | 5.7 | 1.4 | 3.79 | 3514 | 0.0 | 0.0 | 1.00 |
| 0.1 | 71.26 | 5.3 | 1.2 | 4.95 | 3247 | 7.6 | 19.0 | 0.92 | |
| 0.2 | 69.15 | 4.8 | 1.1 | 5.01 | 3453 | 15.3 | 25.6 | 0.98 | |
| 0.3 | 64.46 | 4.4 | 1.0 | 4.84 | 3554 | 23.2 | 32.4 | 1.01 | |
| 0.4 | 55.73 | 4.0 | 0.9 | 4.89 | 3699 | 30.9 | 39.0 | 1.05 |