Transformer-based models achieve strong performance on blind compressed image restoration (CIR) under unknown compression settings, but their densely stacked projection operators make deployment costly. We study training-free acceleration for such models through an empirical, operator-level analysis. Specifically, we profile where computation is concentrated, characterize channel-wise activation distributions under compressed inputs, and examine the singular-value decay of representative projection weights. These measurements identify the recurring projections in the attention, feed-forward, convolutional, and prompt-conditioning pathways as effective low-rank replacement targets that preserve the original inference graph. They further establish truncated SVD (TSVD) as a principled training-free baseline for blind CIR and reveal why a uniform low-rank objective is mismatched to the channel-imbalanced feature distributions of compressed images. Accordingly, we formulate Activation-Calibrated Singular Value Decomposition (ASVD) for this restoration setting, deriving low-rank factors entirely offline from pretrained weights and lightweight activation statistics gathered from only a few compressed calibration images. By weighting the decomposition with channel-wise activation magnitudes, our ASVD assigns larger penalties to errors on more active channels, requiring neither retraining nor online calibration and leaving the restoration pipeline intact. When instantiated on PromptCIR at rank 64, ASVD consistently improves PSNR-B over TSVD across four standard benchmarks under identical parameter and MAC budgets. At this operating point, the factorized model cuts parameters by 59% (34.75M to 14.34M) and computation by 16.1% (258.79 to 217.16 GMACs for a 256 x 256 input), enabling acceleration without retraining.
Figures & tables
Fig. 1: Empirical motivation for projection-level ASVD. (a) Parameter distribution and convolutional and linear MAC distribution across operator groups in PromptCIR, evaluated at an input resolution of 128×128 . (b) Output activations are channel-imbalanced: a relatively small fraction of channels often accounts for 50% of activation energy, while the fraction required for 90% energy varies by operator family. (c) The rank required to preserve 95% spectral energy also varies across operators; bars show the mean over representative layers and error bars show the min–max range.
Fig. 2: Overview of our ASVD. We select high-cost projection operators in a transformer-based blind CIR model, collect lightweight per-channel activation statistics on a small calibration set, and form diagonal scaling factors. Truncated SVD is applied to the scaled weights, and the low-rank factors are mapped back via inverse scaling, yielding factorized replacements that preserve the original inference flow.
Method
Rank
Params. (M)
LIVE1
BSDS500
ICB
DIV2K
JPEG
–
–
30.02/0.872/28.18
30.58/0.876/28.29
33.87/0.842/33.05
31.71/0.885/30.02
FBCNN [ 28 ]
–
71.92
32.08/0.904/31.44
32.41/0.903/31.45
35.90/0.865/36.33
33.96/0.918/33.41
Contrastive JPEG [ 29 ]
–
11.11
32.26/0.907/31.70
32.75 / 0.908 / 31.78
35.93/0.865/36.35
34.13/ 0.919 /33.62
PromptIR [ 30 ]
–
36.54
32.34/0.907/31.75
32.67 / 0.907 / 31.67
36.35/ 0.868 /36.75
34.38 / 0.922 / 33.81
PromptCIR [ 5 ]
–
34.75
32.50 / 0.909 / 31.91
32.23/0.906/31.35
36.49 / 0.866 / 36.88
34.25 / 0.919 / 33.71
PromptCIR-TSVD
64
14.34
31.75/0.898/30.89
31.71/0.896/30.62
34.91/0.850/35.03
33.19/0.907/32.44
TABLE I: Quantitative comparison on compressed image restoration benchmarks. Results are PSNR / SSIM / PSNR-B ( ↑ ). Best and second-best values at the displayed precision are bold and underlined. Parameter counts are per model; PromptCIR and its factorized variants use the same ensemble inference protocol.
Fig. 3: Visual comparison among the low-quality input (LQ), the ASVD output, the PromptCIR output, and the ground-truth reference (GT). Red boxes mark the zoomed region.
Fig. 4: Feature difference visualization on DIV2K at rank 64 , using our ASVD instantiated on the PromptCIR backbone. Top row: encoder stage ( enc1 ). Bottom row: decoder stage ( dec1 ). Left: absolute feature error between the base model and TSVD. Right: absolute feature error between the base model and ASVD. Errors are computed on high-activation channels only. ASVD exhibits lower-magnitude and more spatially coherent error responses.
Rank
TSVD
ASVD
8
28.26/0.854/28.21
28.34/0.846/28.39
16
30.35/0.879/30.05
30.23/0.874/30.11
32
30.82/0.889/30.61
31.69/0.892/31.36
64
33.19/0.907/32.44
33.10/0.907/32.56
96
34.15/0.918/33.64
34.15/0.918/33.69
128
34.22/0.918/33.69
34.22/0.918/33.67
TABLE II: Effect of rank on DIV2K. Results are PSNR / SSIM / PSNR-B ( ↑ ).
Images
PSNR
SSIM
PSNR-B
1
32.861±0.192
0.9053±0.0019
32.314±0.157
2
33.006±0.069
0.9058±0.0008
32.429±0.090
4
33.064±0.059
0.9065±0.0008
32.495±0.044
8
33.073±0.032
0.9067±0.0006
32.518±0.026
16
33.062±0.032
0.9070±0.0004
32.518±0.028
32
33.041±0.043
0.9070±0.0003
32.511±0.034
TABLE III: Calibration sample size on DIV2K (ASVD, rank 64). Mean ± sample standard deviation over five seeds.
Dataset
Test QF
[10,30]
[40,70]
[10,70]
LIVE1
10
27.490±0.004
27.487±0.006
27.490±0.004
30
31.280±0.008
31.284±0.012
31.284±0.013
50
32.927±0.014
32.935±0.012
32.934±0.016
70
34.447±0.031
34.459±0.025
34.453±0.035
90
36.595±0.070
36.630±0.051
36.605±0.077
DIV2K
10
29.130±0.005
29.128±0.005
29.135±0.007
TABLE IV: Sensitivity to the calibration QF range (rank 64, 16 images). PSNR mean ± sample standard deviation over three seeds. QF 90 is outside all calibration ranges.
Rank
Weighting
PSNR
SSIM
PSNR-B
Output NMSE
32
TSVD
30.821±0.000
0.8891±0.0000
30.611±0.000
0.0802±0.0000
Input only
31.453±0.165
0.8893±0.0004
31.135±0.160
0.0721±0.0003
Output only
30.282±0.178
0.8825±0.0002
30.350±0.090
0.0774±0.0001
Bilateral (ASVD)
31.661±0.135
0.8919±0.0013
31.310±0.101
0.0737±0.0004
Input only ( α=0.5 )
31.523±0.075
0.8916±0.0009
31.222±0.048
0.0693±0.0001
64
TSVD
33.194±0.000
0.9072±0.0000
32.437±0.000
0.0271±0.0000
TABLE V: Weighting-side ablation on DIV2K. Calibrated results are mean ± sample standard deviation over three seeds; TSVD is deterministic. Output NMSE ( ↓ ) is measured on common reference inputs.
Dataset
Uniform 64
Spectrum-based
LIVE1
31.69/0.899/31.03
31.74/0.899/30.97
BSDS500
31.71/0.898/30.81
31.72/0.898/30.69
ICB
34.79/0.847/35.12
35.20/0.853/35.40
DIV2K
33.10/0.907/32.56
33.28/0.908/32.61
TABLE VI: Rank allocation at matched MACs. Results are PSNR / SSIM / PSNR-B ( ↑ ). Both methods use the same fixed DIV2K calibration set.
Rank
Params.
MACs
TSVD
ASVD
Speedup
(M)
(G)
(ms)
(ms)
( × )
32
9.79
164.15
208.26
213.19
1.37
64
14.34
217.16
215.72
238.42
1.23
96
18.44
254.38
228.53
240.56
1.21
128
21.80
263.19
244.83
246.16
1.19
TABLE VII: Complexity and 256×256 forward latency. Parameters and MACs are per model; MACs include attention matrix products. Latencies are in ms; speedup is ASVD relative to the original.
Activation-aware low-rank factorization techniques yield strong compression results but are generally confined to linear layers, while existing whitening-based theory typically makes an implicit full-rank assumption on activations. We introduce a layer representation framework that extends activation-aware factorization beyond linear layers, including standard and grouped convolutions. Within this framework, our whitening-based formulation is more general than prior ones, naturally covering rank-deficient activations, and yields an optimal low-rank projection that attains the reconstruction error of the best low-rank approximation to layer activations. The resulting singular spectrum provides a closed-form per-layer distortion proxy, which we use to allocate per-layer ranks under explicit FLOP or parameter-count budgets via a Lagrangian relaxation with negligible overhead. Together, these components form BALF, an end-to-end pipeline for efficient vision model compression. Across CNNs and vision transformers on CIFAR-10 and ImageNet-1K, BALF generally achieves higher accuracy than SVD-based factorization baselines at matched FLOP or parameter count targets and remains competitive with other fine-tuning-free compression techniques.
David González-Martínez
Department of Computer Science and Artificial Intelligence, Universidad de Sevilla · Tübingen AI Center
Post-training compression of transformer models commonly relies on truncated singular value decomposition (SVD) strategies leading to low-rank approximations of the original model weights. However, the underlying single shared subspace modeling can degrade accuracy even at moderate compression. Sparse dictionary learning provides a more flexible union-of-subspaces representation, but existing approaches often suffer from costly iterative dictionary and coefficient updates. We propose COMPOT (Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers), a training-free compression framework that uses a small calibration dataset to estimate a sparse weight factorization. COMPOT employs orthogonal dictionaries that enable closed-form Procrustes updates for the dictionary and analytical single-step sparse coding for the coefficients, reducing significantly the overall optimization cost. To handle heterogeneous layer sensitivity under a global compression budget, COMPOT further introduces a one-shot dynamic allocation strategy that adaptively redistributes layer-wise compression rates. Extensive experiments across diverse architectures and tasks show that COMPOT consistently delivers a superior quality-compression trade-off over strong low-rank and sparse baselines, while remaining fully compatible with post-training quantization for extreme compression.
Model compression is key to mitigate deployment challenges of ever growing machine learning models. In this area of research, singular value decomposition (SVD)-based compression offers a compelling trade-off between computational efficiency and model accuracy. Fisher-weighted SVD in particular provides principled, loss-aware compression. However, we find that improving the fidelity of Fisher approximation used in the compression is poorly predictive of post-compression accuracy for Vision Transformers (ViTs). Motivated by this observation, we propose FACTS, a structured Fisher Approximation tailored to Compressing ViTs with Fisher-weighted SVD, which enforces token-local aggregation while preserving within-token activation-gradient dependence. Additionally, we introduce a fast Constrained Rank Search (CoRS), that optimizes layer-wise rank allocation while adhering to a fixed floating point operation (FLOP) constraint. Extensive experiments across ViTs and hybrid architectures demonstrate that FACTS consistently improves accuracy-efficiency trade-offs without requiring finetuning. Notably, it outperforms the strongest SVD baseline by up to +5.8 percentage points (p.p.) Top-1 on Swin-B, with further gains driven by our search method. Code is available at https://github.com/MoritzTho/FACTS.
Moritz Thoma, Maximilian Groezinger, Maximilian Forstenhäusler +7
Technical University of Munich, DE · BMW Group, DE · University of Glasgow, GB +1