Secure Transformer inference protects sensitive inputs but incurs substantial cryptographic overhead, with nonlinear operations such as Softmax and GeLU becoming major bottlenecks. Existing compression methods reduce nonlinear complexity, sequence-dependent computation, or model structure through separately defined compression variables. Under aggressive compression, however, these independently optimized perturbations can accumulate: at a matched compression level, stacking representative approximation, token-pruning, and model-pruning methods reduces ViT-S accuracy from 80.20% to 76.41%. We introduce DegreeSpar, which formulates secure Transformer compression as structured sparsification over nonlinear polynomial degrees. Polynomial degree directly controls the cost of secure nonlinear evaluation, while computation-aligned zero-degree structures expose token-level and model-dimension computation as removable within the same optimization space. DegreeSpar further incorporates approximation-aware training for low-degree Softmax and GeLU, enabling aggressive degree reduction and creating the optimization headroom required for structured computation removal. Across vision and language Transformers, DegreeSpar consistently improves the accuracy-latency trade-off across model scales, tasks, and sequence lengths, achieving speedups from 2.29x to 6.63x over the corresponding baselines. Under the same network setting, DegreeSpar achieves 92.68% accuracy on BERT/SST-2 in 110.55 s, compared with 92.66% in 167.26 s for CipherPrune, the closest prior hybrid secure-inference approach. These results establish structured polynomial degree as an effective shared optimization space for secure Transformer compression.
Figures & tables
Figure 1: (a) Secure Transformer inference. (b) Secure execution shifts a substantial fraction of inference cost toward nonlinear operations under WAN1 (400 Mbps, 4 ms) network conditions. (c) DegreeSpar organizes polynomial degrees into a shared structured optimization space. Entry-wise degree reduction lowers nonlinear cost, while token and dimension-aligned zero-degree sparsity enables the corresponding token-level and dimension-level computations to be removed.
Figure 2: Overview of DegreeSpar within a Transformer layer. Softmax and GeLU are represented by structured polynomial-degree matrices, MN×N and MN×D . Entry-wise degree reduction lowers nonlinear evaluation cost, while token-aligned and dimension-aligned degree sparsity expose removable token-level and dimension-level computation, respectively.
Figure 3: Token-aligned joint structured degree optimization. Tokens are aligned with their corresponding Softmax and GeLU degree variables according to significance scores. Computation-aligned group regularization jointly reduces these degree variables and induces removable token-aligned sparsity.
Figure 4: Dimension-aligned joint structured degree optimization. GeLU degree variables are aligned with their associated model dimensions and surrounding linear computation. The resulting degree-aligned sparsity enables the associated model weights and intermediate-dimension computation to be removed together.
Training
ViT-S
ViT-B
Naive degree-2
79.38%
78.12%
Approx-aware
80.13 %
81.28 %
Table 1: Validation of two key properties of DegreeSpar. (a) Accuracy under the same degree-2 approximation with and without approximation-aware training. (b) DegreeSpar versus independently stacked compression methods under a matched compression budget.
Method
LAN
WAN1
WAN2
Acc.
Baseline
152.26
373.11
1144.43
81.40%
WDPruning
120.54
221.37
911.65
80.76%
PowerSoftmax
150.49
323.64
940.15
81.35%
EViT
93.68
297.39
665.59
81.13%
DegreeSpar
56.85
95.57
366.17
81.24 %
Table 2: End-to-end comparison with existing secure Transformer compression approaches. (a) ViT-B/ImageNet-1K under different network conditions. (b) BERT/SST-2 comparison with CipherPrune; CipherPrune* is configured to match DegreeSpar’s latency. Latency is reported in seconds.
Model
Dataset
Seq. Len.
Baseline
DegreeSpar
Speedup
Acc.
Latency
Acc.
Latency
ViT-Small
ImageNet-1K
197
80.20%
538.52
80.12%
132.02
4.08×
ViT-Base
ImageNet-1K
197
81.40%
1144.43
81.24%
366.17
3.13×
ViT-Large
ImageNet-1K
197
81.89%
3171.59
80.95%
1384.31
2.29×
BERT-Base
IMDB
197
89.90%
1144.43
88.78%
423.67
2.70×
BERT-Base
CoLA
64
58.80%
241.12
58.06%
58.64
4.11×
Table 3: End-to-end performance across model scales, tasks, and sequence lengths under WAN2. Performance is reported as MCC (%) for CoLA and Accuracy (%) for all other tasks. Latency is reported in seconds.
Figure 5: Layer-wise inference-cost breakdown of DegreeSpar on ViT-S/ImageNet under WAN2.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Approx-EXP
#Seg.
f_less
f_square
Total
Approx-GeLU
#Seg.
#Degree
f_less
f_square
f_mul
Total
n=6
2
2.58
32.21
34.79
poly-6
4
6
6.31
6.91
17.79
31.01
n=5
2
2.58
29.70
32.28
poly-5
3
5
4.14
6.91
11.86
24.61
n=4
2
2.58
23.19
25.77
poly-4
3
4
4.14
6.91
5.93
16.98
n=3
2
2.58
16.10
18.68
poly-3
3
3
4.14
3.46
5.93
15.15
n=2
2
2.58
10.74
13.32
poly-2
3
2
4.14
3.46
0.00
7.60
n=1
2
2.58
5.94
8.52
poly-1
2
1
2.05
0.00
0.00
2.05
Appendix
Table 4: Secure computational cost of Softmax exponential and GeLU approximations at different degree levels under our secure-inference configuration.
Figure 6: Approximation behavior of low-degree Softmax and GeLU and the training-time mechanisms used by approximation-aware training.
Figure 7: Comparison between attention-based and VCA token scoring across shallow ViT layers. Red squares indicate tokens predicted to be dropped by each scoring rule at the corresponding layer, without actually removing them. Token-drop predictions are made independently at each layer and are not propagated to subsequent layers.
Figure 8: Visualization of VCA-guided token skipping across Transformer layers. The ranking supports token removal from shallow layers while preserving the fixed per-layer retention budgets learned offline.
Information
Status
Per-layer retention/removal budget Kl
Public and fixed
Retained tensor shape
Public and fixed
Model architecture under the assumed threat model
Public and fixed
Client input
Hidden
Token values/features
Hidden
Token-significance scores
Hidden
Appendix
Table 5: Observable and protected information during DegreeSpar secure execution.
Method
ViT-S / ImageNet
ViT-B / ImageNet
BERT / IMDB
Baseline
80.09±0.2
81.45±0.1
89.78±0.2
DegreeSpar
80.31±0.2
81.05±0.2
88.57±0.2
Appendix
Table 6: Accuracy stability across five independent training runs. Results are reported as mean ± standard deviation (%).
λgrp
Latency (s)
Accuracy (%)
3×10−3
63.71
76.95
6×10−4
111.29
79.20
3×10−4 (default)
132.02
80.12
1.5×10−4
207.65
80.10
3×10−5
310.04
80.17
Appendix
Table 7: Sensitivity to the structured group-regularization coefficient on ViT-S/ImageNet-1K under WAN2.
Transition Layer
Accuracy (%)
2
78.85
3
79.80
4 (default)
80.12
5
80.09
6
79.94
Appendix
Table 8: Sensitivity to the VCA transition layer on ViT-S/ImageNet-1K.
Transformer inference studies how trained Transformer models produce outputs efficiently, accurately, and securely across hardware and context lengths.
Fully homomorphic encryption (FHE) enables computation on encrypted data, but practical encrypted Transformer inference is bottlenecked by the sequential composition of many nonlinear blocks. We study whether Structured Newton Layer Parallelism (SNLP) can make this inter-layer composition more FHE-friendly: each Transformer block still requires polynomial approximations for operations such as softmax and RMSNorm, but SNLP reduces the layerwise sequential nonlinear depth from L stages to a small number of solver iterations plus linear structured corrections. Using a simulation framework based on Chebyshev polynomial approximations, we measure error accumulation under sequential versus SNLP inference across 8 models and 4 architecture families. On a 0.5B IDN-trained model, SNLP reduces symbolic bootstraps from 53 to 20 (2.65x) with only +1.2% perplexity degradation, while lowering error amplification (1.36x vs. 1.42x). Across all tested models, SNLP has lower amplification than sequential inference. Ablations show that softmax approximation dominates the error budget and CKKS arithmetic noise is negligible in our setting, suggesting that SNLP is complementary to block-level FHE-friendly operator design rather than a replacement for it.
Ligong Han, Kai Xu, Hao Wang +3
MBZUAI IFM · Red Hat AI Innovation · MIT-IBM Watson AI Lab +3
Compressing transformer weights makes large language models cheaper to deploy. But each layer's compression introduces an error. These errors accumulate as the signal passes through later layers, and how they accumulate is not well understood. We measure this directly: at each layer, we take the ratio of output to input error, calling it rho. A value below one means the layer absorbs the error; above one means it grows. Computing rho on six transformers (117M to 8B parameters) yields three findings. (i) Errors at layer t scale downstream by the product of later rho values, predicting representation drift (Spearman r = -0.44, p < 10^-4). This explains why compressing early layers hurts more than late ones, and why depth-decreasing sparsity schedules outperform uniform ones. Across architecture families, however, model width and redundancy matter more than rho alone. (ii) Within a layer, naive pruning shows a ~600x spread in component sensitivity. Activation-aware pruning (Wanda) shrinks this to 3-7x; the ranking reverses across architectures, so fixed importance scores do not transfer. (iii) For depth pruning, ranking layers by how far rho is from one takes two forward passes. It beats ShortGPT's Block Influence with 1.6x lower perplexity at eight layers removed, and physical deletion delivers 1.22x wall-clock speed-up. A blend of the two criteria does best (perplexity 14.2, 60.0% downstream accuracy on LLaMA-2-7B). Twelve Lean 4 norm inequalities provide machine-checked per-matrix error bounds. The contraction profile thus gives a training-free instrument for two decisions: where to compress within layers, and which to remove.
Abhinaba Basu, Kumkum Basu, Koushik Deb
Indian Institute of Information Technology Allahabad, India · National Institute of Electronics and Information Technology (NIELIT), India · Indian Institute of Technology Patna, India +2
Post-training compression of transformer models commonly relies on truncated singular value decomposition (SVD) strategies leading to low-rank approximations of the original model weights. However, the underlying single shared subspace modeling can degrade accuracy even at moderate compression. Sparse dictionary learning provides a more flexible union-of-subspaces representation, but existing approaches often suffer from costly iterative dictionary and coefficient updates. We propose COMPOT (Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers), a training-free compression framework that uses a small calibration dataset to estimate a sparse weight factorization. COMPOT employs orthogonal dictionaries that enable closed-form Procrustes updates for the dictionary and analytical single-step sparse coding for the coefficients, reducing significantly the overall optimization cost. To handle heterogeneous layer sensitivity under a global compression budget, COMPOT further introduces a one-shot dynamic allocation strategy that adaptively redistributes layer-wise compression rates. Extensive experiments across diverse architectures and tasks show that COMPOT consistently delivers a superior quality-compression trade-off over strong low-rank and sparse baselines, while remaining fully compatible with post-training quantization for extreme compression.