Secure Transformer inference protects sensitive inputs but incurs substantial cryptographic overhead, with nonlinear operations such as Softmax and GeLU becoming major bottlenecks. Existing compression methods reduce nonlinear complexity, sequence-dependent computation, or model structure through separately defined compression variables. Under aggressive compression, however, these independently optimized perturbations can accumulate: at a matched compression level, stacking representative approximation, token-pruning, and model-pruning methods reduces ViT-S accuracy from 80.20% to 76.41%. We introduce DegreeSpar, which formulates secure Transformer compression as structured sparsification over nonlinear polynomial degrees. Polynomial degree directly controls the cost of secure nonlinear evaluation, while computation-aligned zero-degree structures expose token-level and model-dimension computation as removable within the same optimization space. DegreeSpar further incorporates approximation-aware training for low-degree Softmax and GeLU, enabling aggressive degree reduction and creating the optimization headroom required for structured computation removal. Across vision and language Transformers, DegreeSpar consistently improves the accuracy-latency trade-off across model scales, tasks, and sequence lengths, achieving speedups from 2.29x to 6.63x over the corresponding baselines. Under the same network setting, DegreeSpar achieves 92.68% accuracy on BERT/SST-2 in 110.55 s, compared with 92.66% in 167.26 s for CipherPrune, the closest prior hybrid secure-inference approach. These results establish structured polynomial degree as an effective shared optimization space for secure Transformer compression.
Figures & tables
Figure 1: (a) Secure Transformer inference. (b) Secure execution shifts a substantial fraction of inference cost toward nonlinear operations under WAN1 (400 Mbps, 4 ms) network conditions. (c) DegreeSpar organizes polynomial degrees into a shared structured optimization space. Entry-wise degree reduction lowers nonlinear cost, while token and dimension-aligned zero-degree sparsity enables the corresponding token-level and dimension-level computations to be removed.
Figure 2: Overview of DegreeSpar within a Transformer layer. Softmax and GeLU are represented by structured polynomial-degree matrices, MN×N and MN×D . Entry-wise degree reduction lowers nonlinear evaluation cost, while token-aligned and dimension-aligned degree sparsity expose removable token-level and dimension-level computation, respectively.
Figure 3: Token-aligned joint structured degree optimization. Tokens are aligned with their corresponding Softmax and GeLU degree variables according to significance scores. Computation-aligned group regularization jointly reduces these degree variables and induces removable token-aligned sparsity.
Figure 4: Dimension-aligned joint structured degree optimization. GeLU degree variables are aligned with their associated model dimensions and surrounding linear computation. The resulting degree-aligned sparsity enables the associated model weights and intermediate-dimension computation to be removed together.
Training
ViT-S
ViT-B
Naive degree-2
79.38%
78.12%
Approx-aware
80.13 %
81.28 %
Table 1: Validation of two key properties of DegreeSpar. (a) Accuracy under the same degree-2 approximation with and without approximation-aware training. (b) DegreeSpar versus independently stacked compression methods under a matched compression budget.
Method
LAN
WAN1
WAN2
Acc.
Baseline
152.26
373.11
1144.43
81.40%
WDPruning
120.54
221.37
911.65
80.76%
PowerSoftmax
150.49
323.64
940.15
81.35%
EViT
93.68
297.39
665.59
81.13%
DegreeSpar
56.85
95.57
366.17
81.24 %
Table 2: End-to-end comparison with existing secure Transformer compression approaches. (a) ViT-B/ImageNet-1K under different network conditions. (b) BERT/SST-2 comparison with CipherPrune; CipherPrune* is configured to match DegreeSpar’s latency. Latency is reported in seconds.
Model
Dataset
Seq. Len.
Baseline
DegreeSpar
Speedup
Acc.
Latency
Acc.
Latency
ViT-Small
ImageNet-1K
197
80.20%
538.52
80.12%
132.02
4.08×
ViT-Base
ImageNet-1K
197
81.40%
1144.43
81.24%
366.17
3.13×
ViT-Large
ImageNet-1K
197
81.89%
3171.59
80.95%
1384.31
2.29×
BERT-Base
IMDB
197
89.90%
1144.43
88.78%
423.67
2.70×
BERT-Base
CoLA
64
58.80%
241.12
58.06%
58.64
4.11×
Table 3: End-to-end performance across model scales, tasks, and sequence lengths under WAN2. Performance is reported as MCC (%) for CoLA and Accuracy (%) for all other tasks. Latency is reported in seconds.
Figure 5: Layer-wise inference-cost breakdown of DegreeSpar on ViT-S/ImageNet under WAN2.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Approx-EXP
#Seg.
f_less
f_square
Total
Approx-GeLU
#Seg.
#Degree
f_less
f_square
f_mul
Total
n=6
2
2.58
32.21
34.79
poly-6
4
6
6.31
6.91
17.79
31.01
n=5
2
2.58
29.70
32.28
poly-5
3
5
4.14
6.91
11.86
24.61
n=4
2
2.58
23.19
25.77
poly-4
3
4
4.14
6.91
5.93
16.98
n=3
2
2.58
16.10
18.68
poly-3
3
3
4.14
3.46
5.93
15.15
n=2
2
2.58
10.74
13.32
poly-2
3
2
4.14
3.46
0.00
7.60
n=1
2
2.58
5.94
8.52
poly-1
2
1
2.05
0.00
0.00
2.05
Appendix
Table 4: Secure computational cost of Softmax exponential and GeLU approximations at different degree levels under our secure-inference configuration.
Figure 6: Approximation behavior of low-degree Softmax and GeLU and the training-time mechanisms used by approximation-aware training.
Figure 7: Comparison between attention-based and VCA token scoring across shallow ViT layers. Red squares indicate tokens predicted to be dropped by each scoring rule at the corresponding layer, without actually removing them. Token-drop predictions are made independently at each layer and are not propagated to subsequent layers.
Figure 8: Visualization of VCA-guided token skipping across Transformer layers. The ranking supports token removal from shallow layers while preserving the fixed per-layer retention budgets learned offline.
Information
Status
Per-layer retention/removal budget Kl
Public and fixed
Retained tensor shape
Public and fixed
Model architecture under the assumed threat model
Public and fixed
Client input
Hidden
Token values/features
Hidden
Token-significance scores
Hidden
Appendix
Table 5: Observable and protected information during DegreeSpar secure execution.
Method
ViT-S / ImageNet
ViT-B / ImageNet
BERT / IMDB
Baseline
80.09±0.2
81.45±0.1
89.78±0.2
DegreeSpar
80.31±0.2
81.05±0.2
88.57±0.2
Appendix
Table 6: Accuracy stability across five independent training runs. Results are reported as mean ± standard deviation (%).
λgrp
Latency (s)
Accuracy (%)
3×10−3
63.71
76.95
6×10−4
111.29
79.20
3×10−4 (default)
132.02
80.12
1.5×10−4
207.65
80.10
3×10−5
310.04
80.17
Appendix
Table 7: Sensitivity to the structured group-regularization coefficient on ViT-S/ImageNet-1K under WAN2.
Transition Layer
Accuracy (%)
2
78.85
3
79.80
4 (default)
80.12
5
80.09
6
79.94
Appendix
Table 8: Sensitivity to the VCA transition layer on ViT-S/ImageNet-1K.
Transformer inference studies how trained Transformer models produce outputs efficiently, accurately, and securely across hardware and context lengths.
Indian Institute of Information Technology Allahabad, India · National Institute of Electronics and Information Technology (NIELIT), India · Indian Institute of Technology Patna, India +2