GrayShield: Bit-Level Sanitization for Transformer Model Supply-Chain Security

Armstrong Foundjem; Tsung-Hsien Chuang; Foutse Khomh; Mohamed Amine Merzouk

cs.CROct 3, 2026

GrayShield: Bit-Level Sanitization for Transformer Model Supply-Chain Security

Authors: Armstrong Foundjem, Tsung-Hsien Chuang, Foutse Khomh, Mohamed Amine Merzouk

Organizations: Mila - Quebec Artificial Intelligence Institute and Polytechnique de Montréal, QC, Canada · Research Center for Information Technology Innovation, Academia Sinica, Taipei, Taiwan · Mila - Quebec Artificial Intelligence Institute and McGill University Montréal, QC, Canada

Abstract

Transformer models such as BERT and Vision Transformer~(ViT) achieve strong performance via densely parameterized attention backbones. However, the least significant bits~(LSBs) of their 32-bit floating-point weights can be abused as covert channels to conceal malicious payloads, posing a serious threat to the AI model supply chain. We propose \GS (\GSabbr), a lightweight, post-training, zero-data sanitization method that completely replaces the declared mantissa-LSB channel with a Gray-code-guided low-transition sequence. Complete payload-independent overwrite, whether keyed or public, makes the sanitized target bits independent of the embedded payload and gives that declared channel zero capacity. Gray coding supplies overwrite structure, while a keyed per-tensor phase supplies pattern diversity. Benchmarked against seven post-training defenses on four Transformer model presets and two real-world malware payloads, \GSabbr maintains sub- $1\%$ accuracy impact and achieves $49.96\pm0.66$ percentage-point Recovery Reduction (RR) under five implemented attacker variants. Because pre-defense recovery is effectively $100\%$ , RR near 50 percentage points corresponds to post-sanitization bit accuracy at binary chance. Its main empirical advantage is stable near-chance sanitization with substantially smaller weight-distribution shift than the evaluated near-chance baselines PatternMask (PM) and Post-Training Quantization (PTQ).

Figures & tables

Figure 1: GrayShield evaluation pipeline. (A) Inputs define the empirical study space: pretrained models, evaluation datasets, and malware payloads from MalwareBazaar. (B) LSB injection overwrites the lowest $x$ mantissa bits at depths $x\in\{4,8,16,19,21,23\}$ to produce (C) poisoned models. (D) Seven sanitization strategies transform poisoned checkpoints into defended models. (E) Payload recovery is assessed by LSB extraction, and (F) security, fidelity, and utility metrics are computed. All four research questions are answered from the complete pipeline: RQ1 characterises the capacity boundary of injection; RQ2 compares defense effectiveness against the Tier-1 naive attacker; RQ3 evaluates adaptive robustness under five Tier-2 attacker variants (re-running stages B–F with varied attacker encodings); and RQ4 synthesises the Pareto trade-off from RQ2–RQ3 results. Dashed arrows denote analytical dependencies across research questions.

Figure 2: RQ1 — Injection feasibility across six overwrite depths (

x\in\{4,8,16,19,21,23\}

), four Transformer architectures, and two entropy-stratified payloads (

n=48

runs). (a) Cosine deviation vs. depth on a log scale; Spearman

\rho=0.954

p<0.001

. (b) Relative

\ell_{2}

distance vs. depth on a log scale;

\rho=0.954

p<0.001

. (c) Signed accuracy drop

\Delta\mathrm{Acc}

vs. depth. The green region (

x\leq 16

) marks depths satisfying the stealth constraints, while the red region (

x\geq 19

) marks the onset of measurable degradation, with a worst-case high-entropy drop of

0.78\%

. Mann–Whitney comparison between low and high depth regimes yields

p=0.005

. (d) Spearman depth–metric association with 95% Fisher-

z

confidence intervals.

Defense	Data	RR (pp)	$\Delta\mathrm{Acc}$	$W_{1}$	Target erasure
GS	None	$49.72\pm 0.54$	$-0.10\%$	$2.6{\times}10^{-5}$	Prop. 4
PM	None	$50.00\pm 2.64$	$+0.09\%$	$7.6{\times}10^{-4}$	Prop. 4
PTQ	Calibration	$49.93\pm 0.40$	$+0.07\%$	$8.5{\times}10^{-4}$	None
FT	Clean set	$45.34\pm 1.11$	$+0.38\%$	$7.6{\times}10^{-7}$	None
GN	None	$31.49\pm 15.08$	$-0.02\%$	$5.8{\times}10^{-7}$	None
SWP	None	$10.01\pm 1.50$	$+0.05\%$	$4.6{\times}10^{-5}$	None

Table 1: RQ2 defense comparison. RR denotes Recovery Reduction (pp); values near 50 pp indicate chance-level recovery, the target under complete payload-independent overwrite.

Figure 3: RQ2 — Defense effectiveness and multi-metric trade-offs. (a) Recovery Reduction (RR) across defenses, stratified by payload entropy. GrayShield reaches near-chance disruption ( $\approx 50$ pp) consistently. (b) Accuracy drop ( $|\Delta\mathrm{Acc}|$ ), where lower values indicate better utility preservation. (c) Cumulative distribution of RR, showing that GS, PM, and PTQ concentrate near the 50 pp target while weaker defenses remain far below. (d) Radar chart of normalized deployment metrics. (e) Defense-by-metric Z-score heatmap, where green indicates above-average performance. (f) Grouped normalized bar chart.

Figure 4: RQ3 — Adaptive attacker robustness. (a) RR across attacker variants shows that GS remains stable around 50 pp. (b) Defense-level RR distributions show significant differences across defenses, with GS maintaining high and consistent performance. (c) Effectiveness–stability trade-off shows GS closest to the target region of high RR and low $\sigma$ . (d) Attacker-by-defense heatmap confirms that GS remains resilient across all implemented adaptive variants.

Figure 5: RQ4 deployment profile: (a) RR–accuracy Pareto plane; (b) criteria satisfied; (c) runtime; (d)

-\log_{10}(W_{1})

fidelity; and (e) normalized deployment radar.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Null hypothesis $H_{0}$	Test	Effect	Decision
RQ1 — Injection Feasibility
$H_{0}^{(1a)}$	$\rho(\text{depth},\text{metric})=0$	Spearman	$\rho=0.954$ , $p<0.001$	R
	(LE $\Delta\mathrm{Acc}$ )	Spearman	$\rho=0.338$ , $p=0.512$	FTR
$H_{0}^{(1b)}$	Equal $\Delta\mathrm{Acc}$ across 6 depths	KW ( $n=8$ )	$\eta^{2}_{KW}$ , $p<0.001$	R
$H_{0}^{(1c)\uparrow}$	$\Delta\mathrm{Acc}(x\geq 19)\leq\Delta\mathrm{Acc}(x\leq 16)$	MW-U	$p=0.005$	R
RQ2 — Baseline Comparison (naive)

Appendix

Table 3: Consolidated pre-specified statistical hypotheses for RQ1–RQ4. All tests are two-tailed at

\alpha=0.05

unless marked (

\uparrow

) one-tailed. ∗ Bonferroni-corrected over

H_{0}^{(4b)}

–

H_{0}^{(4c)}

Symbol / Abbrev.	Definition
Weight-space and attack symbols
$W,\mathcal{W}$	Weight tensor; full model weight space
$x$	LSB overwrite depth ( $1\leq x\leq 23$ for FP32)
$m_{x}$	Bitmask $(1\ll x)-1$ ; selects lowest $x$ bits
$u_{i}$	uint32 bitcast of weight $w_{i}$
$N$	Total number of target parameters

Appendix

Table 4: Consolidated notation and abbreviations used throughout the paper.

Figure 6: Security–utility trade-off across defenses. Panel (a) plots recovery reduction (RR) versus accuracy drop for all defense configurations. Solid markers denote configurations evaluated under the RQ3 adaptive attacker setting; hollow markers indicate the additional swin_cifar10 configuration from RQ2. Hollow markers show the additional swin_cifar10 configuration from RQ2. The horizontal dashed line marks the desired $50\%$ chance-level recovery threshold, while the vertical dashed line indicates the practical $0.25\%$ accuracy budget. GS consistently occupies the upper-left “ideal zone,” achieving near-chance RR with minimal accuracy degradation and low variance. FT achieves competitive RR only at substantially higher utility cost, whereas GN , SWP , and RF fail to provide effective sanitization. The weak global trend ( $R^{2}=0.08$ ) indicates that stronger sanitization does not necessarily require larger accuracy degradation. Panel (b) shows the accuracy drop at each defense’s best operating point, with whiskers indicating the full observed range across configurations. GS achieves near-optimal RR at effectively zero accuracy cost while maintaining substantially tighter variability than competing high-RR defenses.

Transformer

Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.

See all 544 papers

On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference
May 6, 2026Zhengyi Li, Yakai Wang, Kang Yang +6Transformer Inference Model Extraction Attacks
Shanghai Jiao Tong University · State Key Laboratory of Cryptology · Shanghai Qizhi Institute
DegreeSpar: Structured Degree Sparsity for Efficient Secure Transformer Inference
Sep 26, 2026Yifei Cai, Zhuoran Li, Xiaozuo Shen +2Transformer Inference Privacy-Preserving ML
Iowa State University, Ames, IA, USA · University of Arizona, Tucson, AZ, USA
Gradient Immunity: Null-Space Resistance to Malicious Fine-Tuning
Aug 5, 2026Yuxuan Huang, Xingyu Zeng, Tianhang Zheng +1LLM Safety Alignment LLM Security
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Shenzhen University of Advanced Technology +1

May 6, 2026J/K move · Enter open · S save

On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference

Zhengyi Li, Yakai Wang, Kang Yang, Yu Yu+5

Shanghai Jiao Tong University · State Key Laboratory of Cryptology · Shanghai Qizhi Institute

Transformer Inference Model Extraction Attacks LLM Security Privacy-Preserving ML

Open paper AskPDF arXiv

For Transformer models, cryptographically secure inference ensures that the client learns only the final output, while the server learns nothing about the client's input. However, securely computing nonlinear layers remains a major efficiency bottleneck due to the substantial communication rounds and data transmission required. To address this issue, prior works reveal intermediate activations to the client, allowing nonlinear operations to be computed in plaintext. Although this approach significantly improves efficiency, exposing activations enables adversaries to extract model weights. To mitigate this risk, existing works employ a shuffling defense that reveals only randomly permuted activations to the client. In this work, we show that the shuffling defense is not as robust as previously claimed. We propose an attack that aligns differently shuffled activations to a common permutation and subsequently exploits them to extract model weights. Experiments on Pythia-70m and GPT-2 demonstrate that the proposed attack can align shuffled activations with mean squared errors ranging from

10^{-9}

10^{-6}

. With a query cost of approximately $1, the adversary can recover model weights with L1-norm differences ranging from

10^{-4}

10^{-2}

compared to the oracle weights.

Abstract

Transformer

Explore similar work

On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference

DegreeSpar: Structured Degree Sparsity for Efficient Secure Transformer Inference

Gradient Immunity: Null-Space Resistance to Malicious Fine-Tuning