Split learning lets a client train a language model on a server without sending its text. The client runs the first layers itself and sends the server only their output, a vector of numbers for each token. During training, the server sends gradients back. We show that an observer at the split can rebuild most of the client's text from this traffic, and we measure how much the gradients help. On GPT-2, an attacker who holds only the publicly released weights of the client's layers recovers 94.20% of tokens from the activations alone and 97.38% when it also sees the gradients, 3.17 percentage points more 95% interval [2.72, 3.64]. Counted by document, the difference is much larger. The attacker rebuilds 13.71% of 32-token documents exactly without the gradients and 37.77% with them, because a document only counts when every token is right. How we count also changes how good a defence looks. Secret mixup, which blends each outgoing vector with a decoy, stops the attacker from rebuilding almost any document exactly, yet the attacker still recovers 83-91% of tokens. In a second experiment on GPT-2 and Qwen3-0.6B, where the server trains only a run of consecutive layers, the layer at which the run starts changes both model quality and leakage, even when the run's length is fixed. We recommend reporting leakage both per token and per document, and treating what a split model sends as being as sensitive as the text itself.
Figures & tables
Figure 1: The gradients barely change how many tokens the attacker recovers, but nearly triple how many documents it rebuilds exactly. (a) What leaves the client. The client runs GPT-2’s first six layers and sends their activations to the server, which runs the other six. During training only, gradients come back. The attacker sits at the split, only watches this traffic, holds the publicly released GPT-2 weights and never sees the text. (b) What the attacker reads back from one document in one run: the true text, the reconstruction from activations only, and the reconstruction from activations and gradients, with wrong tokens highlighted. We chose this document as the one whose per-document accuracies are closest to the reported averages. The highlighted first token is a quotation mark predicted with a leading space, which looks the same in print. (c) Across all 256 evaluation documents and all runs: token recovery and whole-document recovery for both attackers, with 95% intervals, and the 2.75-fold rise in documents rebuilt exactly.
Figure 2: Split learning and what the attacker sees. The client holds the private text and runs the first layers of the model, while the server runs the rest. In both training and inference, the client sends one activation per token across the split. In training only, the server sends back the gradient of the next-token loss with respect to those activations. The attacker watches this traffic without changing it. It holds a copy of the client’s layers, either the publicly released weights or the client’s trained copy, and a small set of public documents. It never gets the private text, the server’s layers or the labels of the next-token loss. Inset: in the partial-offloading experiment, the server trains only a run of s consecutive layers starting at layer b (example: b=2 , s=4 ).
Signal
When it crosses
How the attack uses it
What it reveals
Activations, one vector per token
Training and inference
The ridge fit turns each activation into a short list of likely tokens, and a beam search keeps the guesses whose recomputed activations match (Figure 3 ). SipIt also reads them.
Token, whole-document and span recovery (Section 4 )
Gradients of the next-token loss
Training only
Each gradient carries a signal about the token that follows it, so it suggests half of the candidate tokens at each position.
What the gradients add to the same three scores
Activations of the final trained model
Training
The ridge fit decodes each activation to the nearest token (adapted model inversion).
Tokens recovered beyond the majority control (Section 5 )
Size of the returned gradient, per document
Training
The summed gradient size scores whether a document contains a given token.
Properties of the text, as ROC AUC (Section 5 )
Controls: another document’s gradient; random tokens in place of the gradient’s suggestions
Training
Same attack, with the gradient’s information removed
Whether the gain comes from the gradient
Table 1: The signals an observer at the split reads, and how each becomes a measured leak. The first two rows belong to the main experiment and the next two to the partial-offloading experiment. The controls replace the gradient’s information and show that the gain comes from what the gradient carries.
Figure 3: How the attacker rebuilds a document. (1) From 32 public documents, the attacker learns a map from activations back to token embeddings. (2) At each position, it lists the k most likely tokens. (3) The joint attacker replaces half of that list with tokens suggested by the gradient. The first position has no gradient help and keeps the list from the activations. (4) A beam search of width two builds the text from left to right, keeping the guesses whose activations, recomputed through the attacker’s copy of the client’s layers, best match the observed ones.
Attack
L=3
L=6
Backward-only heuristic
24.0
50.0
SipIt (128 steps)
98.2
96.7
Activation-only ( k=32 )
97.6
94.5
Joint, shuffled gradient ( k=32 )
97.5
94.2
Joint, true gradient ( k=32 )
98.5
97.8
Table 2: Token recovery (%) of each attack at depths three and six, averaged over the public and the trained copy of the client’s layers. The joint attacker has the highest token recovery at both depths. Each entry pools 10,240 document-level evaluations built from 256 distinct evaluation documents (10 victim seeds × 2 attack seeds × 2 copies of the client’s layers). Values are rounded to one decimal. Table 7 gives two.
Figure 4: Gradients add little to token recovery but nearly triple whole-document recovery. The panels show token recovery (left) and whole-document recovery (right) for the activation-only attacker and for the joint attacker, which also sees the gradients. GPT-2 is split after its sixth layer, the attacker holds the public copy of the client’s layers, and the candidate budget is 32. Token recovery is the share of tokens reconstructed correctly. Whole-document recovery is the share of 32-token documents reconstructed without error. Bars are 95% crossed-bootstrap intervals over documents and trained models. The left panel prints the paired difference in token recovery with its interval, and the right panel prints how many times higher whole-document recovery is with the gradients.
Token recovery
Whole-document recovery
Depth
Copy
Activation-only
Joint
Activation-only
Joint
L=3
public
97.39
98.29
43.13
55.51
L=3
trained
97.81
98.72
51.41
66.13
L=6
public
94.20
97.38
13.71
37.77
L=6
trained
94.81
98.27
20.23
58.55
Table 3: Token and whole-document recovery (%) for the activation-only and joint attackers at k=32 , by split depth and by the attacker’s copy of the client’s layers. The gradients’ gain is small per token and large per document. Each entry pools 5,120 document-level evaluations over 256 distinct documents. The setting of the main test is in bold.
Figure 5: Token recovery and whole-document recovery for each gradient condition at depth L=6 , with the public copy of the client’s layers and k=32 (5,120 document-level evaluations per condition over 256 distinct documents). Noise and rounding of the gradient leave most of its gain in place. The dashed line marks the activation-only attacker. Every condition stays above it except the shuffled-gradient and random-candidate controls, which remove the gradient’s information and fall slightly below it.
Class
Documents
Spans
No defence
Secret mixup
Name
249
716
67.36–97.44
47.85–66.15
Number
142
297
97.64–100.00
93.94–96.50
Identifier
2
2
100.00–100.00
90.00–100.00
Table 4: Exact span recovery (%) by annotated class on the fixed evaluation set, without a defence and under secret mixup. Secret mixup protects names far more than numbers. Ranges are over the same 24 conditions: six attacks × two depths × two copies of the client’s layers. Each rate pools exact span recoveries within an evaluation and then averages over its ten victim seeds and two attack seeds. Documents and spans are reused across conditions.
Figure 6: Leakage under each defence, plotted against its cost in model quality. Leakage is token recovery (left) and whole-document recovery (right), pooled over the six attacks run against each defence at each depth. Cost is the rise in held-out perplexity over the final model trained without a defence. Secret mixup almost removes whole-document recovery at a 27 – 32% perplexity cost, while 83 – 91% of tokens stay recoverable. Replica-median changes token recovery little and lowers whole-document recovery by up to 7.7 points at almost no cost in model quality.
Model
Defence
Outcome
Partial R2 [98.75% CI]
p
Passes
GPT-2
8-bit
Utility
1.00 [1.00, 1.00]
0.0001
Yes
GPT-2
8-bit
Leakage index
0.99 [0.99, 1.00]
0.0001
Yes
GPT-2
4-bit noisy
Utility
0.90 [0.85, 0.97]
0.0001
Yes
GPT-2
4-bit noisy
Leakage index
0.21 [0.13, 0.56]
0.019
No
Qwen3
8-bit
Utility
1.00 [1.00, 1.00]
0.0001
Yes
Qwen3
8-bit
Leakage index
0.91 [0.85, 0.98]
0.0001
Yes
Table 5: Variance tests for where the server-trained layers start, in GPT-2 and Qwen3. The start explains variation beyond length in seven of eight tests. The one failure is GPT-2 leakage under the 4-bit noisy defence. Each row gives the partial R2 of the start and its interaction with length, a 98.75% seed-bootstrap interval, and the permutation p value from shuffling the start within each seed and length. A test passes if partial R2≥0.10 and p<0.0125 . Values are rounded to two decimals. Appendix C gives four decimals.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Label in the released files
Name in this paper
protocol v3 , v3 campaign
main experiment
corrected P6 follow-up
partial-offloading experiment (the second experiment)
original P6 pilot
the excluded pilot of the partial-offloading experiment
Qwen3 FP64 amendment
Qwen3 replication
D0
8-bit defence
D1
4-bit noisy defence
Appendix
Table 6: Internal labels in the released files and the names used in this paper.
Figure 7: Token recovery for all 40 attack configurations of the main experiment, at depths L∈{3,6} , with the public and trained copies of the client’s layers. The 28 configurations without a defence are above the divider, the 12 defended ones below. Each entry aggregates 5,120 document-level evaluations over 256 distinct held-out documents. Depth three generally leaks more than depth six, and the trained copy leaks more than the public copy throughout. The backward-only heuristic reverses the depth ordering ( ≈24% at depth three against 50% at depth six). In the defended rows, token recovery drops under secret mixup and stays close to the undefended rows under replica-median, as described in Section 4.5 .
Attack
L=3
L=6
Backward-only heuristic
24.04
50.01
SipIt (128 steps)
98.17
96.69
Activation-only ( k=32 )
97.60
94.51
Joint, shuffled gradient ( k=32 )
97.46
94.21
Joint, true gradient ( k=32 )
98.50
97.82
Appendix
Table 7: Token recovery (%) from Table 2 , given to two decimals and averaged over the public and the trained copy of the client’s layers. Each entry aggregates 10,240 document-level evaluations.
Defence
Outcome
Partial R2
Interval
p
Passes rule
D0
Utility
0.9995
[0.9993, 0.9998]
0.00010
Yes
D0
Leakage index
0.9898
[0.9856, 0.9967]
0.00010
Yes
D1
Utility
0.8977
[0.8493, 0.9673]
0.00010
Yes
D1
Leakage index
0.2076
[0.1283, 0.5571]
0.01930
No
Appendix
Table 8: GPT-2 variance tests to four decimals, showing leftover variation explained by the start and its interaction with length, with 98.75% seed-bootstrap intervals. The start passes three of four tests. A test passes when it meets both the partial R2 threshold and the corrected permutation threshold.
Defence
Outcome
Partial R2
Interval
p
D0
Utility
0.9982
[0.9972, 0.9994]
0.00010
D0
Leakage index
0.9089
[0.8510, 0.9759]
0.00010
D1
Utility
0.9230
[0.8833, 0.9815]
0.00010
D1
Leakage index
0.9040
[0.8821, 0.9624]
0.00010
Appendix
Table 9: Qwen3 variance tests to four decimals, reporting partial R2 for the start and its interaction with length, with 98.75% seed-bootstrap intervals and permutation p values from shuffling the start within each seed and length. The start passes all four tests.
Defence
Length s
Start b
ΔU
VMI excess
I
D0
2
0
0.5342
0.9199
0.9199
D0
2
1
0.3854
0.9227
0.9227
D0
2
2
1.3715
0.6917
0.6917
D0
2
3
4.2361
0.0170
0.1468
D0
4
0
0.3173
0.9158
0.9158
D0
4
1
0.2342
0.9227
0.9227
Appendix
Table 10: GPT-2 means across eight seeds for every placement. Utility is excess held-out loss in nats. VMI excess is recovery minus the majority control (fraction). I is the normalised leakage index.
Seed
Gain
Seed
Gain
11
0.017826762050
55
0.020145211369
22
0.018502995372
66
0.018460044637
33
0.015580737963
77
0.018363717943
44
0.014817791060
88
0.009917549789
Appendix
Table 11: Recovered P2 gains in nats, kept for the historical arithmetic audit.
Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.
Split language models keep input text on a client but send intermediate representations to a server. Attacks can reconstruct text from these messages. Comparing privacy evaluations is difficult when attackers see different messages, know different model weights or use different datasets and recovery measures. We present LLM-Attack Suite, a framework that records these choices and lets researchers check, compare and extend an evaluation. Compatible attacks reuse recorded messages, and their predictions are saved for independent scoring. The scoring rule requires a linked undefended reference before a defence can pass. Reports also identify attacks that remain untested. In a controlled GPT-2 experiment, adding gradients returned during training increased token recovery from 94.20% to 97.38% and exact-document recovery from 13.71% to 37.77%. The 95% interval for the token gain lay above the two-percentage-point threshold fixed before testing. This experiment is shared with a companion study; here it tests how the evaluation responds to a change in attacker access. Further studies measure how corpus choice and compression defences affect recovery, compare an adapted published attack with its reference implementation, and test the measurement code. The framework reports implementation checks separately from measured attack effectiveness. Each privacy claim should state the attacker, dataset and scoring rule, report its controls and identify untested attacks.
We present a systems-security case study of a two-node split-LLM training system whose privacy evaluation passed while leaving an observable channel untested. The Trusted Local Node (TLN) sends protected activations to the Untrusted Cloud Node (UCN), the UCN returns its output, and TLN, holding the private loss, returns the output gradient. The frame the UCN receives mixes real rows with decoys, and the loss ignores the decoys. Their gradients are exactly zero, so the pattern of zeros reveals which rows were real. We measure it with a protocol fixed in advance: a leak injected at known strength to prove the instrument can see one, a shuffled-label control to prove it does not report absent leaks, and a threshold set before the runs. Across nine seeds, the zeros identified the real rows on every frame, 4,096 of 4,096 per run. An attack on the frame contents recovered about one extra token per hundred over a constant-guess baseline (+0.65 to +1.50 percentage points); the shuffled controls recovered nothing. A second set of runs repeated this on a configuration that keeps model quality within budget, so the finding is not confined to a setting nobody would deploy. On both datasets, every such run passed the forward-channel privacy check and the quality check, yet failed that same check once the returned gradient was included. Clipping and noising each row of the gradient closed the leak for about 0.01 nats of held-out cross-entropy. The system is not thereby safe: five classes of attack, including those accumulating observations across training steps, were never measured.
Gradient matching attacks (GMAs) in LLM split learning (SL) rely on a critical yet underexplored assumption: the gradient exposed at the split interface is a faithful derivative of the client's full-label training objective. This gradient-objective consistency allows a curious server to recover private labels by searching for a sequence whose induced gradient explains the observation. We propose Gradient Mirage, a defense that breaks this consistency without discarding the optimization utility of the backward signal. Our key idea is to induce the adversary to solve a misspecified inverse problem, in which no plausible label sequence in the sequence space can explain the observed gradients. Concretely, Gradient Mirage achieves this by inducing inconsistency across three dimensions: objective, direction, and scale. Selective Autoregressive Supervision derives the exposed gradient from a masked surrogate loss rather than the full-label objective assumed by the attacker; Scale Blinding then applies randomized multiplicative rescaling, obscuring the gradient's natural magnitude; and Directional Privatization further randomizes the gradient direction while preserving its magnitude through the von Mises-Fisher (vMF) mechanism under a directional metric differential privacy guarantee. Crucially, utility is preserved: the Top segment still learns from all target tokens via Dual-Track Backpropagation, the exposed gradient remains informative since each supervised token retains its complete autoregressive context, and Bottom-Gradient Recovery restores the effective gradient for Bottom-segment optimization. Extensive experiments show that Gradient Mirage provides substantially stronger protection than existing defenses under comparable fine-tuning performance, achieving a better privacy-utility trade-off.
Shiyu Miao, Yunlong Mao, Zirui Huang +5
State Key Laboratory for Novel Software Technology, Nanjing University · College of Computer Science and Software Engineering, Hohai University · School of Mathematics, Nanjing University +1