SOMP: Scalable Gradient Inversion for Large Language Models via Subspace-Guided Orthogonal Matching Pursuit

Yibo Li; Qiongxiu Li

cs.LGMar 17, 2026

SOMP: Scalable Gradient Inversion for Large Language Models via Subspace-Guided Orthogonal Matching Pursuit

Authors: Yibo Li, Qiongxiu Li

Organizations: Politecnico di Milano / Italy · Aalborg University

Abstract

Gradient inversion attacks reveal that private training text can be reconstructed from shared gradients, posing a privacy risk to large language models (LLMs). While prior methods perform well in small-batch settings, scaling to larger batch sizes and longer sequences remains challenging due to severe signal mixing, high computational cost, and degraded fidelity. We present SOMP (Subspace-Guided Orthogonal Matching Pursuit), a scalable gradient inversion framework that casts text recovery from aggregated gradients as a sparse signal recovery problem. Our key insight is that aggregated transformer gradients retain exploitable head-wise geometric structure together with sample-level sparsity. SOMP leverages these properties to progressively narrow the search space and disentangle mixed signals without exhaustive search. Experiments across multiple LLM families, model scales, and five languages show that SOMP consistently outperforms prior methods in the aggregated-gradient regime.For long sequences at batch size B=16, SOMP achieves substantially higher reconstruction fidelity than strong baselines, while remaining computationally competitive. Even under extreme aggregation (up to B=128), SOMP still recovers meaningful text, suggesting that privacy leakage can persist in regimes where prior attacks become much less effective.

Figures & tables

Figure 1: Overview of SOMP. It consists of Stage I head-structured token pooling, Stage II geometry-driven diverse decoding, and Stage III gradient-space sparse reconstruction.

$B$	Reference Inputs	DAGER Reconstruction	SOMP Reconstruction
	If JognThaw had never … is absolute must include.	If JognThaw had never … is absolute must include.	If JognThaw had never … is absolute must include.
	This sequel is quite … lack of choreography.	This sequel is quite … movie is lack of money .	This sequel is quite … lack of choreography. .
4	I saw this movie when … in another movie.	I saw this movie when … in another movie.	I saw this movie when … in another movie.
	I won’t ask you … give her any money.	I saw this movie when … in another movie .	I won’t ask you … give her any money.
	Jill Dunne (played by Mitzi Kapture), is an … <br/><br/>3 stars.	Jill Dunne (played by Mitzi Kapture),is an … nothing happens!	Jill Dunne (played by Mitzi Kapture), is an … Jill. <br/> 3 stars.
	$...$	$...$	$...$

Table 1: Reconstruction examples under aggregated gradients at

B=2,4,8

. Highlighted spans mark incorrect tokens. As

B

increases, DAGER degrades faster, while SOMP remains more faithful to the reference inputs.

Figure 2: Reconstruction performance (ROUGE-L) under aggregated gradients at batch sizes B = 1, 2, 4, 8, and 16 on IMDB, CoLA, and SST-2 using GPT-J-6B (top) and GPT-2 (bottom). SOMP shows the largest gains on IMDB, whose longer inputs make multi-sample inversion harder than on shorter datasets such as CoLA and SST-2.

Dataset	Method	$B=4$	$B=8$	$B=16$
	LAMP	$1.2\pm 0.8$	$0.3\pm 0.3$	$0.1\pm 0.1$
	GRAB	$8.6\pm 2.4$	$2.0\pm 0.6$	$0.9\pm 0.9$
IMDB	DAGER	$58.4\pm 2.4$	$39.2\pm 0.8$	$31.6\pm 1.5$
	SOMP	89.3 $\pm$ 1.9	73.5 $\pm$ 5.1	76.4 $\pm$ 2.1
	LAMP	$10.6\pm 1.8$	$6.3\pm 0.7$	$1.4\pm 0.3$
	GRAB	$42.6\pm 2.4$	$35.1\pm 2.1$	$24.3\pm 2.6$

Table 2: Reconstruction performance (ROUGE-L) under aggregated gradients at

B=4,8,16

on GPT-J-6B.

GPT-2 (runtime measured on RTX 5060)
Dataset	Method	B=1	B=2	B=4	B=8	B=16
	DAGER	3	6	10	19	36
CoLA	SOMP(Bw=4)	7	10	13	21	26
	SOMP(Bw=8)	7	10	14	24	31
	SOMP(Bw=16)	8	13	21	33	41
	DAGER	3	4	9	18	31

Table 3: Average per-batch runtime in seconds. Bw denotes beam width.

Figure 4: Ablation at batch sizes ( $B=2,4,8$ ). Variants remove LM prior (SOMP1), OMP (SOMP2), sparsity scoring (SOMP3), cross-head consistency (SOMP4), and subspace-distance scoring (SOMP5).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Hyperparameter	Value
Stage I	$\lambda_{1}$	$0.8$
	$\lambda_{2}$	$0.5$
	$\lambda_{3}$	$0.5$
	$\tau^{(h)}_{m}$	$0.5\times$ head-wise median
Stage II	$\beta_{\text{LM}}$	$0.33$
	$\lambda_{\text{div}}$	$0.15$

Appendix

Table 4: Batch-size-independent hyperparameters used in SOMP.

Hyperparameter	$B{=}1$	$B{=}4$	$B{=}8$	$B{=}16{+}$
Token pool size $k$	960	1600	2400	3200
Informative heads $\|H_{\text{act}}\|$	$H/4$	$H/4$	$H/3$	$H/3$
Sparsity heads $\|H_{\text{top}}\|$	$H/6$	$H/6$	$H/4$	$H/4$
Beam width $W$	2	4	6	12
Number of beam groups $G$	1	4	8	16
OMP iterations (max)	$B$	$B$	$B$	$B$

Appendix

Table 5: Batch-size-dependent hyperparameters as a function of aggregation level

B

Parameter	Value	ROUGE-L
Stage I: Token Pooling
Token pool size $k$	800	59.7
	1200	71.5
	1600	73.4
	2000	74.5
	2400	73.3

Appendix

Table 6: Hyperparameter sensitivity analysis of SOMP. Each parameter is varied while keeping the others fixed.

Dataset	Method	R-1	R-2	R-1	R-2	R-1	R-2	R-1	R-2
		$B=1$		$B=2$		$B=4$		$B=8$
GPT-J
	LAMP	31.9 $\pm$ 2.4	16.5 $\pm$ 3.1	12.7 $\pm$ 3.7	7.6 $\pm$ 2.0	5.1 $\pm$ 0.7	1.4 $\pm$ 0.3	1.3 $\pm$ 0.4	0.3 $\pm$ 0.3
IMDB	GRAB	46.1 $\pm$ 4.4	33.4 $\pm$ 1.4	38.1 $\pm$ 3.7	21.7 $\pm$ 0.9	33.1 $\pm$ 2.1	9.2 $\pm$ 1.3	19.7 $\pm$ 1.7	2.0 $\pm$ 0.6
	DAGER	99.2 $\pm$ 0.8	94.3 $\pm$ 3.2	93.2 $\pm$ 3.4	72.0 $\pm$ 2.8	88.5 $\pm$ 3.1	58.4 $\pm$ 2.4	69.3 $\pm$ 1.3	39.2 $\pm$ 0.8
	SOMP	99.5 $\pm$ 0.5	96.5 $\pm$ 2.1	98.3 $\pm$ 1.7	93.6 $\pm$ 3.8	94.4 $\pm$ 2.7	89.3 $\pm$ 1.9	88.4 $\pm$ 4.3	73.5 $\pm$ 5.1

Appendix

Table 7: Full reconstruction results for the main experiments, reporting ROUGE-1 and ROUGE-2 across datasets and batch sizes for GPT-J and GPT-2.

Language	B=1	B=4	B=8	B=16
English	99.7 $\pm$ 0.3	94.3 $\pm$ 1.4	88.2 $\pm$ 1.7	83.5 $\pm$ 2.1
German	99.6 $\pm$ 0.4	94.6 $\pm$ 0.8	89.7 $\pm$ 2.1	82.3 $\pm$ 2.6
Chinese	99.8 $\pm$ 0.2	93.1 $\pm$ 1.2	88.1 $\pm$ 1.4	84.4 $\pm$ 3.4
Italian	99.7 $\pm$ 0.3	90.2 $\pm$ 1.5	83.3 $\pm$ 1.8	78.1 $\pm$ 1.8
French	95.7 $\pm$ 2.1	86.3 $\pm$ 3.3	80.5 $\pm$ 3.6	72.8 $\pm$ 3.7

Appendix

Table 8: Multilingual reconstruction performance of SOMP on Qwen3 across batch sizes on stanfordNLP/imdb .

$E$	$\eta$	$B_{\text{mini}}$	R-1	R-2
2	$10^{-4}$	4	99.1 $\pm$ 0.9	98.2 $\pm$ 1.1
5	$10^{-4}$	4	97.5 $\pm$ 1.3	95.8 $\pm$ 1.3
10	$10^{-4}$	4	95.3 $\pm$ 1.8	93.9 $\pm$ 1.6
20	$10^{-4}$	4	95.7 $\pm$ 0.9	94.4 $\pm$ 1.4
10	$10^{-5}$	4	99.8 $\pm$ 0.2	99.1 $\pm$ 0.9
10	$5\times 10^{-5}$	4	98.7 $\pm$ 0.9	97.9 $\pm$ 1.2

Appendix

Table 9: Reconstruction performance of SOMP under FedAvg on GPT-2 (Rotten Tomatoes).

E

denotes the number of local epochs,

\eta

the learning rate, and

B_{\text{mini}}

the mini-batch size.

Noise Level $\bm{\sigma}$	ROUGE-1	ROUGE-2
$10^{-5}$	78.1 $\pm$ 3.3	72.8 $\pm$ 4.1
$5\times 10^{-5}$	55.9 $\pm$ 4.2	34.5 $\pm$ 3.9
$10^{-4}$	26.7 $\pm$ 4.0	16.6 $\pm$ 3.6
$5\times 10^{-4}$	9.6 $\pm$ 3.5	1.9 $\pm$ 1.9

Appendix

Table 10: Exploratory reconstruction performance of SOMP under additive Gaussian noise on GPT-2 (Rotten Tomatoes,

B=1

). Higher noise levels correspond to stronger perturbation of the aggregated gradient.

Reference Inputs	LAMP	GRAB	DAGER	SOMP
I received this movie as a gift, I knew from … movies, don’t watch this movie.	I camcorder movie" DVD sucks filmmaker knew wane bee … movies, <br />Sgt this movie.	town watched it pro- duce movie, Ben Draper … ries HORROR revenge woman absolute.	I received this movie as a gift, I knew from … movies, don’t watch this movie.	I received this movie as a gift, I knew from … movies, don’t watch this movie.
I very much looked forward to this movie. … a good love story.	I very Michael place Willie generally to they … Chris capt. <br /> ’s con- text ’re romantic.	Its Jr.’s editing to much is be Too. better care … love Willie and Missy;series.	I very much looked forward to Michael … a good love Michael.	I very much looked forward to this movie. … a good love story.
Protocol is an implausible movie whose only … way to see this film.	Protocol Love’s implausi- ble Missy very much … because see Plus.	Jr.’s is But, implausible romantic always only … always enjoy role film.	Protocol is an implausible movie whose only … way to see this movie.	Protocol is an implausible movie whose only … way to see this film.
I was excited to view a Cataluña´s film in … to go to see this film.	I was excited Berlin´s a Cataluña´s film in … , was horrible. competition.	Cataluña, too horr to view a Cataluña´s see film … to Molina this film.	I was excited to view a Cataluña´s film in … to go to see this film.	I was excited to view a Cataluña´s film in … to go ´s to see Cataluña´s film.

Appendix

Table 11: Qualitative reconstruction examples under aggregated gradients with batch size

B=2

. Highlighted spans indicate incorrect tokens caused by cross-sample mixing.

Reference Inputs	LAMP	GRAB	DAGER	SOMP
A model named Laura is working in South America when … jungle. It would probably be more fun.<br /><br />3/10	A Laura named ive fake Laura in Amer- ica America … vision. far). Franco happening. probab be film fun.<br /><br />3/10	primitives Hunter named Devil.<br /> films working drags ity in (25 …laugh. hilariously killed real fun more fun.<br /><br />. <br /><br />	A model named Laura is working in hotel when Devil Hunter is film film her hotel … It would probably be more fun.<br /><br />3/10	A model named Laura is work in South America when …jungle. It would probably be more fun.3/10 .3/10.3/10.
Devil Hunter gained notoriety for the fact that … no reason to bother with this one.	Devil Hunter film notoriety for the ’Video Nasty’ that’s films the DPP … with this one devil blood horror.	DPP Hunter gained notoriety for films kid- napped that title). the the … She Kills in Ec- stasy Nasty list but it re- ally needn’t have been	Devil Hunter gained notoriety for the fact that … basic Venus film films Devil this one.	Devil Hunter gained notoriety for the fact that … no reason to bother with this one one
This film seemed way too long even … probably didn’t see an uncut version.	This film version too uncut probably way … scene blood camera long didn’t see .	75 zombie too long seemed even … dub- bing didn’t rocky dubbing way cliff version.	This film seemed way too long even … film films’t zombie / flick film films…	This film seemed way too long even … probably didn’t see an uncut version.
I cannot stay indifferent to Lars van Trier’s films. I … to the viewers to decide this alone.	I films alone Trier’s viewers … cannot decide horror camera devil Lars stay.	Dancer cannot to consider different van van Lars’s masterpiece … pre- ambiguous <br /><br /> decide this Europa Europa .	I cannot stay indifferent to Lars van Trier’s films. I … differ- ent filmmaker van Lars van demonstrative this alone.’s..	I cannot stay different to Lars van Trier’s films. I … to the viewers to decide this alone …
Ah, Channel 5 of local Mexican t.v. Everyday, at 2:00 … film-making.<br/><br />You have been warned.	Ah, warned Mexican 2:00 Channel … film- making blood devil street local been Every- day.	Ah, Mexican Ah, Ah, Mexican t.v. Ah,… Ev- eryday at 2:00 been film-making. You have warned.	Ah, Channel 5 of local Mexican Ah, Mexican, at 2:00… film-film film. <br/><br/> later films, cinematography warned.	Ah, Channel 5 of local Mexican Ah, Everyday, at 2:00 … film- making.Ah,Ah, You have been warned.

Appendix

Table 12: Qualitative reconstruction examples under aggregated gradients with batch size

B=4

. Highlighted spans indicate incorrect tokens caused by cross-sample mixing.

Reference Inputs	LAMP	GRAB	DAGER	SOMP
Sloppily directed, witless comedy that supposedly … be begging for some peace and quiet. (*1/2)	peace quiet comedy begging sloppily * … hotel blood ’t camera witless scene .	quiet comedy peace … witless directed ’s begging some film street .	Sloppily directed, witless comedy that supposedly … doomed James James 50s Royale Royale Royale Royale..(2(.	Sloppily directed, witless comedy that comedy spoofs … be begging some Royale and Royale..(*1/2)…
I was disgusted by this movie. No it wasn’t … movie, but find out the true story of Artemisia.	movie disgusted Artemisia true … story Beaver devil blood find out .	Artemisia movie disgusted … Beaver Beaver find scene film .	I was disgusted by this movie. No it wasn’t … <br /><br /> movie, but movie movie the Artemisia Gentileschi.	I was disgusted by this movie. No it it ’t … movie, but find out the true Artemisia of Artemi- sia….
I didn’t think the French could make a bad movie, but … pornography. So the French can fail, after all!	French pornography bad movie … after fail camera street horror French .	pornography think French … bad movie fail after film camera .	I was disgusted by this movie. No it wasn’t … <br /><br /> movie, but movie movie the Artemisia Gentileschi.	I didn’t think the French could make a bad movie, but … pornography. So the French can fail, after all! , after all!, after all!!!!!
I read somewhere that when Kay Francis refused to … and you’ll be doing yourself a favor.	favor somewhere yourself doing … Kay Francis camera jungle blood .	favor Francis somewhere … doing yourself Kay film scene .	I read somewhere that when Kay Francis refused to … and Francis’ll movie <br />Also, a favor. And her hair!!!	I read somewhere that when Kay Francis refused to … and you’ll be yourself a favor fa- vor a a a a…
A stale "misfits-in-the-army" saga, which … Me", has only two or three brief scenes. What a waste! (*1/2)	saga brief scenes … waste army devil hotel camera three .	brief films … scenes to waste army film .	A stale "misfits-in-the-army" saga, which … Me", saga only two misfits misfits brief scenes. What a waste! (*1/2)	A stale "misfits-in-the-army" saga, which … Me", has only two three scenes. What a waste! (1/2)! (1/2))
When I was at the movie store the other day, I passed up … and Blonder is a huge blonde BOMBshell.<br /><br />1/10	When I movie store … Michelle Romy Anderson blood street scene day .	Blonder store … huge blonde stupid movie Michelle film .	When I was at the movie store the other day, I passed up … and Blonder is a huge BOMB. <br /><br />1/10 <br /><br />1/10	When I was at the movie store the other day, I passed up … and Blonder is a blonde Blonder .<br /><br />1/10 ….. D

Appendix

Table 13: Qualitative reconstruction examples under aggregated gradients with batch size

B=8

. Highlighted spans indicate incorrect tokens caused by cross-sample mixing.

Gradient Inversion Attacks

Gradient inversion attacks infer private training examples or labels from gradients shared during distributed model training.

See all 20 papers

FedSpy-LLM: Towards Scalable and Generalizable Data Reconstruction Attacks from Gradients on LLMs
Apr 7, 2026Syed Irfan Ali Meerza, Feiyi Wang, Jian LiuGradient Inversion Attacks Privacy Leakage in Language Models
University of Tennessee, Knoxville, TN, USA · Oak Ridge National Laboratory, Oak Ridge, USA · University of Georgia, Athens, GA, USA
Gradient Mirage: Trainable yet Label-Unidentifiable Gradients in Large Language Model Split Learning
Aug 19, 2026Shiyu Miao, Yunlong Mao, Zirui Huang +5Privacy-Preserving Language Models Gradient Inversion Attacks
State Key Laboratory for Novel Software Technology, Nanjing University · College of Computer Science and Software Engineering, Hohai University · School of Mathematics, Nanjing University +1
GPT, But Backwards: Exactly Inverting Language Model Outputs
Jul 2, 2025Adrians Skapars, Edoardo Manino, Youcheng Sun +1Privacy Leakage in Language Models LLM Security
Department of Computer Science, University of Manchester, Manchester, United Kingdom · Mohamed bin Zayed University of Artificial Intelligence, United Arab Emirates · Federal University of Amazonas, Manaus, Brazil

Apr 7, 2026J/K move · Enter open · S save

FedSpy-LLM: Towards Scalable and Generalizable Data Reconstruction Attacks from Gradients on LLMs

Syed Irfan Ali Meerza, Feiyi Wang, Jian Liu

University of Tennessee, Knoxville, TN, USA · Oak Ridge National Laboratory, Oak Ridge, USA · University of Georgia, Athens, GA, USA

Gradient Inversion Attacks Privacy Leakage in Language Models Federated Learning

Open paper AskPDF arXiv

Given the growing reliance on private data in training Large Language Models (LLMs), Federated Learning (FL) combined with Parameter-Efficient Fine-Tuning (PEFT) has garnered significant attention for enhancing privacy and efficiency. Despite FL's privacy benefits, prior studies have shown that private data can still be extracted from shared gradients. However, these studies, mainly on full-parameter model training, are limited to reconstructing small batches, short input sequences, and specific model architectures, such as encoder-based or decoder-based models. The reconstruction quality becomes even worse when dealing with gradients from PEFT methods. To fully understand the practical attack surface of federated LLMs, this paper proposes FedSpy-LLM, a scalable and generalizable data reconstruction attack designed to reconstruct training data with larger batch sizes and longer sequences while generalizing across diverse model architectures, even when PEFT methods are deployed for training. At the core of FedSpy-LLM is a novel gradient decomposition strategy that exploits the rank deficiency and subspace structure of gradients, enabling efficient token extraction while preserving key signal components at scale. This approach further mitigates the reconstruction challenges introduced by PEFT's substantial null space, ensuring robustness across encoder-based, decoder-based, and encoder-decoder model architectures. Additionally, by iteratively aligning each token's partial-sequence gradient with the full-sequence gradient, FedSpy-LLM ensures accurate token ordering in reconstructed sequences.

Abstract

Gradient Inversion Attacks

Explore similar work

FedSpy-LLM: Towards Scalable and Generalizable Data Reconstruction Attacks from Gradients on LLMs

Gradient Mirage: Trainable yet Label-Unidentifiable Gradients in Large Language Model Split Learning

GPT, But Backwards: Exactly Inverting Language Model Outputs