Sequential Functional Structured Tucker Compression for Large Language Model Attentions

Jiangfeng Chen; Xinyu Wang; Tianshuo Yan; Hanwei Wu; Xiao-Wen Chang; Yang Zhang; Lei Ding

cs.CLSep 30, 2026

Sequential Functional Structured Tucker Compression for Large Language Model Attentions

Authors: Jiangfeng Chen, Xinyu Wang, Tianshuo Yan, Hanwei Wu, Xiao-Wen Chang, Yang Zhang, Lei Ding

Organizations: University of Manitoba · McGill University · Simpleway · The University of Hong Kong · McMaster University

Abstract

Post-training compression of LLM attention is often formulated as independent matrix approximation, ignoring both the shared structure among attention projections and the representation shift introduced by earlier compression. We propose FTC, a sequential structured compression framework that adapts the approximation to the current compressed model while jointly exploiting the native Q/K/V head structure under a fixed storage budget. The output projection is handled separately to account for the changed post-attention representation. FTC requires neither fine-tuning nor gradient-based recovery. Across seven decoder-only LLMs from 6B to 32B parameters, FTC achieves the lowest WikiText-2 perplexity among the compared methods at every tested keep ratio on five modern GQA models, with the largest gains under aggressive compression. The improvements transfer to downstream tasks and remain substantial at the 32B scale.

Figures & tables

Figure 1: Overview of FTC. At each layer, FTC builds an activation-induced metric from the current student representations, jointly compresses the Q/K/V projections with a shared Tucker representation, and allocates the layer budget between Tucker ranks and sparse-core capacity. The core is then refined progressively, while $W_{O}$ is calibrated and compressed separately using the updated attention representation. The resulting student representation is passed to the next layer, forming a sequential compression process.

	$\rho=0.6$							$\rho=0.2$
Model	Ftc	Seq-SVD	enh-L	TLLM	MPIFA	SVD	Basis	Ftc	Seq-SVD	enh-L	TLLM	MPIFA	SVD	Basis
GPT-J-6B (8.86)	9.30	9.47	9.93	9.68	9.46	10.26	10.16	13.60	14.84	17.47	19.22	17.24	25.04	23.02
Llama-2-13B (4.89)	5.11	5.27	5.13	5.52	5.26	5.68	5.57	7.52	9.00	9.79	13.01	10.21	14.20	12.77
Mistral-7B (5.32)	5.44	5.69	5.58	6.07	5.63	5.92	5.75	7.33	12.10	146.31	18.41	15.92	25.40	16.18
Llama-3-8B (6.14)	6.64	7.03	6.82	8.70	7.26	8.02	7.60	10.44	19.46	20.84	29.85	27.02	48.09	35.60
Qwen2.5-7B (6.85)	7.20	7.98	7.76	34.10	7.84	8.56	7.82	12.55	150.93	106.59	188.22	7791	255.23	140.65

Table 1: WikiText-2 perplexity at

\rho\in\{0.6,0.2\}

(dense ppl in parentheses; best per row in bold). All methods use the same realised compression ratio. Here, “enh-L”, “TLLM”, and “SVD” abbreviate enh-LeSTD, TLLM-gen, and SVD-LLM, respectively.

Figure 2: WikiText-2 perplexity versus retained attention-projection storage ratio across seven models. Dashed lines denote the corresponding dense-model perplexity.

		Ftc		Seq-SVD		enh-LeSTD		TLLM-gen		MPIFA		SVD-LLM		Basis
Model	$\rho$	a6	G8K	a6	G8K	a6	G8K	a6	G8K	a6	G8K	a6	G8K	a6	G8K
Mistral-7B (0.740 / 0.285)	0.6	0.726	0.24	0.684	0.17	0.707	0.15	0.654	0.10	0.695	0.20	0.667	0.17	0.688	0.20
	0.4	0.689	0.10	0.569	0.01	0.625	0.01	0.501	0.01	0.552	0.02	0.505	0.02	0.549	0.03
	0.2	0.512	0.01	0.359	0.00	0.320	0.00	0.342	0.01	0.314	0.01	0.307	0.01	0.294	0.00
Llama-3-8B (0.735 / 0.435)	0.6	0.716	0.29	0.642	0.09	0.694	0.20	0.597	0.04	0.639	0.01	0.575	0.02	0.615	0.04
	0.4	0.668	0.10	0.510	0.04	0.601	0.04	0.447	0.03	0.448	0.02	0.410	0.04	0.474	0.03

Table 2: Downstream performance on modern GQA models at

\rho\in\{0.6,0.4,0.2\}

. avg6 is the mean normalized accuracy over six multiple-choice tasks; GSM8K is 5-shot exact match. Dense results are shown in the row labels; full per-task results are in Appendix A.3 .

Figure 3: Downstream retention versus retained storage ratio on five GQA models. Top: avg6 relative to dense; bottom: 5-shot GSM8K relative to dense. The dashed line marks 10% GSM8K retention.

Model	$\rho$	full	w/o whitening	w/o slice-wise norm	w/o seq. calibration	w/o anchoring	K/V repeat-expand	dense core	no joint tensor	w/o $W_{O}$ target
Mistral-7B	0.6	5.440	5.56 (+2.2%)	5.50 (+1.1%)	5.46 (+0.4%)	5.44 (+0.0%)	5.53 (+1.6%)	6.37 (+17.2%)	5.69 (+4.7%)	5.46 (+0.4%)
Mistral-7B	0.2	7.334	13.19 (+79.8%)	9.33 (+27.2%)	13.24 (+80.6%)	7.45 (+1.5%)	10.41 (+41.9%)	7.65 (+4.3%)	12.10 (+65.0%)	10.15 (+38.4%)
Llama-3-8B	0.6	6.640	6.73 (+1.4%)	7.01 (+5.5%)	6.94 (+4.5%)	6.60 (-0.6%)	6.87 (+3.4%)	8.45 (+27.3%)	7.03 (+5.9%)	6.94 (+4.5%)
Llama-3-8B	0.2	10.440	20.56 (+96.9%)	19.77 (+89.4%)	33.65 (+222.4%)	10.87 (+4.1%)	17.18 (+64.6%)	11.19 (+7.2%)	19.46 (+86.4%)	29.50 (+182.6%)
Qwen2.5-7B	0.6	7.197	7.77 (+7.9%)	7.17 (-0.3%)	7.24 (+0.5%)	7.24 (+0.5%)	7.24 (+0.6%)	12.71 (+76.6%)	7.98 (+10.9%)	7.19 (-0.1%)
Qwen2.5-7B	0.2	12.548	221.15 (+1662.4%)	74.34 (+492.4%)	21.36 (+70.2%)	14.36 (+14.5%)	12.88 (+2.6%)	26.89 (+114.3%)	150.93 (+1102.8%)	12.88 (+2.6%)

Table 3: Component ablations on WikiText-2. “full” denotes the final fixed head-concat recipe; each column removes or replaces one component at the same realized storage ratio and reports perplexity with the relative change in parentheses. Relative changes are computed against the matching full-recipe result in the same row.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

		Whole-model storage removed (%)
Model	Attn. share	$\rho=.8$	$\rho=.6$	$\rho=.4$	$\rho=.2$
GPT-J-6B	31.1%	6.2	12.4	18.7	24.9
Llama-2-13B	32.2%	6.4	12.9	19.3	25.8
Mistral-7B	18.5%	3.7	7.4	11.1	14.8
Llama-3-8B	16.7%	3.3	6.7	10.0	13.4
Qwen2.5-7B	10.8%	2.2	4.3	6.5	8.6

Appendix

Table 4: Storage accounting at each attention-projection storage budget

\rho

. Realized storage is determined by the selected layerwise Tucker ranks

(R_{1,l},R_{2,l})

, sparse-core cardinality

K_{l}

, output-projection rank

r_{O,l}

, and bit-packed support mask.

Model	mode	0.8	0.6	0.4	0.2
GPT-J-6B	1 card	38	34	26	21
Llama-2-13B	1 card	166	93	75	39
Mistral-7B	1 card	33	30	26	26
Llama-3-8B	1 card	34	29	27	25
Qwen2.5-7B	1 card	21	20	16	16
Qwen3-8B	1 card	39	33	28	27

Appendix

Table 5: Ftc compression wall-clock time (minutes) per model–ratio configuration. Columns correspond to

\rho=0.8/0.6/0.4/0.2

. All models use the fixed head-concat recipe on a single A100-40GB GPU; Qwen3-32B additionally uses CPU offloading and is compressed one decoder layer at a time.

Model	$\rho$	Ftc	Seq-SVD	enh-LeSTD	TLLM-gen	MPIFA	SVD-LLM	Basis
GPT-J-6B	0.8	9.04	9.14	8.99	9.24	9.04	9.60	9.61
	0.6	9.30	9.47	9.93	9.68	9.46	10.26	10.16
	0.4	10.03	10.46	10.31	11.05	10.68	12.16	11.80
	0.2	13.60	14.84	17.47	19.22	17.24	25.04	23.02
Llama-2-13B	0.8	4.98	5.06	4.92	5.13	5.00	5.31	5.28
	0.6	5.11	5.27	5.13	5.52	5.26	5.68	5.57

Appendix

Table 6: Full WikiText-2 perplexity results over all models and

\rho\in\{0.8,0.6,0.4,0.2\}

. Best results in each row are highlighted.

		PTB							C4
Model	$\rho$	F	qS	eL	TG	MP	SL	BS	F	qS	eL	TG	MP	SL	BS
GPT-J-6B	0.8	16.33	16.80	16.28	17.03	16.30	17.45	17.42	13.62	14.12	13.29	14.06	13.62	14.84	14.83
	0.6	17.20	18.39	17.85	18.43	17.46	19.74	19.44	14.33	15.24	14.18	15.09	14.82	16.45	16.27
	0.4	20.34	25.08	21.93	25.25	24.20	30.19	28.64	16.61	18.80	16.94	18.75	18.69	21.89	21.26
	0.2	50.52	67.11	82.87	101.12	88.92	162.55	115.96	29.12	34.64	40.79	42.92	42.27	65.53	59.42
Llama-2-13B	0.8	52.27	61.58	53.14	54.14	56.42	61.75	60.82	7.05	7.60	7.01	7.45	7.04	8.22	8.14

Appendix

Table 7: PTB (left block) and C4 (right block) perplexity, method order as in Table 1 (F = Ftc , qS = Seq-SVD, eL = enh-LeSTD, TG = TLLM-gen, MP = MPIFA, SL = SVD-LLM, BS = Basis). Mistral’s high PTB values reflect its tokenizer/domain mismatch and affect all methods equally.

Model	Method	$\rho$	PIQA	ARC-e	ARC-c	HSwag	WinoG	LAMB	MathQA	GSM8K	avg6
GPT-J-6B	dense	1.0	0.756	0.616	0.367	0.654	0.654	0.678	0.281	0.015	0.621
	Ftc	0.8	0.747	0.607	0.358	0.642	0.659	0.665	0.269	0.010	0.613
	Ftc	0.6	0.742	0.589	0.333	0.626	0.640	0.658	0.264	0.005	0.598
	Ftc	0.4	0.735	0.577	0.317	0.567	0.618	0.573	0.263	0.025	0.564
	Ftc	0.2	0.684	0.464	0.270	0.438	0.575	0.255	0.258	0.000	0.448
	enh-LeSTD	0.8	0.758	0.620	0.362	0.649	0.651	0.682	0.270	0.010	0.620

Appendix

Table 8: Complete downstream grid, MHA models (acc_norm; GSM8K exact match).

Model	Method	$\rho$	PIQA	ARC-e	ARC-c	HSwag	WinoG	LAMB	MathQA	GSM8K	avg6
Mistral-7B	dense	1.0	0.814	0.807	0.530	0.796	0.740	0.753	0.367	0.285	0.740
	Ftc	0.8	0.805	0.812	0.521	0.779	0.745	0.737	0.345	0.235	0.733
	Ftc	0.6	0.805	0.811	0.506	0.765	0.741	0.728	0.336	0.240	0.726
	Ftc	0.4	0.795	0.781	0.474	0.726	0.701	0.655	0.303	0.105	0.689
	Ftc	0.2	0.720	0.595	0.321	0.555	0.619	0.263	0.247	0.015	0.512
	Seq-SVD	0.6	0.781	0.759	0.450	0.711	0.716	0.690	0.320	0.175	0.684

Appendix

Table 9: Complete downstream grid, GQA and hybrid models.

Model	Method	$\rho$	PIQA	ARC-e	ARC-c	HSwag	WinoG	LAMB	MathQA	GSM8K	avg6
Qwen3-8B	dense	1.0	0.786	0.801	0.561	0.774	0.724	0.704	0.562	0.855	0.725
	Ftc	0.8	0.788	0.832	0.583	0.763	0.735	0.664	0.563	0.765	0.728
	Ftc	0.6	0.781	0.793	0.564	0.750	0.715	0.640	0.515	0.735	0.707
	Ftc	0.4	0.730	0.641	0.441	0.712	0.670	0.553	0.316	0.105	0.624
	Ftc	0.2	0.690	0.550	0.324	0.550	0.582	0.305	0.247	0.005	0.500
	Seq-SVD	0.6	0.765	0.712	0.487	0.718	0.699	0.627	0.423	0.455	0.668

Appendix

Table 10: Complete downstream grid, GQA and hybrid models (continued).

$\rho{=}0.2$ , wt2 ppl	$\mu{=}0.001$	$0.03$	$0.3$	$1$	$3$	$10$	no QKV map (final)
Mistral-7B	8.10	8.13	8.04	7.85	7.64	7.91	7.33
Llama-3-8B	12.40	12.35	11.87	11.32	11.18	11.37	10.44
Qwen2.5-7B	16.52	16.80	17.42	16.84	14.72	13.97	12.55
Qwen3-8B	12.26	12.09	11.38	11.07	10.93	11.61	10.70

Appendix

Table 11: Sensitivity to QKV teacher-map regularization at

\rho=0.2

Model	$\rho$	Variant	variant	full	$\Delta$
Mistral-7B	0.2	fixed $R_{1}{=}640,R_{2}{=}128$	7.786	7.33	$+6.2\%$
Llama-3-8B	0.2	fixed $R_{1}{=}640,R_{2}{=}128$	10.975	10.44	$+5.1\%$
Qwen3-8B	0.2	fixed $R_{1}{=}640,R_{2}{=}128$	11.100	10.70	$+3.7\%$
Mistral-7B	0.6	$W_{O}$ with QKV metric	8.456	5.44	$+55.4\%$
Mistral-7B	0.2	$W_{O}$ with QKV metric	303.397	7.33	$+4039.1\%$
Llama-3-8B	0.6	$W_{O}$ with QKV metric	53.677	6.64	$+708.4\%$

Appendix

Table 12: Perplexity ablations for rank allocation and the output-projection metric.

Model	Variant	Full	$\Delta$
Llama-3-8B	6.655	6.613	6.607
Qwen3-8B	7.855	7.764	7.759
Qwen2.5-7B	7.322	7.312	7.308

Appendix

Table 13: Perplexity sensitivity to one versus two refit iterations.

	$\rho=0.8$			$\rho=0.6$			$\rho=0.4$			$\rho=0.2$
Model	ppl	avg6	GSM8K	ppl	avg6	GSM8K	ppl	avg6	GSM8K	ppl	avg6	GSM8K
GPT-J-6B	9.04	0.613	0.010	9.30	0.598	0.005	10.03	0.564	0.025	13.60	0.448	n/a
Llama-2-13B	4.98	0.715	0.190	5.11	0.703	0.115	5.52	0.660	0.030	7.52	0.497	n/a
Mistral-7B	5.36	0.733	0.235	5.44	0.726	0.240	5.70	0.689	0.105	7.33	0.512	0.015
Llama-3-8B	6.36	0.729	0.395	6.64	0.716	0.290	7.31	0.668	0.100	10.44	0.488	0.020
Qwen2.5-7B	6.99	0.715	0.780	7.20	0.689	0.715	7.84	0.659	0.445	12.55	0.405	0.015

Appendix

Table 14: The final recipe on the six primary evaluation models at all four keep ratios. WikiText-2 perplexity, avg6, and GSM8K accuracy. GSM8K was not evaluated for GPT-J-6B and Llama-2-13B at

\rho=0.2

Model	$\rho$	$\varepsilon{=}10^{-4}$ (recipe)	$\varepsilon{=}10^{-2}$	$\varepsilon{=}1$
Llama-3-8B	.6	6.64	6.63 ( $-0.1\%$ )	6.54 ( $-1.5\%$ )
Llama-3-8B	.2	10.44	10.59 ( $+1.4\%$ )	12.21 ( $+17.0\%$ )
Mistral-7B	.6	5.44	5.44 ( $-0.0\%$ )	5.45 ( $+0.2\%$ )
Mistral-7B	.2	7.33	7.43 ( $+1.4\%$ )	8.19 ( $+11.7\%$ )
Qwen3-8B	.6	7.36	7.39 ( $+0.4\%$ )	7.58 ( $+3.0\%$ )
Qwen3-8B	.2	10.70	10.99 ( $+2.7\%$ )	13.36 ( $+24.8\%$ )

Appendix

Table 15: Sensitivity to whitening damping

\epsilon

H_{\epsilon,l}=C_{s,l}+\epsilon\bar{t}_{l}I

, where

\bar{t}=\frac{\operatorname{tr}(C_{s,l})}{d_{\mathrm{model}}}

. The recipe uses

\epsilon=10^{-4}

Model	QKV target	layers escalated	reaching $\varepsilon{=}1$
Mistral-7B	no QKV map (final)	4/32	0
Mistral-7B	$\mu{=}1$ map	5/32	1
Llama-3-8B	no QKV map (final)	2/32	0
Llama-3-8B	$\mu{=}1$ map	2/32	0
Qwen2.5-7B	no QKV map (final)	9/28	3
Qwen2.5-7B	$\mu{=}1$ map	5/28	0

Appendix

Table 16: Layers whose whitening damping had to be escalated above

\varepsilon{=}10^{-4}

\rho{=}0.2

, and how many of those ended at the ridge metric

\varepsilon{=}1

. Parsed from the run logs. At

\rho\geq 0.4

all counts are zero on all four models.

Model	$\rho$	256 windows	128 windows	$\Delta$
Llama-3-8B	0.6	6.640	6.660	$+0.3\%$
Llama-3-8B	0.2	10.440	10.821	$+3.7\%$
Mistral-7B	0.6	5.440	5.447	$+0.1\%$
Mistral-7B	0.2	7.330	7.499	$+2.3\%$
Qwen2.5-7B	0.6	7.200	7.192	-0.1%
Qwen2.5-7B	0.2	12.550	43.901	+249.8%

Appendix

Table 17: Sensitivity of Ftc to calibration-set size. We compare the same final recipe using 256 versus 128 WikiText-2 calibration windows of 2,048 tokens.

Model	$\rho$	Unstructured core	2:4 core	$\Delta_{\rm ppl}$
Mistral-7B	0.6	5.44	5.47	$+0.6\%$
Mistral-7B	0.4	5.70	5.76	$+1.1\%$
Mistral-7B	0.2	7.33	8.64	$+17.9\%$
Llama-3-8B	0.6	6.64	6.72	$+1.2\%$
Llama-3-8B	0.4	7.31	7.46	$+2.0\%$
Llama-3-8B	0.2	10.44	12.16	$+16.5\%$

Appendix

Table 18: Unstructured vs. 2:4 core at equal

\rho

. Both columns are the final recipe at a realised

\rho

, differing only in the core structure.

Model	$\rho$	Method	Calibration	wt2	PTB	C4
Mistral-7B	0.6	SVD-LLM	WikiText-2	5.92	48.17	10.76
Mistral-7B	0.6	SVD-LLM	C4	6.59	46.76	9.74
Mistral-7B	0.6	MPIFA	WikiText-2	5.63	41.47	9.78
Mistral-7B	0.6	MPIFA	C4	6.01	41.09	9.17
Mistral-7B	0.6	Basis	WikiText-2	5.75	46.79	10.11
Mistral-7B	0.6	Basis	C4	6.21	46.32	9.38

Appendix

Table 19: Data-dependent methods under WikiText-2 and C4 calibration.

Figure 4: Layer-wise relative Frobenius drift of the residual stream between compressed and dense models on 16 sequences disjoint from the calibration set. We compare Ftc with five representative methods on four GQA models at

\rho\in\{0.6,0.2\}

. Lower is better. Ftc generally maintains smaller drift across depth, particularly under aggressive compression on Mistral-7B and Llama-3-8B. On the Qwen models, the metric becomes less discriminative through the middle layers at

\rho=0.2

, with clearer separation near the output.

Model	WT2 ( $\rho=.6$ )	WT2 ( $\rho=.2$ )	avg6 ( $\rho=.2$ )
Mistral-7B	$5.44\pm 0.00$	$7.30\pm 0.02$	$0.511\pm 0.005$
Llama-3-8B	$6.65\pm 0.00$	$10.45\pm 0.03$	$0.493\pm 0.003$
Qwen2.5-7B	$7.20\pm 0.01$	$12.65\pm 0.07$	$0.405\pm 0.002$
Qwen3-8B	$7.39\pm 0.03$	$10.70\pm 0.02$	$0.494\pm 0.008$

Appendix

Table 20: Calibration-seed robustness of Ftc . Mean

\pm

std over three calibration sets (seeds

\{42,1,2\}

Explore similar work

A general tensor-structured compression scheme for efficient large language models
May 25, 2026Ying Lu, Peng-Fei Zhou, Qi-Xuan Fang +3Large Language Model Compression Efficient Large Language Model
School of Physical Sciences, University of Chinese Academy of Sciences, Beijing, 100049, China. · Kavli Institute for Theoretical Sciences, University of Chinese Academy of Sciences, Beijing, 100190, China. · Center for Quantum Physics and Intelligent Sciences, Department of Physics, Capital Normal University, Beijing, 100048, China. +1
Compressing Sequences in the Latent Embedding Space: $K$ -Token Merging for Large Language Models
Apr 16, 2026Zihao Xu, John Harvill, Ziwei Fan +3Large Language Model Compression Task-Aware Compression
♢Rutgers University · ♡Mistral AI · ♠AWS AI Labs +3
CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning
Jul 20, 2026Zhiren Gong, Zihao Zeng, Zijie Wang +3Structured Pruning Large Language Model Compression
College of Computing and Data Science, Nanyang Technological University, Singapore · Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore

May 25, 2026, cs.CLJ/K move · Enter open · S save

A general tensor-structured compression scheme for efficient large language models

Ying Lu, Peng-Fei Zhou, Qi-Xuan Fang, Pan Zhang+2

School of Physical Sciences, University of Chinese Academy of Sciences, Beijing, 100049, China. · Kavli Institute for Theoretical Sciences, University of Chinese Academy of Sciences, Beijing, 100190, China. · Center for Quantum Physics and Intelligent Sciences, Department of Physics, Capital Normal University, Beijing, 100048, China.+1

Large Language Model Compression Efficient Large Language Model

Open paper AskPDF arXiv

Large language models (LLMs) are dominated by dense linear transformations, whose storage, memory and computational overheads hinder efficient adaptation and deployment while masking the functional impacts of structural simplification. Here we present Tensor Mixture (MixT), a general tensor-structured compression scheme that replaces targeted dense linear layers with natively executable mixtures of tensor operators. Operating directly on generic linear projections instead of model-specific components, MixT is potentially applicable across Transformer-based LLMs and other dense neural mappings. We evaluate MixT on Qwen3-8B and LLaMA2-7B under a unified recovery protocol, identifying a broad compressible regime in which MMLU accuracy is largely preserved before an abrupt transition at model-specific boundaries. This transition coincides with coordinated shifts in output entropy, prediction entropy and inter-layer geometry. At the LLaMA2-7B transition boundary, MixT reduces full-model parameters by 47.5%, inference FLOPs by 37.1%, training FLOPs by 52.1% and peak inference memory by 60.4%, demonstrating its practical potential for lower-cost LLM compression.

Abstract

Explore similar work

A general tensor-structured compression scheme for efficient large language models

Compressing Sequences in the Latent Embedding Space: KKK-Token Merging for Large Language Models

CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning

Compressing Sequences in the Latent Embedding Space: $K$ -Token Merging for Large Language Models