COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression

Denis Makhov; Dmitriy Shopkhoev; Magauiya Zhussip; Ammar Ali; Baher Mohammad; Stamatios Lefkimmiatis

cs.LGFeb 16, 2026

COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression

Authors: Denis Makhov, Dmitriy Shopkhoev, Magauiya Zhussip, Ammar Ali, Baher Mohammad, Stamatios Lefkimmiatis

Organizations: Fundamental Research Center MWS AI · ITMO

Abstract

Post-training compression of Transformer models commonly relies on truncated singular value decomposition (SVD). However, enforcing a single shared subspace can degrade accuracy even at moderate compression. Sparse dictionary learning provides a more flexible union-of-subspaces representation, but existing approaches often suffer from iterative dictionary and coefficient updates. We propose COMPOT (Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers), a training-free compression framework that uses a small calibration dataset to estimate a sparse weight factorization. COMPOT employs orthogonal dictionaries that enable closed-form Procrustes updates for the dictionary and analytical single-step sparse coding for the coefficients, eliminating iterative optimization. To handle heterogeneous layer sensitivity under a global compression budget, COMPOT further introduces a one-shot dynamic allocation strategy that adaptively redistributes layer-wise compression rates. Extensive experiments across diverse architectures and tasks show that COMPOT consistently delivers a superior quality-compression trade-off over strong low-rank and sparse baselines, while remaining fully compatible with post-training quantization for extreme compression. Code is available here.

Figures & tables

Figure 1: Comparison of low-rank decomposition, dictionary learning with sparse coding, and COMPOT. Low-rank uses a rigid shared orthogonal basis $\mathbf{B}$ ; dictionary learning enables a flexible union-of-subspaces ; COMPOT lies in between by using a union-of- orthogonal -subspaces (denoted with O ), which enables fast closed-form dictionary updates and a lightweight coefficient update, improving compression performance.

Figure 2: Overview of COMPOT framework. On the left part the alternating minimization process is visualized while on the right we represent our single-shot strategy for dynamic compression ratio allocation based on singular values of normalized projection matrices.

Table 3

		Accuracy $\uparrow$									Perplexity $\downarrow$
Method	CR	PIQA	Hella Swag	LAMBADA	ARC-e	ARC-c	SciQ	Race	MMLU	Avg.	Wiki Text	LAMBADA
Llama3 8B	–	80.7	79.1	75.6	77.7	53.5	93.9	40.3	62.2	70.4	7.3E+00	3.1E+00
SVD-LLM		71.1	58.4	59.3	55.5	34.0	86.4	35.5	32.6	54.1	4.1E+01	1.1E+01
CoSpaDi		75.2	66.5	73.8	66.5	41.6	89.5	38.2	42.8	61.8	2.0E+01	4.3E+00
COMPOT †	0.2	77.8	72.7	69.4	71.0	44.5	89.8	41.0	50.1	64.5	1.3E+01	5.2E+00
SVD-LLM		65.8	46.4	38.1	41.9	27.7	70.0	31.8	27.2	43.6	1.5E+02	6.1E+01

Table 3: Performance comparison under static compression ratio allocation of COMPOT † (static CR) vs state-of-the-art SVD-based SVD-LLM and dictionary-learning CoSpaDi on Llama3-8B and Qwen3-8B at different compression levels on different benchmarks. Best results are highlighted with bold .

Figure 3: Average accuracy as a function of the number of alternating minimization steps on Llama3.2-1B at $0.2$ compression, comparing random and SVD-based dictionary initialization.

Method	Training-free	CR	Perplexity $\downarrow$		Accuracy $\uparrow$
Method	Training-free	CR	WikiText-2	C4	Openb.	ARC_e	WinoG.	HellaS.	ARC_c	PIQA	MathQA	Avg.
Llama2-7B	—		5.49	7.27	44.0	74.6	69.0	76.3	45.5	78.6	28.4	59.5
Dobi-SVD †	✗	0.2	9.39	19.70	32.8	47.6	58.5	51.5	29.3	65.2	22.7	43.9
COMPOT	✓	0.2	6.22	9.34	30.4	72.6	67.6	51.8	39.6	76.4	26.2	52.1
Dobi-SVD †	✗	0.4	17.50	55.58	29.2	34.4	54.4	35.4	24.5	57.3	21.3	36.6
COMPOT	✓	0.4	8.91	20.40	25.2	63.1	63.5	39.6	28.5	67.0	23.6	44.4

Table 4: Performance comparison under dynamic compression ratio allocation of training-free COMPOT vs training-based state-of-the-art SVD-based Dobi-SVD ( Qinsi et al., 2025 ) on Llama2-7B at different compression levels on different benchmarks. Best results are highlighted with bold .

Methods	Llama-7B	OPT-6.7B	Llama3-8B
Methods	WikiText-2 / C4 $\downarrow$	WikiText-2 / C4 $\downarrow$	WikiText-2 / C4 $\downarrow$
Original	5.67 / 7.33	10.92 / 12.57	6.14 / 9.47
SVD-LLM V2	7.12 / 10.47	13.46 / 17.72	8.01 / 11.72
SVD-LLM V2 (repr.)	8.09 / 16.29	— / —	47.58 / 73.33
COMPOT	6.52 / 9.63	10.92 / 13.17	8.64 / 18.29

Table 5: Performance comparison under dynamic compression ratio allocation between COMPOT and SVD-LLM V2 ( Wang et al., 2025a ) on Llama-7B, OPT-6.7B, and Llama3-8B at a 0.2 compression level across different benchmarks. We present both originally published results and our reproduced results (denoted as ’repr.’). Best results are highlighted in bold . Discrepancies between published and reproduced results are thoroughly explained in A.10

Method	CR	PIQA	Hella Swag	LAMBADA	ARC-e	ARC-c	SciQ	Race	MMLU	Avg.	Wiki Text	LAMBADA
		Accuracy $\uparrow$									Perplexity $\downarrow$
Llama3 8B		80.7	79.1	75.6	77.7	53.5	93.9	40.3	62.2	70.4	7.3E+00	3.1E+00
ReplaceMe	0.22	73.1	65.7	42.1	65.9	43.7	86.4	35.4	51.7	58.0	3.4E+01	2.0E+01
LLM-Pruner		75.5	67.5	51.0	62.1	36.6	87.8	35.1	25.0	55.1	1.6E+01	1.1E+01
COMPOT	0.20	78.0	73.5	70.5	71.3	45.3	91.0	40.1	52.4	65.3	1.2E+01	4.6E+00
ReplaceMe	0.31	66.6	53.8	24.0	50.7	37.9	77.3	34.0	30.6	46.9	6.7E+01	1.3E+02

Table 6: Comparison of the COMPOT with other training-free methods including state-of-the-art structured pruning methods ReplaceMe Shopkhoev et al. (2025a) and LLM-Pruner Ma et al. (2023) on Llama3 8B under different compression ratios. We report accuracy on different benchmarks as well as its average and perplexity. Best results are highlighted with bold

Method	Weight memory	Quant. CR	Factor. CR	PPL
GPTQ-3bit	2.8 GB	0.81	N/A	16.28
SVD-LLM V2+GPTQ-4bit	2.8 GB	0.75	0.25	9.97
COMPOT † +GPTQ-4bit	2.8 GB	0.75	0.25	9.93
COMPOT+GPTQ-4bit	2.8 GB	0.75	0.25	9.62

Table 7: Perplexity (

\downarrow

) of COMPOT vs state-of-the-art SVD-based SVD-LLM V2 ( Wang et al., 2025a ) accompanied by quantization on Llama-7B on WikiText-2. Best results are highlighted with bold .

Method	MMMU $\uparrow$	OCRBench $\uparrow$	RealWorldQA $\uparrow$	MMStar $\uparrow$
Original	52.6	82.6	69.7	63.2
SVD-LLM	29.2	35.1	52.9	31.6
COMPOT †	41.9	64.1	59.2	50.8
COMPOT	44.7	66.9	62.2	54.2

Table 8: Multimodal performance of Qwen3-VL-8B-Instruct under

0.2

compression ratio (CR) on vision-language benchmarks. Best results are highlighted in bold .

Method	CR	$\textbf{WER}_{\textbf{test-clean}}$ $\downarrow$	$\textbf{WER}_{\textbf{test-other}}$ $\downarrow$
Whisper Large V3	-	2.74	4.53
SVD-LLM	0.2	4.12	6.8
COMPOT †	0.2	2.46	4.51
SVD-LLM	0.3	12.78	15.54
COMPOT †	0.3	2.74	5.21

Table 9: ASR Performance (WER

\downarrow

) on LibriSpeech benchmarks for Whisper Large V3 model. Best results are highlighted with bold .

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

P(K)=\sum_{i\in\textsc{dense}}m_{i}n_{i}\;+\!\!\sum_{i\notin\textsc{dense}}r_{i}(K)(m_{i}+n_{i})\leq P_{\text{tgt}},\quad r_{i}(K)=L_{i}-t_{i}(K).

Appendix

Algorithm 2 Allocate-Global : pooled-SV truncation with per-matrix CR guards (Frobenius-normalized, no calibration)

Method	CR	PIQA	Hella Swag	LAMBADA	ARC-e	ARC-c	SciQ	Race	MMLU	Avg.	Wiki Text	LAMBADA
		Accuracy $\uparrow$									Perplexity $\downarrow$
Llama3.2-1B	–	74.5	63.7	63.0	60.5	36.2	88.3	37.8	37.0	57.6	1.2E+01	5.7E+00
SVD-LLM		62.1	36.4	24.4	36.0	25.1	64.9	29.0	23.0	37.6	1.7E+02	1.7E+02
CoSpaDi		66.1	42.9	38.4	39.9	26.0	71.6	31.7	24.8	42.7	6.4E+01	3.5E+01
COMPOT †		69.7	52.1	42.1	49.9	30.2	81.8	33.9	28.5	48.5	2.5E+01	2.1E+01
COMPOT	0.2	70.6	54.6	47.6	52.5	31.1	81.6	35.7	26.7	50.1	2.1E+01	1.4E+01

Appendix

Table 10: Performance comparison under static (COMPOT † ) and dynamic (COMPOT) vs state-of-the-art SVD-based SVD-LLM and dictionary-learning CoSpaDi on Llama3.2-1B at different compression levels on different benchmarks. Best results are highlighted with bold .

Method	CR	PIQA	Hella Swag	LAMBADA	ARC-e	ARC-c	SciQ	Race	MMLU	Avg.	Wiki Text	LAMBADA
		Accuracy $\uparrow$									Perplexity $\downarrow$
Qwen3-0.6B	–	67.5	47.4	40.0	55.6	34.3	83.5	33.7	40.2	50.3	2.6E+01	2.5E+01
SVD-LLM		57.0	33.0	12.6	33.6	24.8	54.9	29.5	26.1	33.9	3.1E+03	7.3E+03
CoSpaDi		59.3	37.5	18.7	37.2	27.1	68.6	32.4	26.6	38.4	1.9E+03	9.4E+02
COMPOT †		62.2	40.6	15.9	41.3	29.6	70.8	33.0	26.3	40.0	1.4E+03	4.2E+02
COMPOT	0.2	61.2	42.6	20.6	42.9	28.8	74.0	32.3	24.9	40.9	7.0E+01	1.9E+02

Appendix

Table 11: Performance comparison under static (COMPOT † ) and dynamic (COMPOT) vs state-of-the-art SVD-based SVD-LLM and dictionary-learning CoSpaDi on Qwen3-0.6B at different compression levels on different benchmarks. Best results are highlighted with bold .

	Accuracy $\uparrow$
Method	CR	BBH	GPQA	IFEVAL	MATH HARD	MMLU Pro	MUSR
Qwen3-0.6B	–	36.85	28.27	27.58	12.69	24.26	31.35
SVD-LLM		28.95	24.75	25.3	0	11.29	35.58
COMPOT †		29.2	27.94	27.22	1.06	12.81	35.19
COMPOT	0.2	30.19	25.34	25.9	0.91	14.81	33.2
SVD-LLM		29.73	25.5	24.34	0	11.22	33.73

Appendix

Table 12: Performance comparison under static (COMPOT † ) and dynamic (COMPOT) vs SVD-LLM on Open LLM Leaderboard v2 Fourrier et al. (2024)

Layer	Dimensions	SVD-LLM	CoSpaDi	COMPOT	Speedup over CoSpaDi
model.layers.0.mlp.up_proj	(2048, 8192)	1.96	508.82	17.84	28.52x
model.layers.0.mlp.down_proj	(8192, 2048)	13.99	255.51	19.23	13.29x
model.layers.0.mlp.gate_proj	(2048, 8192)	2.19	522.17	17.78	29.37x
model.layers.0.self_attn.q_proj	(2048, 2048)	1.98	154.01	5.37	28.68x
model.layers.0.self_attn.k_proj	(2048, 512)	1.03	44.70	1.99	22.41x
model.layers.0.self_attn.v_proj	(2048, 512)	0.50	44.56	1.77	25.12x

Appendix

Table 13: Wall-clock time in seconds required for optimization with SVD-LLM, CoSpaDi and COMPOT for Llama3.2-1B with static 0.2 compression ratio.

Figure 4: Results of COMPOT allocation strategy for Llama3.2-1B

Figure 5: Results of COMPOT allocation strategy for Qwen3-0.6B

Figure 6: Results of COMPOT allocation strategy for Llama3-8B

Figure 7: Results of COMPOT allocation strategy for Qwen3-8B

Figure 8: Results of COMPOT allocation strategy for Llama2-7B

Figure 9: Results of COMPOT allocation strategy for Llama-7B

Figure 10: Results of COMPOT allocation strategy for OPT-6.7B

Figure 11: Results of COMPOT allocation strategy for Llama-13B

Figure 12: Results of COMPOT allocation strategy for Llama-30B

Method	Tolerance	PIQA	HellaSwag	LAMBADA	ARC-e	ARC-c	SciQ	Race	MMLU	Avg.	WikiText	LAMBADA
	Relative	Accuracy $\uparrow$									Perplexity $\downarrow$
Llama 3.2-1B	–	0.7453	0.6366	0.6295	0.6047	0.3620	0.8830	0.3779	0.3700	0.5761	11.57	5.73
COMPOT †	$10^{-1.0}$	0.6779	0.4758	0.3503	0.4541	0.2645	0.7680	0.3368	0.2463	0.4467	33.61	31.13
COMPOT †	$10^{-1.5}$	0.6736	0.4894	0.3728	0.4760	0.2816	0.7930	0.3292	0.2453	0.4576	29.12	29.19
COMPOT †	$10^{-2.0}$	0.6823	0.5026	0.3831	0.4773	0.2969	0.7980	0.3455	0.2638	0.4687	26.08	27.38
COMPOT †	$10^{-2.5}$	0.6931	0.5136	0.3887	0.4794	0.3029	0.8060	0.3378	0.2710	0.4741	24.20	26.01

Appendix

Table 14: Performance of COMPOT under varying relative tolerance (

\tau

) thresholds for early stopping. Results are evaluated using Llama 3.2-1B with the compression rate and dictionary-to-sparsity ratio fixed at 2. The maximum number of iterations is set to 150. Bold values indicate the best performance for each metric.

	$k/s$	Accuracy $\uparrow$									Perplexity $\downarrow$
Method	ratio	PIQA	HellaSwag	LAMBADA	ARC-e	ARC-c	SciQ	Race	MMLU	Avg.	WikiText	LAMBADA
Llama 3.2-1B	–	0.7453	0.6366	0.6295	0.6047	0.3620	0.8830	0.3779	0.3700	0.5761	11.57	5.73
COMPOT †	1.2	0.6420	0.4027	0.3022	0.3960	0.2577	0.7250	0.3110	0.2311	0.4085	73.73	83.31
COMPOT †	1.4	0.6665	0.4573	0.3664	0.4369	0.2790	0.7790	0.3139	0.2487	0.4434	35.38	41.48
COMPOT †	1.6	0.6877	0.4909	0.4069	0.4726	0.2918	0.7990	0.3340	0.2582	0.4676	25.29	29.75
COMPOT †	1.8	0.6915	0.5142	0.4306	0.4853	0.2918	0.8140	0.3445	0.2585	0.4788	19.95	26.18

Appendix

Table 15: Performance of COMPOT across different dictionary-to-sparsity (

K/S

) ratios. The model compression rate is fixed at 20%. All experiments are conducted using the Llama 3.2-1B architecture. Values in bold indicate the highest performance achieved for each respective metric.

Method	CR	MMMU val $\uparrow$	OCRBench $\uparrow$	RealWorldQA $\uparrow$	MMStar $\uparrow$	Average $\uparrow$
Qwen 3 VL 8B	–	0.5256	0.8259	0.6967	0.6315	0.67
SVD-LLM	0.20	0.2922	0.3510	0.5294	0.3161	0.37
COMPOT †	0.20	0.4189	0.6410	0.5922	0.5080	0.54
COMPOT	0.20	0.4467	0.6690	0.6222	0.5417	0.57
SVD-LLM	0.30	0.2333	0.2120	0.2562	0.0398	0.19
COMPOT †	0.30	0.3378	0.5260	0.5137	0.3574	0.43

Appendix

Table 16: Full vision-language results for Qwen3-VL 8B under compression. We report benchmark scores on MMMU, OCRBench, RealWorldQA, and MMStar, along with the uncompressed model.

Method	CR	$\textbf{WER}_{\textbf{test-clean}}$ $\downarrow$	$\textbf{WER}_{\textbf{test-other}}$ $\downarrow$
Whisper Base-EN	-	4.40	10.40
SVD-LLM	0.05	32.37	32.18
COMPOT †	0.05	5.41	12.34
SVD-LLM	0.10	52.80	51.02
COMPOT †	0.10	5.88	12.75
SVD-LLM	0.15	85.64	83.31

Appendix

Table 17: ASR Performance (WER

\downarrow

) on LibriSpeech benchmarks for Whisper family.

	Llama-13B		Llama-30B
Method	PPL	Avg. Acc.	PPL	Avg. Acc.
Original	5.09	0.59	4.10	0.61
FWSVD	15.98	0.43	20.54	0.42
ASVD	6.74	0.54	22.71	0.44
SVD-LLM	6.61	0.55	5.63	0.57
SVD-LLM v2	5.46	0.56	4.71	0.60

Appendix

Table 18: Perplexity (PPL,

\downarrow

) and average accuracy (Avg. Acc.,

\uparrow

) for Llama-13B and Llama-30B.

Method	Target CR	Fact. CR	Quant. CR	WikiText	C4	Openb.	Arc_e	WinoG.	HellaS.	Arc_c	PIQA	MathQA	Average
Llama2 7B	—	—	—	5.49	7.27	44.0	74.6	69.0	76.3	45.5	78.6	28.4	59.5
Dobi-SVD*	0.2	0.2	—	9.39	19.70	32.8	47.6	58.5	51.5	29.3	65.2	22.7	43.9
Dobi-SVD		-0.6	0.5	5.92	8.68	42.2	72.3	69.1	72.9	44.0	78.1	27.5	58.0
COMPOT		0.2	—	6.22	9.34	30.4	72.6	67.6	51.8	39.6	76.4	26.2	52.1
Dobi-SVD*	0.4	0.4	—	17.50	55.58	29.2	34.4	54.4	35.4	24.5	57.3	21.3	36.6
Dobi-SVD		-0.2	0.5	7.89	14.23	37.2	59.4	62.7	60.6	35.0	72.1	25.5	50.4

Appendix

Table 19: Comparison of COMPOT with Dobi-SVD on Llama2-7B without (denoted with *) and with remapping. We provide also CR from matrix factorization (Fact. CR) and quantization (Quant. CR) to achieve specified target CR. In most cases Dobi-SVD with remapping leads to overparametrization with matrix factorization rather than compression

Orthogonal Procrustes Alignment

See all 11 papers

GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression
Sep 22, 2026Baher Mohammad, Ammar Ali, Stamatios LefkimmiatisModel Compression Low-Rank Compression
MWS AI, ITMO University · MWS AI
Learning Functional Subspaces for Neural Network Compression
Sep 30, 2026Massimo Bini, Anders Christensen, Stephan Alaniz +5LLM Compression Low-Rank Matrix Decomposition
Helmholtz Munich · Technical University of Munich · MCML +6
AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression
Apr 2, 2026Atul Kumar Sinha, François FleuretLLM Compression Low-Rank Compression
University of Geneva, Geneva, Switzerland · FAIR, Meta

Sep 22, 2026J/K move · Enter open · S save

GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression

Baher Mohammad, Ammar Ali, Stamatios Lefkimmiatis

MWS AI, ITMO University · MWS AI

Model Compression Low-Rank Compression Neural Network Compression

Open paper AskPDF arXiv

Transformer architectures exhibit cross-layer redundancies, yet post-training compression pipelines typically optimize layers in isolation or rely on heuristic grouping strategies that disregard layer-specific activation geometries. We introduce a principled, training-free framework that sequentially optimizes cross-layer weight pairings and shared-dictionary factorizations. Rather than forcing weights of adjacent layers to share a basis or heuristically merging activation statistics, our approach identifies structurally compatible projections and learns a shared representation that better preserves each layer's distinct calibration geometry. Coupled with structured sparsity, this yields highly efficient weight decompositions without sacrificing functional fidelity. Across diverse architectures, scales, and modalities, our method achieves state-of-the-art results, consistently outperforming independent structured weight decompositions and alternative pairwise weight factorizations, which operate under heuristic grouping strategies. By replacing heuristic engineering strategies with a convergent, optimization-driven pipeline, we establish a theoretically grounded foundation for scalable, transformer compression across different modalities.

Abstract

Orthogonal Procrustes Alignment

Explore similar work

GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression

Learning Functional Subspaces for Neural Network Compression

AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression