FermiGrad-SVD: An LLM Compression Pipeline with Global Rank Selection and Gauge Fixing

Roman Rausch; David Jansen; Sukhbinder Singh; Román Orús

cs.LGNov 26, 2025

FermiGrad-SVD: An LLM Compression Pipeline with Global Rank Selection and Gauge Fixing

Authors: Roman Rausch, David Jansen, Sukhbinder Singh, Román Orús

Organizations: Multiverse Computing, Paseo de Miramón 170, Planta 2, 20014 Donostia, Spain · Multiverse Computing, 192 Spadina Avenue, Toronto, ON M5T 2C2, Canada · Donostia International Physics Center, San Sebastián, Spain · Ikerbasque Foundation for Science, Bilbao, Spain

Abstract

Low-rank factorization of weight matrices is a promising approach to compress large language models (LLMs), but its practical effectiveness depends on allocating ranks across layers and efficiently representing the resulting factors. We introduce three complementary methods for improving SVD-based LLM compression. First, FermiGrad formulates global layer-wise rank allocation as a differentiable optimization problem by replacing discrete singular-value truncation with continuous Fermi gates. Because FermiGrad optimizes only rank parameters over precomputed low-rank factors, it avoids repeated activation decompositions and backpropagation through SVD operations. Second, the Targeted Compressor extends FermiGrad with a task-specific objective that directs limited model capacity toward a selected domain. At 60% parameter density for Llama-3.1-8B-Instruct, SciQ-targeted compression achieves 60% MMLU accuracy, exceeding the reported quantized Dobi-SVD result by 10 percentage points, while deliberately trading performance on unrelated tasks for domain specialization. Third, PivGa provides a lossless secondary compression of low-rank factors by exploiting the gauge freedom in their parametrization. The proposed framework is modular, computationally efficient, and applicable at scale, as demonstrated by compressing Llama-3.3-70B-Instruct to 50% parameter density.

Figures & tables

Figure 1: a) Illustration of FermiGrad for a single layer: $F$ is the active Fermi tensor that softens the SVD truncation while $A$ and $B$ are frozen, and $\mu$ provides a slider to select the singular values. b) Optimization space for two 1024 $\times$ 1024 layers (2M parameters) and 1.57M target parameters. The grey area shows the box constraint for the ranks: $1\leq r_{l}\leq 1024$ , $l=0,1$ . The red crosses denote the hypothesized optimal solution. The blue (light blue) lines show the parameter constraints for regular (secondary) compression. For the regular constraint, the FermiGrad algorithm starts with full rank and moves along some trajectory to the optimized solution as the penalty term is increased (see text).

Figure 2: Inference speed of the PivGa approach compared to PiFa [ 35 ] and the pure SVD model (without PiFa/PivGa) for random tokens with sequence length 256, batch size 32 on an H200 GPU using Llama-3.1-8B-Instruct in bfloat16.

Figure 3: Change in the accuracy vs. parameter-count curves when secondary compression (PivGa) is applied. This secondary compression is lossless, and so the benchmark curves essentially shift sideways, with a larger shift at lower compression rates.

density	Dataset	MMLU			HellaSwag			WinoGrande			GSM8K
density	Dataset	Unif	STRS	FG	Unif	STRS	FG	Unif	STRS	FG	Unif	STRS	FG
$1.0$	Original	67.8			59.1			73.6			75.9
0.9	Tulu	53.0	55.6	59.1	47.3	44.0	50.2	68.3	66.9	70.2	30.9	23.8	46.3
0.9	MMLU-train	54.3	55.5	60.9	48.7	46.6	52.3	71.1	70.8	73.2	3.6	0.0	18.3
0.8	Tulu	42.3	45.6	51.9	37.8	36.1	43.7	62.2	64.2	65.9	11.3	2.9	18.3
0.8	MMLU-train	45.4	48.7	54.2	41.1	38.3	46.2	66.3	67.4	70.2	0.0	0.0	0.3

Table 1: Benchmark accuracy after compression with uniform SVD-LLM slicing (Unif), sensitivity-based rank allocation STRS [ 33 ] , and FermiGrad (FG) for the Llama-3.1-8B-Instruct model, shown for the two representative calibration datasets Tulu and MMLU-train across target densities (fraction of parameters retained). For each density and benchmark, the single best result across the three methods and both datasets is shown in bold. GSM8K uses strict-match exact-match; the remaining accuracies use standard accuracy. WikiText perplexity and the full comparison across all four calibration datasets are given in Tab. 4 (Appendix C ).

density	Method	MMLU	MedMCQA	MedQA	HellaSwag	WinoGrande
$1.0$	Original	$67.8$	$59.4$	$63.7$	$59.1$	$73.6$
$0.9$	FG-mmlu	$60.9$	$50.2$	$52.5$	$\mathbf{52.3}$	$\mathbf{73.2}$
	TC-sciq	$\mathbf{66.4}$	$56.9$	$\mathbf{60.6}$	$39.1$	$69.7$
	TC-medmcqa	$64.8$	$\mathbf{57.5}$	$59.2$	$33.8$	$66.0$
	TC-mmlu	$59.5$	$44.4$	$36.5$	$40.7$	$71.0$
$0.8$	Dobi-SVD (quant.)	$60.1$	–	–	–	–

Table 2: Benchmark accuracies as a function of the model density for the Targeted Compressor (TC-sciq, TC-mmlu and TC-medmcqa) compression approach. Bold marks the best result for each benchmark within each density block (across all methods shown). The FG-mmlu data are the same as in Tab. 1 .

Model	Params	MMLU	HellaSwag	WinoGrande	BoolQ	GSM8K	IFEval	MedMCQA	PPL
Original	70.55B	81.99	65.75	78.37	88.69	77.94	71.10	72.22	3.96
Block-pruned	50.02B	79.56	48.79	73.09	37.74	2.96	44.84	69.64	195.25
Pruned+QLoRA	50.02B	79.27	59.91	72.06	88.62	65.50	69.90	67.68	9.80
Pruned+QLoRA+FG	35.28B	63.10	46.29	66.38	86.36	2.81	48.44	40.86	24.36
Pruned+QLoRA+FG+SFT	35.28B	73.51	55.41	73.64	89.88	80.59	81.53	59.02	12.99

Table 3: Benchmark comparison for the Llama-3.3-70B-Instruct compression pipeline. GSM8K uses strict exact-match; IFEval [ 36 ] uses instruction-level strict accuracy; BoolQ [ 3 ] and the remaining accuracies use standard accuracy. Pruned layers: 40–43, 54–57, 60–75. Both the QLoRA recovery and the final SFT were performed for 22k iterations (1 epoch) with batch size 128.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

density	Dataset	MMLU			HellaSwag			WinoGrande			GSM8K			WikiText-2 PPL
density	Dataset	Unif	STRS	FG	Unif	STRS	FG	Unif	STRS	FG	Unif	STRS	FG	Unif	STRS	FG
$1.0$	Original	67.8			59.1			73.6			75.9			8.7
0.9	OH	54.8	56.2	59.3	47.3	44.7	50.6	68.7	68.3	70.2	25.1	0.0	41.3	21.7	36.9	15.5
	Tulu	53.0	55.6	59.1	47.3	44.0	50.2	68.3	66.9	70.2	30.9	23.8	46.3	23.4	41.7	16.5
	MMLU-train	54.3	55.5	60.9	48.7	46.6	52.3	71.1	70.8	73.2	3.6	0.0	18.3	23.0	40.2	15.5
	GT	55.9	57.5	61.7	48.1	45.4	51.8	69.7	69.9	71.6	19.7	0.0	33.7	22.6	42.8	16.5

Appendix

Table 4: Benchmark accuracy after compression with uniform SVD-LLM slicing (Unif), sensitivity-based rank allocation (STRS [ 33 ] ), and FermiGrad (FG) for the Llama-3.1-8B-Instruct model, across all four calibration datasets and target densities (fraction of parameters retained). PPL is the word-level perplexity on WikiText (lower is better). For each density and benchmark, the single best result across all three methods and all datasets is shown in bold. GSM8K uses strict-match exact-match; the remaining accuracies use standard accuracy.

Model Compression

Model compression reduces the storage, computation, or latency of machine-learning models while preserving task performance.

See all 124 papers

Advancing SVD-based LLM Compression via Layer-Wise Error Model Search
Oct 23, 2026Moritz Thoma, Maximilian Groezinger, Maximilian Forstenhäusler +8LLM Compression Low-Rank Compression
Technical University of Munich, Germany · BMW Group, Germany · University of Glasgow, United Kingdom +1
LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression
Jul 3, 2026Zhuowen Liu, Longkun Hao, Shiyu Feng +3Model Compression LLM Compression
Zhuowen Liu1 · Longkun Hao2 · Shiyu Feng3 +3
IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression
May 15, 2026Ali Abbasi, Chayne Thrash, Haoran Qin +2LLM Compression Low-Rank Matrix Decomposition
Vanderbilt University · University of California, Davis

Oct 23, 2026J/K move · Enter open · S save

Advancing SVD-based LLM Compression via Layer-Wise Error Model Search

Moritz Thoma, Maximilian Groezinger, Maximilian Forstenhäusler, Emad Aghajanzadeh+7

Technical University of Munich, Germany · BMW Group, Germany · University of Glasgow, United Kingdom+1

LLM Compression Low-Rank Compression Low-Rank Matrix Decomposition

Open paper AskPDF arXiv

Low-rank SVD-based compression offers a powerful strategy to reduce the computational costs of LLMs. However, existing methods face two key limitations: (i) global rank allocation, where uncalibrated error proxies fail to capture complex error propagation, and (ii) decomposition quality, where Fisher-based estimators suffer from severe rank collapse. In this work, we address these limitations by introducing Layer-wise Error Modeling Search (LEMS) and KFAC-SVD. LEMS advances rank allocation by introducing a layer-wise error surrogate that integrates local and global layer importance alongside a propagation bias, enabling effective global rank allocation via an ILP formulation. KFAC-SVD improves decomposition quality by utilizing token-wise statistics, mitigating the rank deficiency observed in prior Fisher-based SVD approaches. Across Mistral, Qwen3, and Llama 3 model families, we show that LEMS consistently outperforms existing search strategies, delivering significant zero-shot accuracy gains of up to 4.7 p.p. that generalize to model sizes of 70B parameters, while KFAC-SVD achieves an average perplexity improvement of 19%. Project page and code: https://lems-svd.github.io

Abstract

Model Compression

Explore similar work

Advancing SVD-based LLM Compression via Layer-Wise Error Model Search

LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression

IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression