Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retraction

Björn Roman Kohlberger

cs.LGApr 1, 2026

Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retraction

Authors: Björn Roman Kohlberger

Organizations: EctoSpace, Dublin, Ireland

Abstract

The memory wall remains the primary bottleneck for training large language models on consumer hardware. We introduce Spectral Compact Training (SCT), a method that replaces dense weight matrices with permanent truncated SVD factors W = U diag(s) V^T, where the full dense matrix is never materialized during training or inference. Gradients flow through the compact spectral factors via standard backpropagation, and U, V are retracted to the Stiefel manifold via QR decomposition after each optimizer step. SCT achieves up to 199x memory reduction per MLP layer at rank 32, enabling full training steps of 70B-parameter architectures on a Steam Deck handheld (7.2 GB peak memory vs. 1,245 GB for dense FP32 training with Adam). Rank-sweep experiments on SmolLM2-1.7B (ranks 32-256, 2000 steps, NVIDIA A100) show that all tested ranks converge to the same loss floor (~4.2-4.5), identifying the learning rate schedule -- not MLP rank -- as the primary bottleneck. Rank 128 emerges as the efficiency sweet spot at 11.7x MLP compression with the lowest perplexity. GPU memory drops 46% at rank 32 while training throughput doubles.

Figures & tables

Model	Layer ( $m\times n$ )	Dense+Adam	SCT ( $k\!=\!32$ )	Compression
SmolLM2-135M	$576\times 1536$	14.2 MB	1.1 MB	13 $\times$
SmolLM2-360M	$1024\times 4096$	67.1 MB	2.6 MB	26 $\times$
SmolLM2-1.7B	$2048\times 8192$	268.4 MB	5.2 MB	51 $\times$
LLaMA-7B	$4096\times 11008$	721.4 MB	7.7 MB	93 $\times$
Qwen-27B	$4096\times 17408$	1,141 MB	11.0 MB	104 $\times$
LLaMA-70B	$8192\times 28672$	3,758 MB	18.9 MB	199 $\times$

Table 1: Per-MLP-layer training memory (weights + gradients + Adam states) at rank 32.

Metric	Apple M4 Pro (48 GB)	Steam Deck (16 GB)
Peak Memory	7,907 MB	7,236 MB
Forward Pass	0.08 s	0.43 s
Backward Pass	0.09 s	0.92 s
Optimizer Step	0.22 s	2.35 s
QR Retraction	3.02 s	2.58 s
Total Step	3.41 s	6.28 s

Table 2: 70B architecture validation on consumer hardware. Both platforms complete a full training step under 8 GB.

Figure 1: Training memory at 70B scale. SCT requires 172

\times

less memory than dense training.

Method	Params	MLP Comp.	Loss	PPL	GPU Mem.	Step Time
Dense	1,711M	1.0 $\times$	1.29	3.6	35.5 GB	1.17 s
SCT $r\!=\!256$	692M	5.9 $\times$	4.33	75.6	21.3 GB	1.05 s
SCT $r\!=\!128$	598M	11.7 $\times$	4.18	65.6	20.0 GB	0.74 s
SCT $r\!=\!64$	551M	23.5 $\times$	4.34	76.7	19.3 GB	0.62 s
SCT $r\!=\!32$	527M	46.9 $\times$	4.47	86.9	19.0 GB	0.56 s

Table 3: Rank sweep results. Loss and PPL are smoothed (window=50). Rank 128 (bold) achieves the best PPL among SCT configurations.

Figure 2: Loss convergence for all ranks. All SCT configurations converge to the same loss floor (

\sim

4.2–4.5) regardless of rank. Dense converges to 1.29.

Figure 3: Left: Compression vs. quality Pareto frontier. Rank 128 achieves the best PPL at 11.7

\times

compression. Right: GPU memory by method. Even rank 256 saves 40% of VRAM.

Method	Final Loss	Final PPL	Trainable Params	PPL Ratio
Dense + AdamW	0.235	1.3	134,515,008	1.0 $\times$
SCT (95% energy)	0.594	1.8	84,333,271	1.38 $\times$

Table 4: SmolLM2-135M fine-tuning (gradient integrity test). The 135M model is below the optimal scale for SCT compression; this test validates gradient flow, not compression utility.

LLM Training

Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.

See all 342 papers

SPQ: An Ensemble Technique for Large Language Model Compression
Feb 20, 2026Jiamin Yao, Eren GultepeLLM Compression LLM Quantization
Dept. of Computer Science, Southern Illinois University Edwardsville, Edwardsville, USA
Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression
Apr 2, 2026Ruoling Qi, Yirui Liu, Xuaner Wu +6LLM Compression Low-Rank Compression
Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Jiao Tong University · University of Maryland +2
SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression
Jun 22, 2026Mahmoud Safari, Frank HutterLLM Pruning LLM Compression
University of Freiburg · Prior Labs · ELLIS Institute Tübingen

Feb 20, 2026J/K move · Enter open · S save

SPQ: An Ensemble Technique for Large Language Model Compression

Jiamin Yao, Eren Gultepe

Dept. of Computer Science, Southern Illinois University Edwardsville, Edwardsville, USA

LLM Compression LLM Quantization LLM Pruning LLM Inference Acceleration Post-Training Quantization

Open paper AskPDF arXiv

This study presents an ensemble technique, SPQ (SVD-Pruning-Quantization), for large language model (LLM) compression that combines variance-retained singular value decomposition (SVD), activation-based pruning, and post-training linear quantization. Each component targets a different source of inefficiency: i) pruning removes redundant neurons in MLP layers, ii) SVD reduces attention projections into compact low-rank factors, iii) and 8-bit quantization uniformly compresses all linear layers. At matched compression ratios, SPQ outperforms individual methods (SVD-only, pruning-only, or quantization-only) in perplexity, demonstrating the benefit of combining complementary techniques. Applied to LLaMA-2-7B, SPQ achieves up to 75% memory reduction while maintaining or improving perplexity (e.g., WikiText-2 5.47 to 4.91) and preserving accuracy on downstream benchmarks such as C4, TruthfulQA, and GSM8K. Compared to strong baselines like GPTQ and SparseGPT, SPQ offers competitive perplexity and accuracy while using less memory (6.86 GB vs. 7.16 GB for GPTQ). Moreover, SPQ improves inference throughput over GPTQ, achieving up to a 1.9x speedup, which further enhances its practicality for real-world deployment. The effectiveness of SPQ's robust compression through layer-aware and complementary compression techniques may provide practical deployment of LLMs in memory-constrained environments. Code is available at: https://github.com/JiaminYao/SPQ_LLM_Compression/

Abstract

LLM Training

Explore similar work

SPQ: An Ensemble Technique for Large Language Model Compression

Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression

SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression