Compressing Transformer Language Models via Matrix Product Operator Decomposition: A Case Study on PicoGPT

Younes Javanmard; Tanmoy Pandit; Masoud Mardani

cs.CLMar 30, 2026

Compressing Transformer Language Models via Matrix Product Operator Decomposition: A Case Study on PicoGPT

Authors: Younes Javanmard, Tanmoy Pandit, Masoud Mardani

Organizations: Institut für Theoretische Physik, Leibniz Universität Hannover, Appelstraße 2, 30167 Hannover, Germany · VTT Technical Research Centre of Finland, Tietotie 3, Espoo, Finland · National High Magnetic Field Laboratory, Tallahassee, FL 32310, USA

Abstract

Transformer-based language models achieve strong performance across NLP tasks, but their quadratic parameter scaling with hidden dimension makes deployment on resource-constrained hardware expensive. We study Matrix Product Operator (MPO) decomposition as a principled compression method for transformers. MPO factorises weight matrices into chains of low-rank cores, with approximation quality controlled by the bond dimension chi. We replace every nn.Linear layer in PicoGPT, a GPT-2-style character-level language model with about 1M parameters, with an MPOLinear module parameterised as an MPO chain. Cores are initialised either by TT-SVD from pretrained dense weights or from random initialisation, and trained using standard PyTorch autograd without a custom backward pass. We derive balanced factorisation schemes for the five distinct weight shapes in PicoGPT and evaluate bond dimensions chi in {4, 8, 16, 32} on Tiny Shakespeare. MPO compression achieves up to 13x compression per transformer block at chi = 4. At chi = 16, the model uses 191,872 parameters instead of 1,020,224 while retaining 97.7% of baseline token accuracy (51.6% vs 52.8%). Reconstruction error follows the expected trend and is lower for three-site than two-site factorisations at the same bond dimension. The chi = 8 model gives the best accuracy per parameter, exceeding the dense baseline by 2.7x on this metric. These results show that MPO parameterisation is a practical and theoretically grounded alternative to low-rank methods and unstructured pruning for transformer compression.

Figures & tables

Parameter	Description	Value
$V$	Vocabulary size (character-level)	65
$D$	Embedding dimension ( $d_{\text{model}}$ )	128
$H$	Attention heads	4
$N$	Transformer layers	4
$T$	Context length	256
$d_{\text{ff}}$	FFN hidden dimension ( $4D$ )	512

Table 1: PicoGPT default hyperparameters.

Layer	out	in	$L$	$(d_{l}^{\text{out}})$	$(d_{l}^{\text{in}})$	Dense	MPO ( $\chi=8$ )
$\mathbf{W}_{Q/K/V/O}$	128	128	2	$[8,16]$	$[8,16]$	16 384	2 560
$\mathbf{W}_{1}$ (up)	512	128	3	$[8,8,8]$	$[4,4,8]$	65 536	4 864
$\mathbf{W}_{2}$ (down)	128	512	3	$[4,4,8]$	$[8,8,8]$	65 536	2 816
$\mathbf{W}_{\text{LM}}$	65	128	2	$[5,13]$	$[8,16]$	8 320	1 984
Total per block (4 attn + 2 FFN)						196 608	17 920
Compression ratio per block						$1\times$	11.0 $\times$

Table 2: MPO factorisation schemes for each PicoGPT linear layer.

\chi

denotes the bond dimension.

Panel of Figure 1 — Figure 1: Tensor network diagram of an MPO weight matrix. Horizontal lines carry virtual (bond) indices of dimension $\chi$ ; vertical lines are physical indices of size $d_{l}^{\mathrm{out}}$ (blue, upward) and $d_{l}^{\mathrm{in}}$ (red, downward). Contracting all bond indices reconstructs the full weight $\widehat{\mathbf{W}}\in\mathbb{R}^{\mathrm{out}\times\mathrm{in}}$ .

Model	Parameters	Ratio	Max rel. err.	Mean rel. err.
Dense	1 020 224	1.0 $\times$	—	—
MPO $\chi=4$	78 336	13.0 $\times$	0.421	0.337
MPO $\chi=8$	110 592	9.2 $\times$	0.283	0.216
MPO $\chi=16$	191 872	5.3 $\times$	0.148	0.107
MPO $\chi=32$	408 832	2.5 $\times$	0.072	0.051

Table 3: Parameter counts and compression ratios versus the dense baseline. Layer-wise reconstruction errors are relative Frobenius norms

\|\mathbf{W}-\widehat{\mathbf{W}}\|_{F}/\|\mathbf{W}\|_{F}

averaged over all linear layers of a randomly initialised dense model.

Figure 2: Per-layer MPO reconstruction error versus bond dimension. Three-site decompositions (

L=3

) achieve lower error per parameter than two-site ones (

L=2

) across the range of

\chi

considered here.

Panel of Figure 3 — Figure 3: Training (left) and validation (right) cross-entropy loss as a function of training step. Higher bond dimensions converge faster and to lower loss values in the train-from-scratch setting studied here.

Panel of Figure 4 — Figure 4: Left : Validation token accuracy during training for all models. Right : Pareto frontier of final validation accuracy versus parameter count. MPO $\chi=16$ retains $97.7\%$ of the dense accuracy at $5.3\times$ parameter compression.

Model	Params	Ratio	Val loss $\downarrow$	Val acc $\uparrow$	Acc. gap $\downarrow$	Per-param acc.
Dense	1 020 224	1.0 $\times$	1.860	52.8%	—	$1.65\times 10^{-3}$
MPO $\chi=4$	78 336	13.0 $\times$	2.220	36.8%	-16.0	$4.15\times 10^{-3}$
MPO $\chi=8$	110 592	9.2 $\times$	1.950	46.8%	-6.0	$4.45\times 10^{-3}$
MPO $\chi=16$	191 872	5.3 $\times$	1.850	51.6%	-1.2	$3.72\times 10^{-3}$
MPO $\chi=32$	408 832	2.5 $\times$	1.850	52.4%	-0.4	$2.59\times 10^{-3}$

Table 4: Final validation metrics after 2 000 training steps. Acc. gap is the absolute reduction in token accuracy versus the dense model. Per-param acc. is the heuristic score

\text{acc}/\sqrt{N}

used here as a rough proxy for parameter efficiency.

Model Compression

Model compression reduces the storage, computation, or latency of machine-learning models while preserving task performance.

See all 118 papers

Apr 4, 2026

AutoCompress: Critical Layer Isolation for Efficient Transformer Compression

We present AutoCompress, a transformer compression method motivated by an empirical finding: in small transformers, Layer 0 carries disproportionately high task-critical information, with an NTK-based importance score of 3.6 compared to a maximum of 0.054 for all other layers -- a gap of over 60x. Based on this finding, we propose Critical Layer Isolation (CLI), an architecture that protects Layer 0 at full dimensionality, compresses all intermediate layers through a learned bottleneck, and restores the full dimension at the final layer. Applied to GPT-2 Medium (354.8M parameters), CLI-GPT2 achieves 204.5 perplexity on WikiText-103 with only 143.8M parameters -- a 2.47x compression ratio and 59.5% parameter reduction. Crucially, an ablation study demonstrates that a uniform bottleneck baseline of comparable size achieves only 571.8 perplexity under identical training conditions, confirming that the architectural decision to protect Layer 0 -- rather than simply reducing model size -- is the primary driver of performance. Code and checkpoints are publicly available.

Archit Thorat

Independent Researcher

Neural Network Compression LLM Compression

Ask PDF

Apr 20, 2026

Predicting LLM Compression Degradation from Spectral Statistics

Matrix-level low-rank compression is a promising way to reduce the cost of large language models, but running compression and evaluating the resulting models on language tasks can be prohibitively expensive. Can compression-induced degradation be predicted before committing to this compute? We systematically analyze the Qwen3 and Gemma3 model families across four representative low-rank compression methods: vanilla SVD, two ASVD variants, and SVD-LLM. We find that stable rank and information density, measured in bits per parameter, dominate performance degradation. The interaction term

γ\cdot \barρ_s

, defined as compression ratio times stable rank, is a robust predictor of accuracy degradation, achieving leave-one-out cross-validation Pearson correlations of

0.890

for attention layers and

0.839

for MLP layers. We provide theoretical intuition for why this predictor succeeds by connecting it to standard SVD truncation bounds and error composition mechanisms in transformer layers. These findings enable a predict-then-compress workflow: compute

γ\cdot \barρ_s

from weights, estimate degradation, and invest compute only in desirable configurations.

Mingxue Xu

Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom

LLM Compression Low-Rank Matrix Decomposition Low-Rank Compression Low-Rank Approximation Spectral Methods

Ask PDF

Apr 6, 2026

Training Transformers in Cosine Coefficient Space

Linear layers hold most of a transformer's parameters. We replace each linear layer with one that stores

K

out of

mn

two-dimensional DCT coefficients per weight matrix and reconstructs the full matrix through an inverse DCT at every forward pass; the

K

coefficients are the trainable parameters. A 4-layer, 128-dim transformer trained from scratch on character-level Shakespeare reaches validation loss

1.604

K = mn/2

, against

1.580

for a standard dense baseline -- a gap of

+0.024

at half the trainable parameter count, within the terminal-epoch variation of the dense run. A rank-48 LoRA factorization at the same trainable parameter count reaches only

1.801

(

+0.221

). The structural advantage of sparse-coefficient over low-rank parameterizations at matched

K

is qualitative. We identify rank flexibility as the mechanism. A random orthonormal basis matches the DCT within noise at

K = mn/2

, and a compression sweep through

K = mn/10

and

K = mn/20

shows that subspaces that can host high-rank matrices keep the loss low, while subspaces that flatten into a low-rank block (zigzag-selection variants) converge onto the observed stable rank and the loss line of the rank-48 LoRA reference in lock-step. Among these orthonormal bases, the DCT is preferred because its separable fast transform admits a fused reconstruction kernel: the materialized weight matrix never leaves on-chip memory, so the parameter saving translates into a bandwidth saving as well.

Mohamed Amine Bergach

Illumina, San Diego, CA, USA

Transformer Neural Network Compression

Ask PDF

Abstract

Model Compression

Explore similar work

AutoCompress: Critical Layer Isolation for Efficient Transformer Compression

Predicting LLM Compression Degradation from Spectral Statistics

Training Transformers in Cosine Coefficient Space