Compressing Transformer Language Models via Matrix Product Operator Decomposition: A Case Study on PicoGPT

Younes Javanmard; Tanmoy Pandit; Masoud Mardani

cs.CLMar 30, 2026

Compressing Transformer Language Models via Matrix Product Operator Decomposition: A Case Study on PicoGPT

Authors: Younes Javanmard, Tanmoy Pandit, Masoud Mardani

Organizations: Institut für Theoretische Physik, Leibniz Universität Hannover, Appelstraße 2, 30167 Hannover, Germany · VTT Technical Research Centre of Finland, Tietotie 3, Espoo, Finland · National High Magnetic Field Laboratory, Tallahassee, FL 32310, USA

Abstract

Transformer-based language models achieve strong performance across NLP tasks, but their quadratic parameter scaling with hidden dimension makes deployment on resource-constrained hardware expensive. We study Matrix Product Operator (MPO) decomposition as a principled compression method for transformers. MPO factorises weight matrices into chains of low-rank cores, with approximation quality controlled by the bond dimension chi. We replace every nn.Linear layer in PicoGPT, a GPT-2-style character-level language model with about 1M parameters, with an MPOLinear module parameterised as an MPO chain. Cores are initialised either by TT-SVD from pretrained dense weights or from random initialisation, and trained using standard PyTorch autograd without a custom backward pass. We derive balanced factorisation schemes for the five distinct weight shapes in PicoGPT and evaluate bond dimensions chi in {4, 8, 16, 32} on Tiny Shakespeare. MPO compression achieves up to 13x compression per transformer block at chi = 4. At chi = 16, the model uses 191,872 parameters instead of 1,020,224 while retaining 97.7% of baseline token accuracy (51.6% vs 52.8%). Reconstruction error follows the expected trend and is lower for three-site than two-site factorisations at the same bond dimension. The chi = 8 model gives the best accuracy per parameter, exceeding the dense baseline by 2.7x on this metric. These results show that MPO parameterisation is a practical and theoretically grounded alternative to low-rank methods and unstructured pruning for transformer compression.

Figures & tables

Parameter	Description	Value
$V$	Vocabulary size (character-level)	65
$D$	Embedding dimension ( $d_{\text{model}}$ )	128
$H$	Attention heads	4
$N$	Transformer layers	4
$T$	Context length	256
$d_{\text{ff}}$	FFN hidden dimension ( $4D$ )	512

Table 1: PicoGPT default hyperparameters.

Layer	out	in	$L$	$(d_{l}^{\text{out}})$	$(d_{l}^{\text{in}})$	Dense	MPO ( $\chi=8$ )
$\mathbf{W}_{Q/K/V/O}$	128	128	2	$[8,16]$	$[8,16]$	16 384	2 560
$\mathbf{W}_{1}$ (up)	512	128	3	$[8,8,8]$	$[4,4,8]$	65 536	4 864
$\mathbf{W}_{2}$ (down)	128	512	3	$[4,4,8]$	$[8,8,8]$	65 536	2 816
$\mathbf{W}_{\text{LM}}$	65	128	2	$[5,13]$	$[8,16]$	8 320	1 984
Total per block (4 attn + 2 FFN)						196 608	17 920
Compression ratio per block						$1\times$	11.0 $\times$

Table 2: MPO factorisation schemes for each PicoGPT linear layer.

\chi

denotes the bond dimension.

Panel of Figure 1 — Figure 1: Tensor network diagram of an MPO weight matrix. Horizontal lines carry virtual (bond) indices of dimension $\chi$ ; vertical lines are physical indices of size $d_{l}^{\mathrm{out}}$ (blue, upward) and $d_{l}^{\mathrm{in}}$ (red, downward). Contracting all bond indices reconstructs the full weight $\widehat{\mathbf{W}}\in\mathbb{R}^{\mathrm{out}\times\mathrm{in}}$ .

Model	Parameters	Ratio	Max rel. err.	Mean rel. err.
Dense	1 020 224	1.0 $\times$	—	—
MPO $\chi=4$	78 336	13.0 $\times$	0.421	0.337
MPO $\chi=8$	110 592	9.2 $\times$	0.283	0.216
MPO $\chi=16$	191 872	5.3 $\times$	0.148	0.107
MPO $\chi=32$	408 832	2.5 $\times$	0.072	0.051

Table 3: Parameter counts and compression ratios versus the dense baseline. Layer-wise reconstruction errors are relative Frobenius norms

\|\mathbf{W}-\widehat{\mathbf{W}}\|_{F}/\|\mathbf{W}\|_{F}

averaged over all linear layers of a randomly initialised dense model.

Figure 2: Per-layer MPO reconstruction error versus bond dimension. Three-site decompositions (

L=3

) achieve lower error per parameter than two-site ones (

L=2

) across the range of

\chi

considered here.

Panel of Figure 3 — Figure 3: Training (left) and validation (right) cross-entropy loss as a function of training step. Higher bond dimensions converge faster and to lower loss values in the train-from-scratch setting studied here.

Panel of Figure 4 — Figure 4: Left : Validation token accuracy during training for all models. Right : Pareto frontier of final validation accuracy versus parameter count. MPO $\chi=16$ retains $97.7\%$ of the dense accuracy at $5.3\times$ parameter compression.

Model	Params	Ratio	Val loss $\downarrow$	Val acc $\uparrow$	Acc. gap $\downarrow$	Per-param acc.
Dense	1 020 224	1.0 $\times$	1.860	52.8%	—	$1.65\times 10^{-3}$
MPO $\chi=4$	78 336	13.0 $\times$	2.220	36.8%	-16.0	$4.15\times 10^{-3}$
MPO $\chi=8$	110 592	9.2 $\times$	1.950	46.8%	-6.0	$4.45\times 10^{-3}$
MPO $\chi=16$	191 872	5.3 $\times$	1.850	51.6%	-1.2	$3.72\times 10^{-3}$
MPO $\chi=32$	408 832	2.5 $\times$	1.850	52.4%	-0.4	$2.59\times 10^{-3}$

Table 4: Final validation metrics after 2 000 training steps. Acc. gap is the absolute reduction in token accuracy versus the dense model. Per-param acc. is the heuristic score

\text{acc}/\sqrt{N}

used here as a rough proxy for parameter efficiency.

Model Compression

Model compression reduces the storage, computation, or latency of machine-learning models while preserving task performance.

See all 118 papers

AutoCompress: Critical Layer Isolation for Efficient Transformer Compression
Apr 4, 2026Archit ThoratNeural Network Compression LLM Compression
Independent Researcher
Predicting LLM Compression Degradation from Spectral Statistics
Apr 20, 2026Mingxue XuLLM Compression Low-Rank Matrix Decomposition
Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom
Training Transformers in Cosine Coefficient Space
Apr 6, 2026Mohamed Amine BergachTransformer Neural Network Compression
Illumina, San Diego, CA, USA

Apr 4, 2026J/K move · Enter open · S save

AutoCompress: Critical Layer Isolation for Efficient Transformer Compression

Archit Thorat

Independent Researcher

Neural Network Compression LLM Compression

Open paper AskPDF arXiv

We present AutoCompress, a transformer compression method motivated by an empirical finding: in small transformers, Layer 0 carries disproportionately high task-critical information, with an NTK-based importance score of 3.6 compared to a maximum of 0.054 for all other layers -- a gap of over 60x. Based on this finding, we propose Critical Layer Isolation (CLI), an architecture that protects Layer 0 at full dimensionality, compresses all intermediate layers through a learned bottleneck, and restores the full dimension at the final layer. Applied to GPT-2 Medium (354.8M parameters), CLI-GPT2 achieves 204.5 perplexity on WikiText-103 with only 143.8M parameters -- a 2.47x compression ratio and 59.5% parameter reduction. Crucially, an ablation study demonstrates that a uniform bottleneck baseline of comparable size achieves only 571.8 perplexity under identical training conditions, confirming that the architectural decision to protect Layer 0 -- rather than simply reducing model size -- is the primary driver of performance. Code and checkpoints are publicly available.

Abstract

Model Compression

Explore similar work

AutoCompress: Critical Layer Isolation for Efficient Transformer Compression

Predicting LLM Compression Degradation from Spectral Statistics

Training Transformers in Cosine Coefficient Space