Structured Multidimensional Representation Learning for Large Language Models

Alaa El Ichi; Khalide Jbilou; Mohamed El Guide; Franck Dufrenois

cs.CLMar 5, 2026

Structured Multidimensional Representation Learning for Large Language Models

Authors: Alaa El Ichi, Khalide Jbilou, Mohamed El Guide, Franck Dufrenois

Organizations: Universit´e du Littoral Cote d’Opale, LMPA, 50 rue F. Buisson, 62228 Calais-Cedex, France. · FGSES, University Mohammed VI Polytechnic, Rabat, Morocco · LISIC, 50 rue F. Buisson, Universit´e du Littoral Cote d’Opale, 62228 Calais-Cedex, France.

Abstract

Transformer architectures achieve state-of-the-art performance across a wide range of pattern recognition and natural language processing tasks, but their scaling is accompanied by substantial parameter growth and redundancy in the embedding dimension. In this work, we introduce a structured spectral factorization of the embedding space based on the L-product for third-order tensors. By reshaping token representations into spectral tensor slices and performing attention and feed-forward operations in the transform domain, we obtain a Tensor Transformer architecture that decomposes the encoder into p independent spectral sub-transformers while preserving standard Transformer semantics. We prove that the proposed L-Transformer is spectrally equivalent to p parallel Transformers operating on reduceddimensional embeddings, which yields approximately 1/p reduction (up to lower-order terms such as biases and normalization parameters) in encoder parameters under fixed total embedding size. When instantiated with a real-valued Discrete Cosine Transform (DCT), the method remains fully differentiable and compatible with existing training pipelines. Beyond compression, the spectral decomposition introduces an inductive bias over embedding frequencies, enabling slice-dependent frequency scaling that improves generalization. Experiments on IMDB and AGNews show that the proposed model can substantially reduce encoder parameters (up to 75% for p=4) while maintaining competitive accuracy. On IMDB, the tensorized encoder matches or improves upon the standard baseline under compression, whereas on AGNews at moderate width we observe a small accuracy decrease in exchange for a 4 times encoder reduction; at BERT-base width (d=768), performance returns to parity.

Figures & tables

Method family	What is structured	When applied	Exact vs. approximate	Primary gain / typical limitation
Low-rank PEFT (LoRA) [ 7 ]	Weight updates	Fine-tune	Approx. to full fine-tuning	Few trainable parameters; base encoder parameter count unchanged.
Post-training rank reduction [ 27 ]	Weights	Post hoc	Approx. modified model	No retraining; accuracy depends on layer selection and rank choice.
Tensorized weight compression (TRAWL; CP/Tucker/TT) [ 19 , 13 , 23 , 32 , 6 ]	Weights as tensor factors	Train or post hoc	Approx. factorization	Storage/parameters reduced; may affect optimization and accuracy.
Spectral mixing (FNet-like) [ 15 ]	Token mixing operator	Train	Exact transform (different layer type)	Global mixing at low parameter cost; not targeted at exact $1/p$ encoder scaling.
This work ( $\mathcal{L}$ -product tensorization) [ 9 ]	Representations and tensor operators	Train-time reparameterization	Exact slice-wise equivalence (transform domain)	Encoder parameters scale as $\approx 1/p$ ; attention-map storage and the $T^{2}d$ attention term remain unchanged; wall-clock gains require batched slice execution.

Table 1: Closest-method comparison and positioning of the proposed

\mathcal{L}

-product tensorization.

Panel of Figure 2 — Fig. 1: Tensorization of token embeddings. The matrix $X\in\mathbb{R}^{T\times d}$ is reshaped into $\mathcal{X}\in\mathbb{R}^{T\times d_{s}\times p}$ with $d_{s}=d/p$ by splitting the feature dimension into $p$ blocks of width $d_{s}$ . The third mode ( $p$ ) is the tube dimension used by the $\mathcal{L}$ -product; applying $\mathcal{L}$ along mode-3 produces $p$ transform-domain frontal slices $\widehat{X}^{(k)}\in\mathbb{R}^{T\times d_{s}}$ .

Operation	Standard	Tensor ( $\mathcal{L}$ -product)
$Q,K,V$ projections	$3Td^{2}$	$3T\,p\,d_{s}^{2}\;=\;3Td^{2}/p$
Attention scores ( $QK^{\top}$ )	$T^{2}d$	$p\,T^{2}d_{s}\;=\;T^{2}d$ (unchanged)
Attention $\times V$	$T^{2}d$	$T^{2}d$ (unchanged)
Output projection	$Td^{2}$	$Td^{2}/p$
FFN (two layers)	$8Td^{2}$	$8Td^{2}/p$
$\mathcal{L}$ -transform (DCT/IDCT)	—	$Td\log p$

Table 2: Per-layer FLOPs comparison. Projection and FFN costs drop by a factor

\approx p

, while attention score computation remains unchanged (it scales as

T^{2}

Component	Value
Encoder depth	$n_{\text{layers}}=4$
Tokenizer / vocabulary	BPE, 30,000 types
Max sequence length	$T=128$ (identical truncation/padding)

Table 3: Shared architecture settings across all experiments. Tokenization uses byte-pair encoding (BPE) [ 26 ] .

Setting	$d$	$H$	$d_{\mathrm{ff}}$
IMDB	128	4	512
AG News (moderate)	256	4	1024
AG News (BERT-width)	768	8	3072

Table 4: Task-specific architecture settings.

Component	Value
Optimizer	AdamW (lr $=3\times 10^{-4}$ , weight decay $=0.01$ )
Scheduler	OneCycleLR (10% linear warmup, cosine annealing to $10^{-5}$ ) [ 29 , 28 ]
Gradient clipping	$1.0$
Precision	Mixed precision (PyTorch 2.1, torch.cuda.amp ) [ 21 ]
Hardware	NVIDIA Tesla T4 (16 GB), CUDA 12.1
Reproducibility	torch.manual_seed , numpy.random.seed , random.seed ; cudnn.deterministic=True

Table 5: Shared training configuration across all experiments.

Setting	Epochs	Batch size
IMDB ( $d=128$ )	20	128
AG News ( $d=256$ )	5	128
AG News ( $d=768$ )	5	64

Table 6: Dataset-specific training schedules.

Fixed across all models	Varies (encoder only)
Embedding layer (vocabulary $\times d$ )	Encoder type (Std vs. $\mathcal{L}$ -product)
Classifier head (mean-pool + linear)	Decomposition factor $p$
Training recipe (optimizer, scheduler, hyperparameters)	Spectral weighting strategy ( pe )
Data protocol (sequence length $T$ , tokenizer, padding/truncation, seeds)	Encoder parameter count (and resulting compression ratio)

Table 7: Controlled comparison: fixed versus varying components.

Dataset	Model	Acc. (%)	Enc. params	Ratio
IMDB ( $d{=}128$ )	Std (4L, $d{=}128$ )	$80.77\pm 0.24$	991 K	$1.000\times$
	Std-1L (1L, $d{=}128$ )	$77.50\pm 0.28$	248 K	$0.250\times$
	T4-linear (4L, $d_{s}{=}32$ )	$\mathbf{81.65\pm 0.06}$	254 K	$0.256\times$
AG News ( $d{=}256$ )	Std (4L, $d{=}256$ )	$\mathbf{91.40\pm 0.31}$	3 949 K	$1.000\times$
	Std-1L (1L, $d{=}256$ )	$86.97\pm 0.15$	987 K	$0.250\times$
	T4-linear (4L, $d_{s}{=}64$ )	$90.76\pm 0.13$	1 000 K	$0.253\times$

Table 8: Parameter-matched comparison (mean

\pm

std over 3 seeds). Encoder parameters and ratio relative to Std.

Model	$p$	Acc. (%)	Encoder	Enc. Ratio
Std	—	$80.77\pm 0.24$	991 K	$1.000\times$
T4-standard	4	$\mathbf{82.02\pm 0.03}$	254 K	$0.256\times$
T4-learnable	4	$81.98\pm 0.20$	270 K	$0.273\times$
T4-linear	4	$81.65\pm 0.06$	254 K	$0.256\times$
T4-exponential	4	$81.60\pm 0.14$	254 K	$0.256\times$
T4-harmonic	4	$81.53\pm 0.13$	254 K	$0.256\times$

Table 9: IMDB results (

d{=}128

, 20 epochs, 3 seeds). Embedding (3.84 M) and classifier are shared. Best in bold .

Model	$p$	Acc. (%)	Encoder	Enc. Ratio
Std	—	$\mathbf{91.40\pm 0.31}$	3.95 M	$1.000\times$
T4-linear	4	$90.76\pm 0.13$	1.00 M	$0.253\times$
T4-harmonic	4	$90.41\pm 0.08$	1.00 M	$0.253\times$

Table 10: AG News results (

d{=}256

, 5 epochs, 3 seeds). Best in bold .

Fig. 2: Encoder parameters versus test accuracy on IMDB (left) and AG News (right). Error bars show

\pm 1

std over 3 seeds. Tensor models achieve competitive or superior accuracy while using approximately

4\times

fewer encoder parameters at

p=4

Fig. 3: PE strategy comparison on IMDB at

p=4

(mean

\pm

std, 3 seeds). The dashed line shows the full-width Std baseline. All strategies outperform Std, and the total spread across strategies is 0.49 pp.

Model	Acc. (%)	Encoder	Ratio	Mem
Std	$91.47\pm 0.09$	28.4 M	$1.00\times$	1.95 GB
T4-linear	$\mathbf{91.52\pm 0.21}$	7.1 M	$0.25\times$	1.66 GB

Table 11: AG News at

d=768

(BERT-base width),

p=4

, from scratch, 3 seeds. The tensor model compresses the encoder by

4\times

and reduces peak memory while maintaining accuracy.

Component	Std	T4-linear
Embedding	23.0 M (45%)	23.0 M (76%)
Encoder	28.4 M (55%)	7.1 M (24%)
Total	51.4 M	30.2 M ( $0.59\times$ )

Table 12: Parameter breakdown at

d=768

. Encoder compression becomes a model-level reduction because the encoder accounts for the majority of parameters.

Fig. 4: Accuracy gap (best T4 minus Std) as a function of model width

d

. The shaded band indicates a

\pm 0.3

pp parity region. At

d=768

, the tensor model reaches parity while compressing the encoder by

4\times

Config	Model	Enc. params	Ratio	Train tok/s	Epoch (s)	Peak mem	Batch
IMDB $d{=}128$	Std	991 K	$1.00\times$	123 K	26 s	0.50 GB	128 / 256
IMDB $d{=}128$	T4-linear	254 K	$0.26\times$	94 K	34 s	0.51 GB	128 / 256
AG News $d{=}256$	Std	3 949 K	$1.00\times$	177 K	87 s	0.99 GB	128 / 256
AG News $d{=}256$	T4-linear	1 000 K	$0.25\times$	132 K	116 s	0.96 GB	128 / 256
AG News $d{=}768$	Std	28 352 K	$1.00\times$	60 K	257 s	1.95 GB	64 / 128
AG News $d{=}768$	T4-linear	7 118 K	$0.25\times$	64 K	241 s	1.66 GB	64 / 128

Table 13: Efficiency comparison at equal depth (4 layers, mean over 3 seeds). Train throughput is measured per epoch; inference uses the full test set under torch.no_grad() with mixed precision.

Configuration	Std	Best T4	$\bm{\Delta}$	Enc. Saving
IMDB $d=128$	$80.77_{\pm 0.24}$	$82.02_{\pm 0.03}$ (std)	$+1.25$	991 K $\to$ 254 K
AG News $d=256$	$91.40_{\pm 0.31}$	$90.76_{\pm 0.13}$ (lin)	$-0.64$	3.95 M $\to$ 1.00 M
AG News $d=768$	$91.47_{\pm 0.09}$	$91.52_{\pm 0.21}$ (lin)	$+0.05$	28.4 M $\to$ 7.1 M

Table 14: Summary across all configurations (mean

\pm

std, 3 seeds).

\Delta

is the accuracy difference of the best tensor variant versus Std.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 19 — Fig. 5: Slice-wise equivalence of the tensor Transformer under the $\mathcal{L}$ -product framework. The tensor is transformed along mode-3, processed independently slice-by-slice by $p$ compact Transformers (dimension $d_{s}$ ), stacked back, and mapped to the original domain via $\mathcal{L}^{-1}$ .

Fig. 6: Training curves on IMDB (

d=128

, seed 42).

Fig. 7: Training curves on AG News (

d=256

, seed 42).

PE strategy	IMDB $d=128$	AG News $d=256$
T4-standard ( $\alpha_{k}=1$ )	$\mathbf{82.02\pm 0.03}$	—
T4-linear ( $\alpha_{k}=k/p$ )	$81.65\pm 0.06$	$\mathbf{90.76\pm 0.13}$
T4-exponential ( $\alpha_{k}=2^{(k-1)/(p-1)}$ )	$81.60\pm 0.14$	—
T4-harmonic ( $\alpha_{k}=k$ )	$81.53\pm 0.13$	$90.41\pm 0.08$
T4-learnable	$81.98\pm 0.20$	—
Std (baseline)	$80.77\pm 0.24$	$91.40\pm 0.31$

Appendix

Table 15: Full PE sweep at

p=4

(mean

\pm

std, 3 seeds). Best per dataset in bold .

Config	Model	Enc.	Train (s/ep)	Mem (GB)	Infer (s)
IMDB ( $d=128$ )	Std	991 K	26	0.50	11.8
	T4-standard	254 K	34	0.51	12.7
	T4-linear	254 K	34	0.51	12.6
	T4-harmonic	254 K	34	0.51	12.8
	T4-learnable	270 K	34	0.51	12.8
	T2-linear	500 K	29	0.49	11.9

Appendix

Table 16: Runtime details. Train is mean wall-clock per epoch; Mem is peak GPU allocation; Infer is forward pass on the full test set. All runs on NVIDIA Tesla T4 with mixed precision.

Transformer

Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.

See all 540 papers

Aug 31, 2026

Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability

Large language models (LLMs) are built from structured high-dimensional objects such as token representations, weights, adaptation updates, caches, and activations, whose multilinear structure is underexploited by the conventional matrix-centric view. Tensor decompositions and tensor networks provide a principled algebraic language for this structure, yet the literature often treats them as isolated compression mechanisms. This survey organizes tensor methods for LLMs through two complementary views: a seven-stage lifecycle taxonomy covering tokenization, embeddings, pre-training, adaptation, compression, inference, and interpretability, and a component view covering embeddings, attention, and feed-forward networks. We provide unified notation and theoretical foundations, analyze tensorization strategies for individual Transformer components, and compare methods at each lifecycle stage while making differences in evaluation protocols and model scales explicit. We further connect tensor methods to neighboring efficiency techniques and probabilistic tensor networks. Finally, we synthesize open challenges and introduce

ρ_{\rm gap}

, a metric for the compression-realization gap between theoretical memory reduction and measured system-level speedup. By treating tensorization as a common structural principle, the survey provides a structured entry point to tensorized language models and clarifies when parameter savings can plausibly translate into memory efficiency, computational efficiency, or interpretability. The GitHub page dedicated to this paper is accessible at this https URL.

Matvei Tarasov, Salman Ahmadi-Asl, Andre L. F. de Almeida +1

Innopolis University, Innopolis 420500, Russia · Department of Teleinformatics Engineering, Federal University of Cear´a, Fortaleza 60455-760, Brazil · Systems Research Institute of Polish Academy of Science and Warsaw University of Technology, Poland

LLM Compression LLM Inference Acceleration Language Modeling

Ask PDF

Jun 2, 2026

Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression

Post-training compression is essential for deploying large language models (LLMs) under tight resource constraints. Tensor decompositions have emerged as a promising direction, offering compact parameterizations well suited to Transformer weight structures. However, existing studies evaluate these methods in narrow settings, leaving unclear whether tensorization is effective at large-scale deployment. We systematically evaluate tensor compression across dense and MoE architectures, establishing performance trade-offs grounded in both empirical analysis and theoretical analysis. We identify a fundamental mismatch between the shared subspaces assumed by tensor decompositions and the heterogeneous representations learned by modern LLMs, thereby delineating their practical limits and clarifying their viable role in large-scale deployment. The code is available at https://github.com/brain-lab-research/TT-LLM.

Artur Zagitov, Alexander Miasnikov, Maxim Krutikov +5

BRAIn Lab, Moscow, Russia

LLM Compression Tensor Decomposition LLM Post-Training

Ask PDF

May 25, 2026

A general tensor-structured compression scheme for efficient large language models

Large language models (LLMs) are dominated by dense linear transformations, whose storage, memory and computational overheads hinder efficient adaptation and deployment while masking the functional impacts of structural simplification. Here we present Tensor Mixture (MixT), a general tensor-structured compression scheme that replaces targeted dense linear layers with natively executable mixtures of tensor operators. Operating directly on generic linear projections instead of model-specific components, MixT is potentially applicable across Transformer-based LLMs and other dense neural mappings. We evaluate MixT on Qwen3-8B and LLaMA2-7B under a unified recovery protocol, identifying a broad compressible regime in which MMLU accuracy is largely preserved before an abrupt transition at model-specific boundaries. This transition coincides with coordinated shifts in output entropy, prediction entropy and inter-layer geometry. At the LLaMA2-7B transition boundary, MixT reduces full-model parameters by 47.5%, inference FLOPs by 37.1%, training FLOPs by 52.1% and peak inference memory by 60.4%, demonstrating its practical potential for lower-cost LLM compression.

Ying Lu, Peng-Fei Zhou, Qi-Xuan Fang +3

School of Physical Sciences, University of Chinese Academy of Sciences, Beijing, 100049, China. · Kavli Institute for Theoretical Sciences, University of Chinese Academy of Sciences, Beijing, 100190, China. · Center for Quantum Physics and Intelligent Sciences, Department of Physics, Capital Normal University, Beijing, 100048, China. +1

LLM Inference Efficiency LLM Compression Neural Network Compression

Ask PDF

Abstract

Transformer

Explore similar work

Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability

Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression

A general tensor-structured compression scheme for efficient large language models