Structured Multidimensional Representation Learning for Large Language Models

Alaa El Ichi; Khalide Jbilou; Mohamed El Guide; Franck Dufrenois

cs.CLMar 5, 2026

Structured Multidimensional Representation Learning for Large Language Models

Authors: Alaa El Ichi, Khalide Jbilou, Mohamed El Guide, Franck Dufrenois

Organizations: Universit´e du Littoral Cote d’Opale, LMPA, 50 rue F. Buisson, 62228 Calais-Cedex, France. · FGSES, University Mohammed VI Polytechnic, Rabat, Morocco · LISIC, 50 rue F. Buisson, Universit´e du Littoral Cote d’Opale, 62228 Calais-Cedex, France.

Abstract

Transformer architectures achieve state-of-the-art performance across a wide range of pattern recognition and natural language processing tasks, but their scaling is accompanied by substantial parameter growth and redundancy in the embedding dimension. In this work, we introduce a structured spectral factorization of the embedding space based on the L-product for third-order tensors. By reshaping token representations into spectral tensor slices and performing attention and feed-forward operations in the transform domain, we obtain a Tensor Transformer architecture that decomposes the encoder into p independent spectral sub-transformers while preserving standard Transformer semantics. We prove that the proposed L-Transformer is spectrally equivalent to p parallel Transformers operating on reduceddimensional embeddings, which yields approximately 1/p reduction (up to lower-order terms such as biases and normalization parameters) in encoder parameters under fixed total embedding size. When instantiated with a real-valued Discrete Cosine Transform (DCT), the method remains fully differentiable and compatible with existing training pipelines. Beyond compression, the spectral decomposition introduces an inductive bias over embedding frequencies, enabling slice-dependent frequency scaling that improves generalization. Experiments on IMDB and AGNews show that the proposed model can substantially reduce encoder parameters (up to 75% for p=4) while maintaining competitive accuracy. On IMDB, the tensorized encoder matches or improves upon the standard baseline under compression, whereas on AGNews at moderate width we observe a small accuracy decrease in exchange for a 4 times encoder reduction; at BERT-base width (d=768), performance returns to parity.

Figures & tables

Method family	What is structured	When applied	Exact vs. approximate	Primary gain / typical limitation
Low-rank PEFT (LoRA) [ 7 ]	Weight updates	Fine-tune	Approx. to full fine-tuning	Few trainable parameters; base encoder parameter count unchanged.
Post-training rank reduction [ 27 ]	Weights	Post hoc	Approx. modified model	No retraining; accuracy depends on layer selection and rank choice.
Tensorized weight compression (TRAWL; CP/Tucker/TT) [ 19 , 13 , 23 , 32 , 6 ]	Weights as tensor factors	Train or post hoc	Approx. factorization	Storage/parameters reduced; may affect optimization and accuracy.
Spectral mixing (FNet-like) [ 15 ]	Token mixing operator	Train	Exact transform (different layer type)	Global mixing at low parameter cost; not targeted at exact $1/p$ encoder scaling.
This work ( $\mathcal{L}$ -product tensorization) [ 9 ]	Representations and tensor operators	Train-time reparameterization	Exact slice-wise equivalence (transform domain)	Encoder parameters scale as $\approx 1/p$ ; attention-map storage and the $T^{2}d$ attention term remain unchanged; wall-clock gains require batched slice execution.

Table 1: Closest-method comparison and positioning of the proposed

\mathcal{L}

-product tensorization.

Panel of Figure 2 — Fig. 1: Tensorization of token embeddings. The matrix $X\in\mathbb{R}^{T\times d}$ is reshaped into $\mathcal{X}\in\mathbb{R}^{T\times d_{s}\times p}$ with $d_{s}=d/p$ by splitting the feature dimension into $p$ blocks of width $d_{s}$ . The third mode ( $p$ ) is the tube dimension used by the $\mathcal{L}$ -product; applying $\mathcal{L}$ along mode-3 produces $p$ transform-domain frontal slices $\widehat{X}^{(k)}\in\mathbb{R}^{T\times d_{s}}$ .

Operation	Standard	Tensor ( $\mathcal{L}$ -product)
$Q,K,V$ projections	$3Td^{2}$	$3T\,p\,d_{s}^{2}\;=\;3Td^{2}/p$
Attention scores ( $QK^{\top}$ )	$T^{2}d$	$p\,T^{2}d_{s}\;=\;T^{2}d$ (unchanged)
Attention $\times V$	$T^{2}d$	$T^{2}d$ (unchanged)
Output projection	$Td^{2}$	$Td^{2}/p$
FFN (two layers)	$8Td^{2}$	$8Td^{2}/p$
$\mathcal{L}$ -transform (DCT/IDCT)	—	$Td\log p$

Table 2: Per-layer FLOPs comparison. Projection and FFN costs drop by a factor

\approx p

, while attention score computation remains unchanged (it scales as

T^{2}

Component	Value
Encoder depth	$n_{\text{layers}}=4$
Tokenizer / vocabulary	BPE, 30,000 types
Max sequence length	$T=128$ (identical truncation/padding)

Table 3: Shared architecture settings across all experiments. Tokenization uses byte-pair encoding (BPE) [ 26 ] .

Setting	$d$	$H$	$d_{\mathrm{ff}}$
IMDB	128	4	512
AG News (moderate)	256	4	1024
AG News (BERT-width)	768	8	3072

Table 4: Task-specific architecture settings.

Component	Value
Optimizer	AdamW (lr $=3\times 10^{-4}$ , weight decay $=0.01$ )
Scheduler	OneCycleLR (10% linear warmup, cosine annealing to $10^{-5}$ ) [ 29 , 28 ]
Gradient clipping	$1.0$
Precision	Mixed precision (PyTorch 2.1, torch.cuda.amp ) [ 21 ]
Hardware	NVIDIA Tesla T4 (16 GB), CUDA 12.1
Reproducibility	torch.manual_seed , numpy.random.seed , random.seed ; cudnn.deterministic=True

Table 5: Shared training configuration across all experiments.

Setting	Epochs	Batch size
IMDB ( $d=128$ )	20	128
AG News ( $d=256$ )	5	128
AG News ( $d=768$ )	5	64

Table 6: Dataset-specific training schedules.

Fixed across all models	Varies (encoder only)
Embedding layer (vocabulary $\times d$ )	Encoder type (Std vs. $\mathcal{L}$ -product)
Classifier head (mean-pool + linear)	Decomposition factor $p$
Training recipe (optimizer, scheduler, hyperparameters)	Spectral weighting strategy ( pe )
Data protocol (sequence length $T$ , tokenizer, padding/truncation, seeds)	Encoder parameter count (and resulting compression ratio)

Table 7: Controlled comparison: fixed versus varying components.

Dataset	Model	Acc. (%)	Enc. params	Ratio
IMDB ( $d{=}128$ )	Std (4L, $d{=}128$ )	$80.77\pm 0.24$	991 K	$1.000\times$
	Std-1L (1L, $d{=}128$ )	$77.50\pm 0.28$	248 K	$0.250\times$
	T4-linear (4L, $d_{s}{=}32$ )	$\mathbf{81.65\pm 0.06}$	254 K	$0.256\times$
AG News ( $d{=}256$ )	Std (4L, $d{=}256$ )	$\mathbf{91.40\pm 0.31}$	3 949 K	$1.000\times$
	Std-1L (1L, $d{=}256$ )	$86.97\pm 0.15$	987 K	$0.250\times$
	T4-linear (4L, $d_{s}{=}64$ )	$90.76\pm 0.13$	1 000 K	$0.253\times$

Table 8: Parameter-matched comparison (mean

\pm

std over 3 seeds). Encoder parameters and ratio relative to Std.

Model	$p$	Acc. (%)	Encoder	Enc. Ratio
Std	—	$80.77\pm 0.24$	991 K	$1.000\times$
T4-standard	4	$\mathbf{82.02\pm 0.03}$	254 K	$0.256\times$
T4-learnable	4	$81.98\pm 0.20$	270 K	$0.273\times$
T4-linear	4	$81.65\pm 0.06$	254 K	$0.256\times$
T4-exponential	4	$81.60\pm 0.14$	254 K	$0.256\times$
T4-harmonic	4	$81.53\pm 0.13$	254 K	$0.256\times$

Table 9: IMDB results (

d{=}128

, 20 epochs, 3 seeds). Embedding (3.84 M) and classifier are shared. Best in bold .

Model	$p$	Acc. (%)	Encoder	Enc. Ratio
Std	—	$\mathbf{91.40\pm 0.31}$	3.95 M	$1.000\times$
T4-linear	4	$90.76\pm 0.13$	1.00 M	$0.253\times$
T4-harmonic	4	$90.41\pm 0.08$	1.00 M	$0.253\times$

Table 10: AG News results (

d{=}256

, 5 epochs, 3 seeds). Best in bold .

Fig. 2: Encoder parameters versus test accuracy on IMDB (left) and AG News (right). Error bars show

\pm 1

std over 3 seeds. Tensor models achieve competitive or superior accuracy while using approximately

4\times

fewer encoder parameters at

p=4

Fig. 3: PE strategy comparison on IMDB at

p=4

(mean

\pm

std, 3 seeds). The dashed line shows the full-width Std baseline. All strategies outperform Std, and the total spread across strategies is 0.49 pp.

Model	Acc. (%)	Encoder	Ratio	Mem
Std	$91.47\pm 0.09$	28.4 M	$1.00\times$	1.95 GB
T4-linear	$\mathbf{91.52\pm 0.21}$	7.1 M	$0.25\times$	1.66 GB

Table 11: AG News at

d=768

(BERT-base width),

p=4

, from scratch, 3 seeds. The tensor model compresses the encoder by

4\times

and reduces peak memory while maintaining accuracy.

Component	Std	T4-linear
Embedding	23.0 M (45%)	23.0 M (76%)
Encoder	28.4 M (55%)	7.1 M (24%)
Total	51.4 M	30.2 M ( $0.59\times$ )

Table 12: Parameter breakdown at

d=768

. Encoder compression becomes a model-level reduction because the encoder accounts for the majority of parameters.

Fig. 4: Accuracy gap (best T4 minus Std) as a function of model width

d

. The shaded band indicates a

\pm 0.3

pp parity region. At

d=768

, the tensor model reaches parity while compressing the encoder by

4\times

Config	Model	Enc. params	Ratio	Train tok/s	Epoch (s)	Peak mem	Batch
IMDB $d{=}128$	Std	991 K	$1.00\times$	123 K	26 s	0.50 GB	128 / 256
IMDB $d{=}128$	T4-linear	254 K	$0.26\times$	94 K	34 s	0.51 GB	128 / 256
AG News $d{=}256$	Std	3 949 K	$1.00\times$	177 K	87 s	0.99 GB	128 / 256
AG News $d{=}256$	T4-linear	1 000 K	$0.25\times$	132 K	116 s	0.96 GB	128 / 256
AG News $d{=}768$	Std	28 352 K	$1.00\times$	60 K	257 s	1.95 GB	64 / 128
AG News $d{=}768$	T4-linear	7 118 K	$0.25\times$	64 K	241 s	1.66 GB	64 / 128

Table 13: Efficiency comparison at equal depth (4 layers, mean over 3 seeds). Train throughput is measured per epoch; inference uses the full test set under torch.no_grad() with mixed precision.

Configuration	Std	Best T4	$\bm{\Delta}$	Enc. Saving
IMDB $d=128$	$80.77_{\pm 0.24}$	$82.02_{\pm 0.03}$ (std)	$+1.25$	991 K $\to$ 254 K
AG News $d=256$	$91.40_{\pm 0.31}$	$90.76_{\pm 0.13}$ (lin)	$-0.64$	3.95 M $\to$ 1.00 M
AG News $d=768$	$91.47_{\pm 0.09}$	$91.52_{\pm 0.21}$ (lin)	$+0.05$	28.4 M $\to$ 7.1 M

Table 14: Summary across all configurations (mean

\pm

std, 3 seeds).

\Delta

is the accuracy difference of the best tensor variant versus Std.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 19 — Fig. 5: Slice-wise equivalence of the tensor Transformer under the $\mathcal{L}$ -product framework. The tensor is transformed along mode-3, processed independently slice-by-slice by $p$ compact Transformers (dimension $d_{s}$ ), stacked back, and mapped to the original domain via $\mathcal{L}^{-1}$ .

Fig. 6: Training curves on IMDB (

d=128

, seed 42).

Fig. 7: Training curves on AG News (

d=256

, seed 42).

PE strategy	IMDB $d=128$	AG News $d=256$
T4-standard ( $\alpha_{k}=1$ )	$\mathbf{82.02\pm 0.03}$	—
T4-linear ( $\alpha_{k}=k/p$ )	$81.65\pm 0.06$	$\mathbf{90.76\pm 0.13}$
T4-exponential ( $\alpha_{k}=2^{(k-1)/(p-1)}$ )	$81.60\pm 0.14$	—
T4-harmonic ( $\alpha_{k}=k$ )	$81.53\pm 0.13$	$90.41\pm 0.08$
T4-learnable	$81.98\pm 0.20$	—
Std (baseline)	$80.77\pm 0.24$	$91.40\pm 0.31$

Appendix

Table 15: Full PE sweep at

p=4

(mean

\pm

std, 3 seeds). Best per dataset in bold .

Config	Model	Enc.	Train (s/ep)	Mem (GB)	Infer (s)
IMDB ( $d=128$ )	Std	991 K	26	0.50	11.8
	T4-standard	254 K	34	0.51	12.7
	T4-linear	254 K	34	0.51	12.6
	T4-harmonic	254 K	34	0.51	12.8
	T4-learnable	270 K	34	0.51	12.8
	T2-linear	500 K	29	0.49	11.9

Appendix

Table 16: Runtime details. Train is mean wall-clock per epoch; Mem is peak GPU allocation; Infer is forward pass on the full test set. All runs on NVIDIA Tesla T4 with mixed precision.

Transformer

Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.

See all 540 papers

Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability
Aug 31, 2026Matvei Tarasov, Salman Ahmadi-Asl, Andre L. F. de Almeida +1LLM Compression LLM Inference Acceleration
Innopolis University, Innopolis 420500, Russia · Department of Teleinformatics Engineering, Federal University of Cear´a, Fortaleza 60455-760, Brazil · Systems Research Institute of Polish Academy of Science and Warsaw University of Technology, Poland
Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression
Jun 2, 2026Artur Zagitov, Alexander Miasnikov, Maxim Krutikov +5LLM Compression Tensor Decomposition
BRAIn Lab, Moscow, Russia
A general tensor-structured compression scheme for efficient large language models
May 25, 2026Ying Lu, Peng-Fei Zhou, Qi-Xuan Fang +3LLM Inference Efficiency LLM Compression
School of Physical Sciences, University of Chinese Academy of Sciences, Beijing, 100049, China. · Kavli Institute for Theoretical Sciences, University of Chinese Academy of Sciences, Beijing, 100190, China. · Center for Quantum Physics and Intelligent Sciences, Department of Physics, Capital Normal University, Beijing, 100048, China. +1

Aug 31, 2026J/K move · Enter open · S save

Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability

Matvei Tarasov, Salman Ahmadi-Asl, Andre L. F. de Almeida, Andrzej Cichocki

Innopolis University, Innopolis 420500, Russia · Department of Teleinformatics Engineering, Federal University of Cear´a, Fortaleza 60455-760, Brazil · Systems Research Institute of Polish Academy of Science and Warsaw University of Technology, Poland

LLM Compression LLM Inference Acceleration Language Modeling

Open paper AskPDF arXiv

Large language models (LLMs) are built from structured high-dimensional objects such as token representations, weights, adaptation updates, caches, and activations, whose multilinear structure is underexploited by the conventional matrix-centric view. Tensor decompositions and tensor networks provide a principled algebraic language for this structure, yet the literature often treats them as isolated compression mechanisms. This survey organizes tensor methods for LLMs through two complementary views: a seven-stage lifecycle taxonomy covering tokenization, embeddings, pre-training, adaptation, compression, inference, and interpretability, and a component view covering embeddings, attention, and feed-forward networks. We provide unified notation and theoretical foundations, analyze tensorization strategies for individual Transformer components, and compare methods at each lifecycle stage while making differences in evaluation protocols and model scales explicit. We further connect tensor methods to neighboring efficiency techniques and probabilistic tensor networks. Finally, we synthesize open challenges and introduce

ρ_{\rm gap}

, a metric for the compression-realization gap between theoretical memory reduction and measured system-level speedup. By treating tensorization as a common structural principle, the survey provides a structured entry point to tensorized language models and clarifies when parameter savings can plausibly translate into memory efficiency, computational efficiency, or interpretability. The GitHub page dedicated to this paper is accessible at this https URL.

Abstract

Transformer

Explore similar work

Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability

Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression

A general tensor-structured compression scheme for efficient large language models