Tensor Decomposition

Follow this topic Ask about this topic

Tensor decomposition studies how to represent multidimensional data using simpler factors, making patterns easier to analyze and large data or models cheaper to store and process. Common approaches include CP and Tucker factorization, tensor trains and matrix product states, and t-SVD, used for tasks such as missing-data recovery, anomaly detection, scientific modeling, and neural-network compression. Current work combines these methods with Bayesian rank selection, adaptive optimization, pretrained or neural priors, and theoretical analyses of recovery and identifiability. A central tension is choosing a factorization and rank that capture useful structure without imposing unrealistic shared-subspace or ordering assumptions, while balancing accuracy, robustness, and computational cost.

Latest papers

70 papers10 papers in the last four weeks, up 11% on the four weeks before. 0.1% of all new papers.

Structured Multidimensional Representation Learning for Large Language Models
Mar 5, 2026Alaa El Ichi, Khalide Jbilou, Mohamed El Guide +1Transformer Embedding Compression
Universit´e du Littoral Cote d’Opale, LMPA, 50 rue F. Buisson, 62228 Calais-Cedex, France. · FGSES, University Mohammed VI Polytechnic, Rabat, Morocco · LISIC, 50 rue F. Buisson, Universit´e du Littoral Cote d’Opale, 62228 Calais-Cedex, France.
Tensor Decomposition Structure Search Framework from an Interaction Perspective
Mar 3, 2026Ting-Wei Zhou, Xi-Le Zhao, Sheng Liu +3Tensor Decomposition
School of Mathematical Sciences/Research Center for Image and Vision Computing, University of Electronic Science and Technology of China, Chengdu, Sichuan 611731, China · School of Information Science and Technology, Southwest Jiaotong University, Chengdu 611756, China · School of Mathematics and Statistics and Ministry of Education Key Lab of Intelligent Networks and Network Security, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China, and also with Pazhou Laboratory (Huangpu), Guangzhou, Guangdong 510335, China
Reparameterized Tensor Ring Functional Decomposition for Multi-Dimensional Data Recovery
Mar 1, 2026Yangyang Xu, Junbo Ke, You-Wei Wen +1Tensor Decomposition Tensor Completion
Key Laboratory of Computing and Stochastic Mathematics (Ministry of Education), School of Mathematics and Statistics, Hunan Normal University · Department of Statistics and Data Science, Southern University of Science and Technology
Neural Learning of Fast Matrix Multiplication Algorithms: A StrassenNet Approach
Feb 25, 2026Paolo Andreini, Alessandra Bernardi, Monica Bianchini +4Tensor Decomposition Automated Algorithm Discovery
LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights
Feb 19, 2026Kasun Dewage, Marianna Pensky, Shankadeep Mondal +1Tensor Decomposition Low-Rank Adaptation
University of Central Florida Orlando, FL 32816, USA
Transforming Behavioral Neuroscience Discovery with In-Context Learning and AI-Enhanced Tensor Methods
Feb 19, 2026Paimon Goulart, Jordan Steinhauser, Dawon Ahn +4AI for Science Tensor Decomposition
UC Riverside Computer Science & Engineering Riverside, CA, USA · UC Riverside Psychology Riverside, CA, USA · UC Riverside Electrical & Computer Engineering Riverside, CA, USA
Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutions
Feb 4, 2026Dmitry Yarotsky, Eugene Golikov, Yaroslav GusevTensor Decomposition Mean-Field Theory
Applied AI Institute, Moscow, Russia · Steklov Mathematical Institute of Russian Academy of Sciences, Moscow, Russia.
MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuning
Jun 10, 2025Javier Lopez-Piqueres, Pranav Deshpande, Archan Ray +3Transformer Tensor Decomposition
Global Technology Applied Research, JPMorgan Chase, New York, NY 10001, USA
Joint Bayesian Parameter and Model Order Estimation for Low-Rank Probability Mass Tensors
Oct 8, 2024Joseph K. Chege, Arie Yeredor, Martin HaardtBayesian Inference Tensor Decomposition
Communications Research Laboratory, Ilmenau University of Technology, Ilmenau, Germany · School of Electrical and Computer Engineering, Tel Aviv University, Tel Aviv, Israel
Tensor-reduced atomic density representations
Oct 2, 2022James P. Darby, Dávid P. Kovács, Ilyes Batatia +4Molecular Representation Learning Invariant Representation Learning
Warwick Centre for Predictive Modelling, School of Engineering, University of Warwick, Coventry, CV4 7AL, UK · Engineering Laboratory, University of Cambridge, Cambridge, CB2 1PZ UK · ENS Paris-Saclay, Université Paris-Saclay, 91190 Gif-sur-Yvette, France +3

Mar 5, 2026J/K move · Enter open · S save

Structured Multidimensional Representation Learning for Large Language Models

Alaa El Ichi, Khalide Jbilou, Mohamed El Guide, Franck Dufrenois

Universit´e du Littoral Cote d’Opale, LMPA, 50 rue F. Buisson, 62228 Calais-Cedex, France. · FGSES, University Mohammed VI Polytechnic, Rabat, Morocco · LISIC, 50 rue F. Buisson, Universit´e du Littoral Cote d’Opale, 62228 Calais-Cedex, France.

Transformer Embedding Compression Neural Network Compression Tensor Decomposition

Open paper AskPDF arXiv

Transformer architectures achieve state-of-the-art performance across a wide range of pattern recognition and natural language processing tasks, but their scaling is accompanied by substantial parameter growth and redundancy in the embedding dimension. In this work, we introduce a structured spectral factorization of the embedding space based on the L-product for third-order tensors. By reshaping token representations into spectral tensor slices and performing attention and feed-forward operations in the transform domain, we obtain a Tensor Transformer architecture that decomposes the encoder into p independent spectral sub-transformers while preserving standard Transformer semantics. We prove that the proposed L-Transformer is spectrally equivalent to p parallel Transformers operating on reduceddimensional embeddings, which yields approximately 1/p reduction (up to lower-order terms such as biases and normalization parameters) in encoder parameters under fixed total embedding size. When instantiated with a real-valued Discrete Cosine Transform (DCT), the method remains fully differentiable and compatible with existing training pipelines. Beyond compression, the spectral decomposition introduces an inductive bias over embedding frequencies, enabling slice-dependent frequency scaling that improves generalization. Experiments on IMDB and AGNews show that the proposed model can substantially reduce encoder parameters (up to 75% for p=4) while maintaining competitive accuracy. On IMDB, the tensorized encoder matches or improves upon the standard baseline under compression, whereas on AGNews at moderate width we observe a small accuracy decrease in exchange for a 4 times encoder reduction; at BERT-base width (d=768), performance returns to parity.