Tensor Decomposition

Ask about this topic

Overview

Tensor decomposition studies how to represent multidimensional data using simpler factors, making patterns easier to analyze and large data or models cheaper to store and process. Common approaches include CP and Tucker factorization, tensor trains and matrix product states, and t-SVD, used for tasks such as missing-data recovery, anomaly detection, scientific modeling, and neural-network compression. Current work combines these methods with Bayesian rank selection, adaptive optimization, pretrained or neural priors, and theoretical analyses of recovery and identifiability. A central tension is choosing a factorization and rank that capture useful structure without imposing unrealistic shared-subspace or ordering assumptions, while balancing accuracy, robustness, and computational cost.

Latest papers

CardsList

70 papers10 papers in the last four weeks, up 11% on the four weeks before. 0.1% of all new papers.

Mar 5, 2026

Structured Multidimensional Representation Learning for Large Language Models

Transformer architectures achieve state-of-the-art performance across a wide range of pattern recognition and natural language processing tasks, but their scaling is accompanied by substantial parameter growth and redundancy in the embedding dimension. In this work, we introduce a structured spectral factorization of the embedding space based on the L-product for third-order tensors. By reshaping token representations into spectral tensor slices and performing attention and feed-forward operations in the transform domain, we obtain a Tensor Transformer architecture that decomposes the encoder into p independent spectral sub-transformers while preserving standard Transformer semantics. We prove that the proposed L-Transformer is spectrally equivalent to p parallel Transformers operating on reduceddimensional embeddings, which yields approximately 1/p reduction (up to lower-order terms such as biases and normalization parameters) in encoder parameters under fixed total embedding size. When instantiated with a real-valued Discrete Cosine Transform (DCT), the method remains fully differentiable and compatible with existing training pipelines. Beyond compression, the spectral decomposition introduces an inductive bias over embedding frequencies, enabling slice-dependent frequency scaling that improves generalization. Experiments on IMDB and AGNews show that the proposed model can substantially reduce encoder parameters (up to 75% for p=4) while maintaining competitive accuracy. On IMDB, the tensorized encoder matches or improves upon the standard baseline under compression, whereas on AGNews at moderate width we observe a small accuracy decrease in exchange for a 4 times encoder reduction; at BERT-base width (d=768), performance returns to parity.

Alaa El Ichi, Khalide Jbilou, Mohamed El Guide +1

Universit´e du Littoral Cote d’Opale, LMPA, 50 rue F. Buisson, 62228 Calais-Cedex, France. · FGSES, University Mohammed VI Polytechnic, Rabat, Morocco · LISIC, 50 rue F. Buisson, Universit´e du Littoral Cote d’Opale, 62228 Calais-Cedex, France.

TransformerEmbedding CompressionNeural Network CompressionTensor Decomposition
AskPDF
Mar 3, 2026

Tensor Decomposition Structure Search Framework from an Interaction Perspective

Recently, tensor decompositions have attracted increasing attention. Fundamentally, different interactions among factors induce distinct tensor decomposition structures (i.e., tensor decomposition). Identifying an appropriate interaction-induced tensor decomposition structure for given data is a fundamental yet challenging problem in tensor modeling, and remains largely under-explored. Existing tensor decomposition structure search methods are typically restricted to a predefined interaction family, such as tensor contraction. To address this problem, we suggest a tensor decomposition structure search framework (I-TSS) from an interaction perspective that can identify either a single structure beyond a predefined interaction family or a mixture of structures involving heterogeneous interaction families. Specifically, we first systematically review existing tensor decomposition structures from an interaction perspective and construct a heterogeneous interaction-induced candidate structure set. Based on it, we propose a unified energy-based rank estimation scheme for heterogeneous interaction-induced candidate structure set. We then introduce the top-kk gating mechanism with learnable gating scores to dynamically select a suitable single-candidate structure or combine suitable multi-candidate structures, thereby identifying the tensor decomposition structure in a data-adaptive manner. Theoretically, we derive an approximation error bound for I-TSS, thereby establishing its approximation capability. Extensive experiments on synthetic and real-world datasets demonstrate that I-TSS consistently outperforms state-of-the-art tensor decomposition methods.

Ting-Wei Zhou, Xi-Le Zhao, Sheng Liu +3

School of Mathematical Sciences/Research Center for Image and Vision Computing, University of Electronic Science and Technology of China, Chengdu, Sichuan 611731, China · School of Information Science and Technology, Southwest Jiaotong University, Chengdu 611756, China · School of Mathematics and Statistics and Ministry of Education Key Lab of Intelligent Networks and Network Security, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China, and also with Pazhou Laboratory (Huangpu), Guangzhou, Guangdong 510335, China

Tensor Decomposition
AskPDF
Mar 1, 2026

Reparameterized Tensor Ring Functional Decomposition for Multi-Dimensional Data Recovery

Tensor Ring (TR) decomposition is a powerful tool for high-order data modeling, but is inherently restricted to discrete forms defined on fixed meshgrids. In this work, we propose a TR functional decomposition for both meshgrid and non-meshgrid data, where factors are parameterized by Implicit Neural Representations (INRs). However, optimizing this continuous framework to capture fine-scale details is intrinsically difficult. Through a frequency-domain analysis, we demonstrate that the spectral structure of TR factors determines the frequency composition of the reconstructed tensor and limits the high-frequency modeling capacity. To mitigate this, we propose a reparameterized TR functional decomposition, in which each TR factor is a structured combination of a learnable latent tensor and a fixed basis. This reparameterization is theoretically shown to improve the training dynamics of TR factor learning. We further derive a principled initialization scheme for the fixed basis and prove the Lipschitz continuity of our proposed model. Extensive experiments on image inpainting, denoising, super-resolution, and point cloud recovery demonstrate that our method achieves consistently superior performance over existing approaches. Code is available at https://github.com/YangyangXu2002/RepTRFD.

Yangyang Xu, Junbo Ke, You-Wei Wen +1

Key Laboratory of Computing and Stochastic Mathematics (Ministry of Education), School of Mathematics and Statistics, Hunan Normal University · Department of Statistics and Data Science, Southern University of Science and Technology

Tensor DecompositionTensor Completion
AskPDF
Feb 25, 2026

Neural Learning of Fast Matrix Multiplication Algorithms: A StrassenNet Approach

Fast matrix multiplication can be described as searching for low-rank decompositions of the matrix--multiplication tensor. We design a neural architecture, StrassenNet, which reproduces the Strassen algorithm for 2×22\times 2 multiplication. Across many independent runs the network always converges to a rank-77 tensor, thus numerically recovering Strassen's optimal algorithm. We then train the same architecture on 3×33\times 3 multiplication with rank r∈{19,…,23}r\in\{19,\dots,23\}. Our experiments reveal a clear numerical threshold: models with r=23r=23 attain significantly lower validation error than those with r≤22r\le 22, suggesting that r=23r=23 could actually be the smallest effective rank of the matrix multiplication tensor 3×33\times 3. We also sketch an extension of the method to border-rank decompositions via an ε\varepsilon--parametrisation and report preliminary results consistent with the known bounds for the border rank of the 3×33\times 3 matrix--multiplication tensor.

Paolo Andreini, Alessandra Bernardi, Monica Bianchini +4

Tensor DecompositionAutomated Algorithm DiscoveryLow-Rank Matrix Decomposition
AskPDF
Feb 19, 2026

LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights

We introduce LoRA-CRAFT (Cross-layer Rank Adaptation via Frozen Tucker), abbreviated CRAFT throughout, an extremely parameter-efficient fine-tuning (PEFT) method that applies Tucker tensor decomposition to pre-trained attention weight matrices stacked across transformer layers and trains only small square adaptation matrices on the resulting frozen Tucker factors. Existing tensor-based PEFT methods decompose gradient updates: LoTR applies Tucker decomposition with shared factor matrices, while SuperLoRA groups and reshapes ΔWΔW across layers before applying Tucker decomposition. Separately, methods such as PiSSA apply SVD to pre-trained weights but operate independently per layer. CRAFT bridges these two lines of work: it performs full Tucker decomposition via Higher-Order SVD (HOSVD) directly on pre-trained weights organized as cross-layer 3D tensors, freezes all resulting factors, and adapts the model through lightweight trainable transformations applied to each factor matrix. Experiments on the GLUE benchmark using RoBERTa-base and RoBERTa-large, as well as commonsense reasoning benchmarks using LLaMA2-7B and LLaMA3-8B, demonstrate that CRAFT achieves competitive performance with existing methods while requiring only \rev{extremely low Tucker adaptation parameters}. \fixw{On LLaMA3-8B, CRAFT} \rev{exceeds the average accuracy of LoRA} using hundreds of times fewer parameters\fixw{; on LLaMA2-7B the same holds at a 0.2520.252M budget}. Our results suggest that CRAFT's efficiency advantage grows with model scale, as the frozen Tucker factors better capture the richer cross-layer structure of larger pre-trained models.

Kasun Dewage, Marianna Pensky, Shankadeep Mondal +1

University of Central Florida Orlando, FL 32816, USA

Tensor DecompositionLow-Rank AdaptationTransformer AttentionParameter-Efficient Fine-Tuning
AskPDF
Feb 19, 2026

Transforming Behavioral Neuroscience Discovery with In-Context Learning and AI-Enhanced Tensor Methods

Scientific discovery pipelines typically involve complex, rigid, and time-consuming processes, from data preparation to analyzing and interpreting findings. Recent advances in AI have the potential to transform such pipelines in a way that domain experts can focus on interpreting and understanding findings, rather than debugging rigid pipelines or manually annotating data. As part of an active collaboration between data science/AI researchers and behavioral neuroscientists, we showcase an example AI-enhanced pipeline, specifically designed to transform and accelerate the way that the domain experts in the team are able to gain insights out of experimental data. The application at hand is in the domain of behavioral neuroscience, studying fear generalization in mice, an important problem whose progress can advance our understanding of clinically significant and often debilitating conditions such as PTSD (Post-Traumatic Stress Disorder). We identify the emerging paradigm of "In-Context Learning" (ICL) as a suitable interface for domain experts to automate parts of their pipeline without the need for or familiarity with AI model training and fine-tuning, and showcase its remarkable efficacy in data preparation and pattern interpretation. Also, we introduce novel AI-enhancements to tensor decomposition model, which allows for more seamless pattern discovery from the heterogeneous data in our application. We thoroughly evaluate our proposed pipeline experimentally, showcasing its superior performance compared to what is standard practice in the domain, as well as against reasonable ML baselines that do not fall under the ICL paradigm, to ensure that we are not compromising performance in our quest for a seamless and easy-to-use interface for domain experts. Finally, we demonstrate effective discovery, with results validated by the domain experts in the team.

Paimon Goulart, Jordan Steinhauser, Dawon Ahn +4

UC Riverside Computer Science & Engineering Riverside, CA, USA · UC Riverside Psychology Riverside, CA, USA · UC Riverside Electrical & Computer Engineering Riverside, CA, USA

AI for ScienceTensor DecompositionIn-Context Learning
AskPDF
Feb 4, 2026

Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutions

We develop a general mathematical framework to analyze scaling regimes and derive explicit analytic solutions for gradient flow (GF) in large learning problems. Our key innovation is a formal power series expansion of the loss evolution, with coefficients encoded by diagrams akin to Feynman diagrams. We show that this expansion has a well-defined large-size limit that can be used to reveal different learning phases and, in some cases, to obtain explicit solutions of the nonlinear GF. We focus on learning Canonical Polyadic (CP) decompositions of high-order tensors, and show that this model has several distinct extreme lazy and rich GF regimes such as free evolution, NTK and under- and over-parameterized mean-field. We show that these regimes depend on the parameter scaling, tensor order, and symmetry of the model in a specific and subtle way. Moreover, we propose a general approach to summing the formal loss expansion by reducing it to a PDE; in a wide range of scenarios, it turns out to be first-order and solvable by the method of characteristics. We observe a very good agreement of our theoretical predictions with experimental results.

Dmitry Yarotsky, Eugene Golikov, Yaroslav Gusev

Applied AI Institute, Moscow, Russia · Steklov Mathematical Institute of Russian Academy of Sciences, Moscow, Russia.

Tensor DecompositionMean-Field TheoryNeural Tangent KernelGradient Descent Dynamics
AskPDF
Jun 10, 2025

MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuning

We present MetaTT, a Tensor Train (TT) adapter framework for fine-tuning of pre-trained transformers. MetaTT enables flexible and parameter-efficient model adaptation by using a single shared TT to factorize transformer sub-modules. This factorization indexes key structural dimensions, including layer and matrix type, and can optionally incorporate heads and tasks. This design allows MetaTT's parameter count to scale with the sum, rather than the product, of the modes, resulting in a substantially more compact adapter. Our benchmarks compare MetaTT with LoRA along with recent state-of-the-art matrix and tensor decomposition based fine-tuning methods. We observe that when tested on single-task standard language modeling benchmarks, MetaTT achieves competitive parameter efficiency to accuracy tradeoff. We further demonstrate that MetaTT performs competitively when compared to state-of-the-art methods on multi-task learning. Finally, we leverage the TT decomposition to design a rank adaptive optimizer inspired by the DMRG method from many-body physics. Our results demonstrate that integrating this approach with AdamW enhances optimization performance for a specified target rank.

Javier Lopez-Piqueres, Pranav Deshpande, Archan Ray +3

Global Technology Applied Research, JPMorgan Chase, New York, NY 10001, USA

TransformerTensor DecompositionTT DecompositionAdapter TuningParameter-Efficient Fine-Tuning
AskPDF
Oct 8, 2024

Joint Bayesian Parameter and Model Order Estimation for Low-Rank Probability Mass Tensors

Obtaining a reliable estimate of the joint probability mass function (PMF) of a set of random variables from observed data is a significant objective in statistical signal processing and machine learning. Modelling the joint PMF as a tensor that admits a low-rank canonical polyadic decomposition (CPD) has enabled the development of efficient PMF estimation algorithms. However, these algorithms require the rank (model order) of the tensor to be specified beforehand. In real-world applications, the true rank is unknown. Therefore, an appropriate rank is usually selected from a candidate set either by observing validation errors or by computing various likelihood-based information criteria, a procedure that could be costly in terms of computational time or hardware resources, or could result in mismatched models which affect the model accuracy. This paper presents a novel Bayesian framework for estimating the low-rank components of a joint PMF tensor and simultaneously inferring its rank from the observed data. We specify a Bayesian PMF estimation model and employ appropriate prior distributions for the model parameters, allowing the rank to be inferred without cross-validation.We then derive a deterministic solution based on variational inference (VI) to approximate the posterior distributions of various model parameters. Numerical experiments involving both synthetic data and real classification and item recommendation data illustrate the advantages of our VI-based method in terms of estimation accuracy, automatic rank detection, and computational efficiency.

Joseph K. Chege, Arie Yeredor, Martin Haardt

Communications Research Laboratory, Ilmenau University of Technology, Ilmenau, Germany · School of Electrical and Computer Engineering, Tel Aviv University, Tel Aviv, Israel

Bayesian InferenceTensor DecompositionLow-Rank Approximation
AskPDF
Oct 2, 2022

Tensor-reduced atomic density representations

Density based representations of atomic environments that are invariant under Euclidean symmetries have become a widely used tool in the machine learning of interatomic potentials, broader data-driven atomistic modelling and the visualisation and analysis of materials datasets.The standard mechanism used to incorporate chemical element information is to create separate densities for each element and form tensor products between them. This leads to a steep scaling in the size of the representation as the number of elements increases. Graph neural networks, which do not explicitly use density representations, escape this scaling by mapping the chemical element information into a fixed dimensional space in a learnable way. We recast this approach as tensor factorisation by exploiting the tensor structure of standard neighbour density based descriptors. In doing so, we form compact tensor-reduced representations whose size does not depend on the number of chemical elements, but remain systematically convergeable and are therefore applicable to a wide range of data analysis and regression tasks.

James P. Darby, Dávid P. Kovács, Ilyes Batatia +4

Warwick Centre for Predictive Modelling, School of Engineering, University of Warwick, Coventry, CV4 7AL, UK · Engineering Laboratory, University of Cambridge, Cambridge, CB2 1PZ UK · ENS Paris-Saclay, Université Paris-Saclay, 91190 Gif-sur-Yvette, France +3

Molecular Representation LearningInvariant Representation LearningMachine Learning Interatomic PotentialsComputational Materials ScienceTensor Decomposition
AskPDF