Tensor Decomposition Method

Tensor decomposition methods aim to represent high-dimensional data (tensors) as a product of lower-dimensional matrices, achieving data compression and efficient computation. Current research focuses on adapting these methods for various applications, including compressing neural networks (e.g., using Kronecker or Tucker decompositions), improving time-series classification, and enhancing key-value caching in large language models. These advancements are significant because they enable efficient processing of massive datasets and deployment of complex models on resource-constrained devices, impacting fields like machine learning, natural language processing, and computer vision.

Papers

November 10, 2024

Activation Map Compression through Tensor Decomposition for Deep Learning
Le-Trung Nguyen, Aël Quélennec, Enzo Tartaglione, Samuel Tardieu, Van-Tam Nguyen
Deep Learning LeArning Abstract Back Propagation Tensor Decomposition Device Inference Tensor Decomposition Method Activation Compression Higher Order Singular Value Decomposition

October 3, 2024

Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Models
Mingxue Xu, Sadia Sharmin, Danilo P. Mandic
Language Model Comprehensive Taxonomy Generative Language Model Linear Compression Tensor Factorization Terminator Economy Tensor Based Tensor Decomposition Method Tensor Compression

September 23, 2024

Enabling Tensor Decomposition for Time-Series Classification via A Simple Pseudo-Laplacian Contrast
Man Li, Ziyue Li, Lijun Sun, Fugee Tsung
Time Series Classification Tensor Decomposition Low Dimensional Representation Low Rank Structure Color Contrast Class Representation Tensor Decomposition Method

August 29, 2024

Convolutional Neural Network Compression Based on Low-Rank Decomposition
Yaping He, Linhao Jiang, Di Wu
Neural Network Matrix Factorization Tensor Decomposition Microbial Decomposition Low Rank Decomposition Tensor Decomposition Method

May 21, 2024

Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression
Peiyu Liu, Ze-Feng Gao, Wayne Xin Zhao, Yipeng Ma, Tao Wang, Ji-Rong Wen
Tensor Decomposition Low Bit Quantization Data Free Quantization Tensor Decomposition Method Matrix Decomposition Efficient Dequantization KV Cache Compression

April 14, 2024

Mitigating Heterogeneity among Factor Tensors via Lie Group Manifolds for Tensor Decomposition Based Temporal Knowledge Graph Embedding
Jiang Li, Xiangdong Su, Yeyun Gong, Guanglai Gao
Tensor Decomposition Temporal Knowledge Graph Tensor Decomposition Method Factor Tensor Tensor Fusion

January 29, 2024

TQCompressor: improving tensor decomposition methods in neural networks via permutations
V. Abronin, A. Naumov, D. Mazur, D. Bystrov, K. Tsarova, Ar. Melnikov, I. Oseledets, S. Dolgov, R. Brasher, M. Perelshtein
Neural Network Neural Network Compression Machine Permutation Tensor Decomposition Method

January 6, 2024

TEN-GUARD: Tensor Decomposition for Backdoor Attack Detection in Deep Neural Networks
Khondoker Murad Hossain, Tim Oates
Deep Neural Network Backdoor Attack Tensor Decomposition Backdoor Detection Tensor Decomposition Method Detection Pipeline

September 18, 2023

Multi-Dictionary Tensor Decomposition
Maxwell McNeil, Petko Bogdanov
Tensor Decomposition Low Rank Tensor Tensor Decomposition Method Sparse Tensor

September 7, 2023

Personalized Tucker Decomposition: Modeling Commonality and Peculiarity on Tensor Data
Jiuyun Hu, Naichen Shi, Raed Al Kontar, Hao Yan
Proximal Gradient Tensor Data Tucker Decomposition Block Coordinate Descent Tensor Decomposition Method

July 13, 2023

Tensor Decompositions Meet Control Theory: Learning General Mixtures of Linear Dynamical Systems
Ainesh Bakshi, Allen Liu, Ankur Moitra, Morris Yau
Tensor Decomposition Mixture Model Latent Variable Model Linear Dynamical System Input Mixture Tensor Decomposition Method

May 9, 2023

How Informative is the Approximation Error from Tensor Decomposition for Neural Network Compression?
Jetze T. Schuurmans, Kim Batselier, Julian F. P. Kooij
Linear Compression Tensor Decomposition Neural Network Compression Tensor Decomposition Method Approximation Error Stagewise Decomposition

April 26, 2023

Tensor Decomposition for Model Reduction in Neural Networks: A Review
Xingyi Liu, Keshab K. Parhi
Neural Network Convolutional Neural Network Narrative Review Tensor Decomposition Model Reduction Tensor Decomposition Method Tensor Approximation

February 10, 2023

Fast Learnings of Coupled Nonnegative Tensor Decomposition Using Optimal Gradient and Low-rank Approximation
Xiulin Wang, Jing Liu, Fengyu Cong
Tensor Decomposition Low Rank Approximation Gradient Based Optimization Tensor Data Fast Learning Tensor Decomposition Method Rank Adaptive Tensor Optimization

October 12, 2022

SeKron: A Decomposition Method Supporting Many Factorization Structures
Marawan Gamal Abdel Hameed, Ali Mosleh, Marzieh S. Tahaei, Vahid Partovi Nia
Tucker Decomposition Factorization Machine Decomposition Method Tensor Decomposition Method Projection Enhancement Network

October 7, 2022

Sampling-Based Decomposition Algorithms for Arbitrary Tensor Networks
Osman Asif Malik, Vivek Bharadwaj, Riley Murray
Tensor Decomposition Tensor Type Calculus Tensor Decomposition Method

July 17, 2022

Towards Programmable Memory Controller for Tensor Decomposition
Sasindu Wijeratne, Ta-Yang Wang, Rajgopal Kannan, Viktor Prasanna
Tensor Decomposition Tensor Decomposition Method

May 21, 2022

Tensor Shape Search for Optimum Data Compression
Ryan Solgi, Zichang He, William Jiahua Liang, Zheng Zhang
Tensor Decomposition Tensor Decomposition Method Optimal Compression Tensor Train Decomposition Tensor Shape

Tensor Decomposition Method

Papers

Activation Map Compression through Tensor Decomposition for Deep Learning

Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Models

Enabling Tensor Decomposition for Time-Series Classification via A Simple Pseudo-Laplacian Contrast

Convolutional Neural Network Compression Based on Low-Rank Decomposition

Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression

Mitigating Heterogeneity among Factor Tensors via Lie Group Manifolds for Tensor Decomposition Based Temporal Knowledge Graph Embedding

TQCompressor: improving tensor decomposition methods in neural networks via permutations

TEN-GUARD: Tensor Decomposition for Backdoor Attack Detection in Deep Neural Networks

Multi-Dictionary Tensor Decomposition

Personalized Tucker Decomposition: Modeling Commonality and Peculiarity on Tensor Data

Tensor Decompositions Meet Control Theory: Learning General Mixtures of Linear Dynamical Systems

How Informative is the Approximation Error from Tensor Decomposition for Neural Network Compression?

Tensor Decomposition for Model Reduction in Neural Networks: A Review

Fast Learnings of Coupled Nonnegative Tensor Decomposition Using Optimal Gradient and Low-rank Approximation

SeKron: A Decomposition Method Supporting Many Factorization Structures

Sampling-Based Decomposition Algorithms for Arbitrary Tensor Networks

Towards Programmable Memory Controller for Tensor Decomposition

Tensor Shape Search for Optimum Data Compression