Tucker Decomposition

Latest papers

14 papers3 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models
Oct 1, 2026Lianjun Liu, Shipeng Li, You Huang +5Tucker Decomposition Mixture-of-Experts Language Models
Hainan University, Haikou, China · Xiamen University, Xiamen, China · Zhejiang Normal University, Jinhua, China
Sequential Functional Structured Tucker Compression for Large Language Model Attentions
Sep 30, 2026Jiangfeng Chen, Xinyu Wang, Tianshuo Yan +4Tucker Decomposition LLM Compression
University of Manitoba · McGill University · Simpleway +2
Tensor Decomposition of Transformer Key-Value Caches: Spectral Structure and Format Comparison
Sep 23, 2026Rahul Krishnan, Volker SchulzTucker Decomposition Tensor Decomposition
Universität Trier · Fachbereich IV, Mathematik, Universität Trier
Self-Routed Tensor Adapters for Parameter-Efficient Universal Visual Adaptation
Aug 17, 2026Suraj YadavTucker Decomposition Vision Foundation Model Adaptation
Indraprastha Institute of Information Technology Delhi
FedDOSE: Federated Learning Framework Decomposing Site Effects for Modeling Brain Dynamic Functional Connectivity
Aug 7, 2026Deepank Girish, Yi Hao Chan, Yubin Zheng +2Tucker Decomposition Functional Connectivity
Nanyang Technological University · Shanghai Jiao Tong University · Indian Institute of Technology Ropar
Structural Loss Metrics for Tensor Approximation via Matrix Low-Rank Approximation
Jul 27, 2026Hiroki HasegawaTensor Decomposition Low-Rank Approximation
Graduate School of Science and Technology, University of Tsukuba, Tsukuba, 305-8573, Japan
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
Jul 14, 2026Rahul Krishnan, Volker SchulzTucker Decomposition KV Caching
Universität Trier · Fachbereich IV, Mathematik, Universität Trier
Robust Neural Tucker Factorization with Bias Correction and Adaptive Initialization
Jun 15, 2026Yuchao Su, Yixin RanTensor Completion Neural Network Initialization
School of Computer Science and Engineering, Chongqing University of Science and Technology, Chongqing, China · College of Computer and Information Science, School of Software, Southwest University, Chongqing, China
Spectra-Guided Neural Tucker Factorization
May 30, 2026Fusheng Wang, Yikai HouTensor Completion Tensor Decomposition
School of Automation, Chongqing University of Posts and Telecommunications, Chongqing, China · College of Computer and Information Science, School of Software, Southwest University, Chongqing, China
Unsupervised feature selection using Bayesian Tucker decomposition
Apr 16, 2026Y-h. Taguchi, Yoh-ichi MototakeTensor Decomposition Feature Selection
Department of Physics, Chuo University, Tokyo, 112-8551, Japan. · Graduate School of Social Data Science, Hitotsubashi University, Tokyo, 186-8601, Japan.
Tucker Attention: A generalization of approximate attention mechanisms
Mar 31, 2026Timon Klein, Jonas Kusch, Sebastian Sager +2Tucker Decomposition Attention Mechanisms
Department of Mathematics; Otto von Guericke University Magdeburg; 39106 Magdeburg; Germany. · Scientific Computing, Norwegian University of Life Sciences, ˚As, Norway · Max Planck Institute for Dynamics of Complex Technical Systems; 39106 Magdeburg; Germany. +1
Asymptotic Optimism for Tensor Regression Models with Applications to Neural Network Compression
Mar 27, 2026Haoming Shi, Eric C. Chi, Hengrui LuoModel Selection Tensor Decomposition
Department of Statistics, Rice University · School of Statistics, University of Minnesota
Adaptive regularization parameter selection for high-dimensional inverse problems: A Bayesian approach with Tucker low-rank constraints
Mar 17, 2026Qing-Mei Yang, Da-Qing ZhangTucker Decomposition Bayesian Inverse Problems
School of Science, University of Science and Technology Liaoning, Anshan, Liaoning Province, 114051, PR China
All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptation
Mar 15, 2026Xudong Wang, Gan Li, Zhiyu Liu +3Tucker Decomposition Vision-Language Navigation
State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · North University of China +1

Oct 1, 2026J/K move · Enter open · S save

ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models

Lianjun Liu, Shipeng Li, You Huang, Weiqi Yan+4

Hainan University, Haikou, China · Xiamen University, Xiamen, China · Zhejiang Normal University, Jinhua, China

Tucker Decomposition Mixture-of-Experts Language Models LLM Compression Mixture-of-Experts Inference Tensor Decomposition Diffusion Language Models

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) Diffusion Language Models (DLMs) offer flexible parallel decoding and increased model capacity, but their large number of expert parameters incurs substantial computation and storage costs. Existing low-rank MoE compression methods largely rely on static factorization and fixed rank allocation, which overlook the distinctive properties of MoE DLMs. Specifically, we identify two properties: cross-mode non-uniform redundancy, where parameter redundancy and sensitivity to rank truncation vary across the input, output, and expert modes, and token-wise utilization variation, where hot and cold tokens exhibit distinct spectral characteristics and expert activation patterns. To address these challenges, we propose ITC-MoE, an Importance-guided Token-aware Compression framework for MoE DLMs. ITC-MoE consists of two complementary components. First, Importance-guided Adaptive Tucker Compression (IATC) incorporates activation and gradient importance into expert weight transformation, jointly factorizes expert weights across multiple modes, and adaptively allocates ranks under a fixed parameter budget. Second, Token-aware Compensation and Routing (TCR) applies lightweight low-rank compensation to compression-sensitive hot tokens and restricts the candidate expert set for cold tokens with concentrated routing patterns. By jointly adapting compression capacity and inference execution to both parameter redundancy and token-wise variation, ITC-MoE substantially reduces the computation and storage costs of MoE DLMs while preserving their generation quality. For example, on SDAR-30B-A3B-Chat-b32, ITC-MoE maintains an accuracy of 96.33% on MultiArith under a 30% compression budget, while achieving up to a 7.22x end-to-end speedup. The code is publicly available at https://github.com/lianjunl13-sudo/ITC-MoE.