Singular Value Decomposition

Also known as SVD

Singular value decomposition expresses a matrix through orthogonal directions ordered by their importance, making it a standard tool for finding low-dimensional structure in data and models. Work spans randomized and incremental algorithms for faster decompositions, tensor and functional variants, and applications such as clustering, imaging, reduced-order models, and language-model compression. In neural networks, researchers use SVD to compress weights and attention caches, guide parameter-efficient adaptation, and study how spectral structure relates to optimization and model behavior. A central challenge is choosing ranks and components that preserve downstream performance, since minimizing matrix reconstruction error alone may not preserve a system’s task performance, while more accurate or adaptive methods can cost additional computation.

Latest papers

36 papers6 papers in the last four weeks, down 33% on the four weeks before. 0.0% of all new papers.

Training-free Acceleration for Blind Compressed Image Restoration via Activation-Calibrated Singular Value Decomposition
Oct 21, 2026Suhang Yao, Xin Li, Xijun Wang +3Inference Acceleration Image Restoration
University of Science and Technology of China, Hefei, China · SparcAI Inc. · National University of Singapore, Singapore
D-SLR: The Disjoint Row-Sparse plus Low-Rank Decomposition
Oct 7, 2026Vincent SzolnokyLow-Rank Matrix Decomposition Low-Rank Compression
Department of Mathematical Sciences Chalmers University of Technology
Singular Value Decomposition: A Geometric Rediscovery, Where Proofs Become Algorithms
Oct 6, 2026Paul AgronSingular Value Decomposition
AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD
Oct 3, 2026Sara Abdali, Jongwoo Ko, Pashmina CameronKV-Cache Compression Low-Rank Compression
Microsoft Applied Sciences Group (ASG)
Kernel Singular Value Decomposition with Extension to Multiple Data Sources
Oct 2, 2026Xinjie Zeng, Qinghua Tao, Johan SuykensKernel Methods Singular Value Decomposition
KU Leuven - ESAT-STADIUS Kasteelpark Arenberg 10, B-3001 Heverlee - Belgium
iS-KV: Online Low-Rank KV Cache Compression via Block-Incremental SVD
Oct 2, 2026Yiren Zhao, Guanghui Song, Tianrui Qin +3KV-Cache Compression Low-Rank Compression
The Hong Kong University of Science and Technology (Guangzhou) · Guangdong OPPO Mobile Telecommunications Corp., Ltd. · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences +2
Fast Differentiable SVD on GPU via Polar Decomposition
Sep 26, 2026Uliana Parkina, Askar Tsyganov, Sergei Kudriashov +2Singular Value Decomposition Matrix Factorization
HSE University
Prescriptive SVD-Inspired Attention via Spectral Energy Retention
Sep 21, 2026Vasileios Arampatzakis, Vasileios Sevetlidis, George PavlidisTransformer Interpretability Low-Rank Attention
Athena Research Center, Greece
Randomized SVD Approximations for Spectral Co-Clustering of Word-Document Matrices
Sep 16, 2026Fateme Mazdarani, Carlos ToxtliClustering Randomized Sketching
School of Computing Clemson University Clemson, SC, USA
Per-Matrix Optimality Is Not Enough: Three-Level Optimization for Low-Rank LLM Compression
Sep 14, 2026Huicheng Zhang, Xiyao Feng, Ze-Tong Li +6Model Compression LLM Compression
Hong Kong University of Science and Technology (Guangzhou), Guangdong 511453, China · QudeLeap Research, Shanghai 200030, China
A Functional SVD Framework for Regularized Multivariate Functional PCA with Dual Penalization
Sep 13, 2026Yue Zhao, Hossein Haghbin, Rebecca Sanders +1Functional Data Analysis Principal Component Analysis
Division of Biostatistics and Health Data Science, University of Minnesota, USA · Faculty of Intelligent Systems Engineering and Data Science, Persian Gulf University, Iran · Department of Mathematical and Statistical Sciences, Marquette University, USA
Semi-Tensor Product-Based Multi-Term Randomized T-SVD and Its Visual Applications
Sep 10, 2026Xingchen Xiao, Feng Zhang, Wenjin Qin +1Tensor Completion Tensor Decomposition
School of Mathematics and Statistics, Southwest University, Chongqing, China
Mind the Approximation: Fisher-Weighted SVD Compression for ViTs
Sep 7, 2026Moritz Thoma, Maximilian Groezinger, Maximilian Forstenhäusler +7Efficient ViTs Low-Rank Compression
Technical University of Munich, DE · BMW Group, DE · University of Glasgow, GB +1
Linear Algebra Foundations of Efficient Attention: A Phase Reversal in Rank Collapse Under SVD Compression
Sep 6, 2026Anjaneya Teja Sarma KalvakolanuLow-Rank Compression Self-Attention
Compact and Infinite-Order Error Analysis for Null-Space SVD Estimation
Aug 31, 2026Xin Li, Jonathan Cohen, Rami PuzisRandom Matrix Theory Singular Value Decomposition
Quasi-SVD: Learning a Lie-constrained matrix factorisation for real-time imaging
Jul 28, 2026Christopher HahneLie Group Methods Computational Imaging
University of Bern 3008 Bern, Switzerland
LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression
Jul 3, 2026Zhuowen Liu, Longkun Hao, Shiyu Feng +3Model Compression LLM Compression
Zhuowen Liu1 · Longkun Hao2 · Shiyu Feng3 +3
CORA: Per-Slice Coherent Orthogonal Rotation for SVD-based Low-Rank Adaptation
Jun 30, 2026Pengcheng Wang, Ziran Liu, Wei Wang +1LLM Fine-Tuning Low-Rank Matrix Decomposition
Purdue University · Shanghai Institute for Mathematics and Interdisciplinary Sciences / Fudan University · Futurewei Technologies, Inc.
SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression
Jun 22, 2026Mahmoud Safari, Frank HutterLLM Pruning LLM Compression
University of Freiburg · Prior Labs · ELLIS Institute Tübingen
Sparse Functional Singular Value Decomposition for Biclustering and Triclustering Longitudinal Data
Jun 3, 2026Yue Zhao, Thierry Chekouo, Sandra SafoFunctional Data Analysis Clustering
Division of Biostatistics and Health Data Science University of Minnesota 2221 University Avenue SE2026 Minneapolis, MN 55414, United States
LSVD: Loss-Aware Low-Rank Approximation for Efficient Low-Precision Vision-Language Models
May 30, 2026Haiyu Wang, Yutong Wang, Leshu Li +2Vision-Language Models Efficient VLM Inference
Tandon School of Engineering, New York University · Courant Institute of Mathematical Sciences, New York University · Brookhaven National Laboratory
History-aware adaptive reduced-order models via incremental singular value decomposition
May 27, 2026Amirpasha Hedayat, Ali Mohaghegh, Laura Balzano +2Reduced-Order Modeling Singular Value Decomposition
Department of Aerospace Engineering, University of Michigan, Ann Arbor, MI, USA · Department of Aerospace Engineering, University of Kansas, Lawrence, KS, USA · Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI, USA
Efficient Pre-Training of LLMs through Truncated SVD Representations
May 27, 2026Kaivan Kamali, Kajetan Schweighofer, Hormoz Shahrzad +3Language Model Pretraining Low-Rank Matrix Decomposition
Cognizant AI Lab
Exact Symmetry as Algebra: Group-Structured Tensors that Enforce Physical Selection Rules
May 19, 2026Paulina Hoyos, Shashanka Ubaru, Dongsung Huh +5Scientific Discovery Symmetry Discovery
UT Austin · IBM Research · Independent +2
Accelerating Power Method with Fast Sketching for Stronger Low-Rank Approximation
May 10, 2026Shabarish Chenakkod, Michał DerezińskiLow-Rank Matrix Decomposition Nyström Method
University of Michigan, Ann Arbor, MI 48105.
Phases of Muon: When Muon Eclipses SignSGD
May 10, 2026Elliot Paquette, Noah Marshall, Lucas Benigni +3Sign-Based Optimization Muon Optimizer
Mathematics and Statistics Department, McGill University · Mathematics and Statistics Department, Université de Montréal · Google DeepMind
FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast
May 8, 2026Wenhao Wu, Zishan Shao, Kangning Cui +5Efficient Transformer Inference LLM Compression
Department of Electrical & Computer Engineering, Duke University · Department of Statistical Science, Duke University · Department of Computer Science, Wake Forest University +1
Sparse Random-Feature Neural Networks with Krylov-Based SVD for Singularly Perturbed ODE
May 8, 2026Kevin Kurian Thomas Vaidyan, Siddharth RoutNeural PDE Solvers Singular Value Decomposition
Institute of Applied Mathematics University of British Columbia Vancouver, BC V6T 1Z4
A Generalized Singular Value Theory for Neural Networks
May 7, 2026Brian Charles Brown, Robert Bridges, David Grimsman +2Neural Representation Geometry Geometric Representation Learning
AI Sweden Gothenburg, Sweden
Structured 3D-SVD: A Practical Framework for the Compression and Reconstruction of Biological Volumetric Images
Apr 18, 2026Mario Aragonés Lozano, Oscar Romero, Antonio LeónImage Reconstruction Tensor Decomposition
Communications Department, Universitat Politècnica de València, 46022 Valencia, Spain

Oct 21, 2026J/K move · Enter open · S save

Training-free Acceleration for Blind Compressed Image Restoration via Activation-Calibrated Singular Value Decomposition

Suhang Yao, Xin Li, Xijun Wang, Yufei Wang+2

University of Science and Technology of China, Hefei, China · SparcAI Inc. · National University of Singapore, Singapore

Inference Acceleration Image Restoration Singular Value Decomposition Low-Rank Compression

Open paper AskPDF arXiv

Transformer-based models achieve strong performance on blind compressed image restoration (CIR) under unknown compression settings, but their densely stacked projection operators make deployment costly. We study training-free acceleration for such models through an empirical, operator-level analysis. Specifically, we profile where computation is concentrated, characterize channel-wise activation distributions under compressed inputs, and examine the singular-value decay of representative projection weights. These measurements identify the recurring projections in the attention, feed-forward, convolutional, and prompt-conditioning pathways as effective low-rank replacement targets that preserve the original inference graph. They further establish truncated SVD (TSVD) as a principled training-free baseline for blind CIR and reveal why a uniform low-rank objective is mismatched to the channel-imbalanced feature distributions of compressed images. Accordingly, we formulate Activation-Calibrated Singular Value Decomposition (ASVD) for this restoration setting, deriving low-rank factors entirely offline from pretrained weights and lightweight activation statistics gathered from only a few compressed calibration images. By weighting the decomposition with channel-wise activation magnitudes, our ASVD assigns larger penalties to errors on more active channels, requiring neither retraining nor online calibration and leaving the restoration pipeline intact. When instantiated on PromptCIR at rank 64, ASVD consistently improves PSNR-B over TSVD across four standard benchmarks under identical parameter and MAC budgets. At this operating point, the factorized model cuts parameters by 59% (34.75M to 14.34M) and computation by 16.1% (258.79 to 217.16 GMACs for a 256 x 256 input), enabling acceleration without retraining.