Low-Rank Approximation

Follow this topic Ask about this topic

Momentum

8 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 76

All topics

SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices
Jun 5, 2026Ernests Lavrinovics, Marco Letizia, Roy Janco +3Model Compression LLM Compression
Department of Computer Science, Aalborg University Copenhagen, Denmark · MaLGa-DIBRIS, University of Genoa, Genoa, Italy · INFN, Sezione di Genova, Genoa, Italy +2
Low-rank Distributional Matrix Completion
Jun 2, 2026Jiayi Wang, Raymond K. W. WongLow-Rank Approximation Matrix Completion
University of Texas at Dallas · Texas A&M University
Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter
Jun 2, 2026Zhengbao He, Ruiqi Ding, Zhehao Huang +3Adapter Tuning Low-Rank Adaptation
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China · Shanghai Key Laboratory of Flexible Medical Robotics, Tongren Hospital, Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China
Riemannian Gradient Descent for Low-Rank Architectures
Jun 1, 2026Nicholas KnightRiemannian Optimization Low-Rank Approximation
NVIDIA
Riemannian Optimization for Hadamard Products of Low-Rank Matrices
May 31, 2026Pratik Jawanpuria, Ankish Chandresh, Bamdev MishraRiemannian Optimization Low-Rank Approximation
Centre for Machine Intelligence and Data Science, Indian Institute of Technology Bombay, India · Microsoft India
Improved Guarantees for Heterogeneous Treatment-Effect Estimation via Matrix Completion
May 28, 2026Anay Mehrotra, Phuc Tran, Van H. Vu +1Causal Effect Estimation Conditional Average Treatment Effect Estimation
Vin University · The University of Hong Kong · Yale University
Manifold-based Algorithms for the Hadamard Decomposition
May 27, 2026Nicolas Gillis, Subhayan Saha, Stefano Sicilia +1Low-Rank Matrix Decomposition Low-Rank Approximation
Department of Mathematics and Operational Research, University of Mons, Mons, Belgium. · SS is a member of the Gruppo Nazionale Calcolo Scientifico-Istituto Nazionale di Alta Matematica (GNCS-INdAM).
Implicit Regularization in Perturbed Deep Matrix Factorization: Spectral Conditions and Stability
May 27, 2026Jingzhe Wang, Hung-Hsu ChouAlgorithmic Stability Low-Rank Matrix Decomposition
Department of Informatics and Networked Systems, University of Pittsburgh · Department of Mathematics, University of Pittsburgh
Efficient Pre-Training of LLMs through Truncated SVD Representations
May 27, 2026Kaivan Kamali, Kajetan Schweighofer, Hormoz Shahrzad +3Language Model Pretraining Low-Rank Matrix Decomposition
Cognizant AI Lab
You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories
May 20, 2026Zhepei Wei, Xinyu Zhu, Wei-Lin Chen +3RL for Language Model Reasoning Low-Rank Approximation
University of Virginia · Washington University in St. Louis
Group-Aware Matrix Estimation and Latent Subspace Recovery
May 19, 2026Hamza Golubovic, Matthew Shen, Genevera I. Allen +1Low-Rank Matrix Decomposition Nuclear Norm Minimization
Department of Statistics, Columbia University New York, NY 10027 · Department of Statistics, Irving Institute for Cellular Dynamics, Zuckerman Institute Columbia University New York, NY 10027 · School of Data and Information Sciences, University of North Carolina at Chapel Hill Chapel Hill, NC 27599
Robust Subspace-Constrained Quadratic Models for Low-Dimensional Structure Learning
May 19, 2026Zheng Zhai, Xiaohui LiHeavy-Tailed Noise Low-Rank Approximation
Department of Statistics, Faculty of Arts and Sciences at Beijing Normal University, Zhuhai. · School of Mathematics and Information Sciences, Yantai University.
An Exterior Method for Nonnegative Matrix Factorization
May 19, 2026Qiujing Lu, Tonmoy Monsoor, Ehsan Ebrahimzadeh +2Non-Negative Matrix Factorization Low-Rank Matrix Decomposition
1ECE, UCLA · 2eBay Search Science Team
LoCO: Low-rank Compositional Rotation Fine-tuning
May 15, 2026An Nguyen, Jaesik Choi, Anh TongFine-Tuning LLM Fine-Tuning
1Korea University · 2KAIST · 3INEEJI
Beyond Perplexity: A Geometric and Spectral Study of Low-Rank Pre-Training
May 13, 2026Namrata Shivagunde, Vijeta Deshpande, Sherin Muckatira +1Language Model Pretraining Low-Rank Approximation
University of Massachusetts Lowell
Accelerating Power Method with Fast Sketching for Stronger Low-Rank Approximation
May 10, 2026Shabarish Chenakkod, Michał DerezińskiLow-Rank Matrix Decomposition Randomized Sketching
University of Michigan, Ann Arbor, MI 48105.
Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compression
May 9, 2026Hengyi Zhu, Zhendong Mi, Grace Li Zhang +1LLM Compression Low-Rank Matrix Decomposition
1Stevens Institute of Technology, Hoboken, USA · 2Technical University of Darmstadt, Darmstadt, Germany
Don't Learn the Shape: Forecasting Periodic Time Series by Rank-1 Decomposition
May 8, 2026Takato HondaLow-Rank Matrix Decomposition Time Series Forecasting
Mellon Inc.
Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization
May 7, 2026Tatsuhiro Nakamori, Laura Gomezjurado Gonzalez, Ganesh Talluri +5Communication-Efficient Distributed Training Distributed Optimization
1Keio University · 2Stanford University · 3Midwestern University +3
Structural Correspondence and Universal Approximation in Diagonal plus Low-Rank Neural Networks
May 7, 2026Ying Chen, Aoxi Li, Jihun Kim +1Low-Rank Matrix Decomposition Low-Rank Approximation
Department of Industrial Engineering & Operations Research University of California, Berkeley, CA 94720
Low Rank Tensor Completion via Adaptive ADMM
May 5, 2026Niclas Führling, Getuar Rexhepi, Giuseppe Thadeu Freitas de AbreuTensor Completion Nuclear Norm Minimization
School of Computer Science and Engineering, Constructor University, Bremen, Germany
ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
May 5, 2026Jiaxi Li, Lu Yin, Li Shen +5Language Model Pretraining Activation Sparsity
University of Surrey · Sun Yat-sen University · Bytedance +2
Active multiple matrix completion with adaptive confidence sets
May 4, 2026Andrea Locatelli, Alexandra Carpentier, Michal ValkoActive Learning Multi-Task Learning
OvGU Magdeburg · SequeL team, Inria Lille
Importance-Guided Basis Selection for Low-Rank Decomposition of Large Language Models
May 2, 2026Daniel Agyei Asante, Ernie Chang, Yang LiLLM Pruning LLM Compression
Iowa State University · Meta
Quaternion Nonlinear Transform-Induced Nuclear Norm for Low-Rank Tensor Completion
May 2, 2026Biswarup Karmakar, Ratikanta BeheraTensor Completion Tensor Decomposition
Department of Computational and Data Sciences, Indian Institute of Science, Bangalore, 560012, India
Post-Optimization Adaptive Rank Allocation for LoRA
Apr 30, 2026Vishnuprasadh Kumaravelu, Sunil Gupta, P. K. SrijithLow-Rank Adaptation Low-Rank Compression
1Indian Institute of Technology Hyderabad · 2Deakin University
Low Rank Adaptation for Adversarial Perturbation
Apr 30, 2026Han Liu, Shanghao Shi, Yevgeniy Vorobeychik +2Adversarial Attacks Low-Rank Approximation
Washington University in St. Louis
Hankel and Toeplitz Rank-1 Decomposition of Arbitrary Matrices with Applications to Signal Direction-of-Arrival Estimation
Apr 29, 2026Georgios I. OrfanidisLow-Rank Matrix Decomposition Direction-of-Arrival Estimation
Center for Connected Autonomy and AI and the Dept. of Electrical Engineering and Computer Science, Florida Atlantic University, Boca Raton, FL 33431 USA
Hierarchical Spatio-Channel Clustering for Efficient Model Compression in Medical Image Analysis
Apr 25, 2026Sisipho Hamlomo, Marcellin Atemkeng, Habte Tadesse Likassa +5Low-Rank Compression Low-Rank Approximation
Department of Mathematics, Rhodes University, Makhanda 6140, South Africa · Department of Statistics, Rhodes University, Makhanda 6140, South Africa · National Institute for Theoretical and Computational Sciences, Stellenbosch 7600, South Africa +8
Pack only the essentials: Adaptive dictionary learning for kernel ridge regression
Apr 24, 2026Daniele Calandriello, Alessandro Lazaric, Michal ValkoKernel Ridge Regression Kernel Regression
SequeL team, INRIA Lille - Nord Europe, France

Jun 5, 2026, cs.CLJ/K move · Enter open · S save

SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices

Ernests Lavrinovics, Marco Letizia, Roy Janco, Shai Segal+2

Department of Computer Science, Aalborg University Copenhagen, Denmark · MaLGa-DIBRIS, University of Genoa, Genoa, Italy · INFN, Sezione di Genova, Genoa, Italy+2

Model Compression LLM Compression Low-Rank Matrix Decomposition LLM Inference Acceleration Low-Rank Compression Low-Rank Approximation

Open paper AskPDF arXiv

We present SigmaScale, a method for learning auxiliary scaling matrices

S

to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression. Instead of deriving scaling matrices analytically, SigmaScale optimizes two sets of vectors that define diagonal row and column scaling transformations under an activation-aware compression loss. We show that learned scaling lowers the effective intrinsic rank of weight matrices, as reflected by reductions in effective-rank entropy, and that this reduction is strongly correlated with compression loss. Experiments on Llama 3.1 8B Instruct and Qwen3-8B show that SigmaScale is competitive with closely related state-of-the-art SVD-based compression methods across perplexity and zero-shot benchmarks. By using learned activation-aware transformations, SigmaScale explores a more flexible route to low-rank LLM compression by adapting to the structure of individual model weights. The advantage observed in specific tasks makes our approach a valid option for applications requiring a reduced LLM-inference computing cost.

Momentum

Related topics

Latest papers 76