Low-Rank Compression

Ask about this topic

Overview

Low-rank compression represents large matrices and tensors with smaller factors, reducing storage and sometimes the computation needed for training or inference. Common approaches include truncated or randomized SVD, activation- and loss-aware factorization, adaptive rank allocation, and combinations with quantization or sparsity; some methods instead train models or optimizer states directly in low-rank form. Work spans vision and language models, with growing attention to long-context key-value caches, where benchmarks such as LongBench and RULER test whether compressed representations preserve retrieval and reasoning. A central tension is that low reconstruction error does not always preserve task behavior, and fewer parameters do not guarantee faster execution without suitable kernels and workload-aware evaluation.

Latest papers

CardsList

108 papers22 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

  1. AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression

    Apr 2, 2026Atul Kumar Sinha, François FleuretLLM CompressionLow-Rank Compression

    University of Geneva, Geneva, Switzerland · FAIR, Meta

  2. Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression

    Apr 2, 2026Ruoling Qi, Yirui Liu, Xuaner Wu +6LLM CompressionLow-Rank Compression

    Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Jiao Tong University · University of Maryland +2

  3. Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retraction

    Apr 1, 2026Björn Roman KohlbergerLLM TrainingMemory-Efficient Training

    EctoSpace, Dublin, Ireland

  4. Asymptotic Optimism for Tensor Regression Models with Applications to Neural Network Compression

    Mar 27, 2026Haoming Shi, Eric C. Chi, Hengrui LuoModel SelectionTensor Decomposition

    Department of Statistics, Rice University · School of Statistics, University of Minnesota

  5. High-Resolution Tensor-Network Fourier Methods for Exponentially Compressed Non-Gaussian Aggregate Distributions

    Mar 24, 2026Juan José Rodríguez-Aldavero, Juan José García-RipollLow-Rank CompressionTT Decomposition

    Institute of Fundamental Physics IFF-CSIC, Calle Serrano 113b, Madrid 28006, Spain

  6. Optimal low-rank stochastic gradient estimation for LLM training

    Mar 21, 2026Zehao Li, Tao Ren, Zishi Zhang +2LLM TrainingMemory-Efficient Training

    Guanghua School of Management, Peking University, Beijing, 100871, China · Stern School of Business, New York University, New York, NY 10012, USA

  7. SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression

    Mar 12, 2026Xinhao Huang, You-Liang Huang, Zeyi WenLLM CompressionLow-Rank Compression

    The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology

  8. A New Tensor Network: Tubal Tensor Train and Its Applications

    Mar 11, 2026Salman Ahmadi-Asl, Valentin Leplat, Anh-Huy Phan +1Tensor DecompositionTT Decomposition

    Lab of Machine Learning and Knowledge Representation, Innopolis University, Innopolis, Russia · Innopolis University, Innopolis, Russia · Skolkovo Institute of Science and Technology, Center for Artificial Intelligence Technology, Moscow, Russia +1

  9. DLRMamba: Distilling Low-Rank Mamba for Edge Multispectral Fusion Object Detection

    Mar 6, 2026Qianqian Zhang, Leon Tabaro, Ahmed M. Abdelmoniem +1Efficient Object DetectionState Space Models

    National Space Science Center, Chinese Academy of Sciences, Beijing, 101499, China · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, 100049, China · School of Electronic Engineering and Computer Science, Queen Mary University of London, London, E1 4NS, UK +1

  10. NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training

    Mar 4, 2026Hadi Mohaghegh Dolatabadi, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7LLM Input OptimizationLLM Compression

    Pluralis Research

  11. 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs

    Mar 2, 2026Mehdi Makni, Xiang Meng, Rahul MazumderLLM CompressionLow-Rank Compression

    Operations Research Center Massachusetts Institute of Technology

  12. Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation

    Feb 27, 2026Zhengbo Wang, Jian Liang, Ran He +2Memory-Efficient TrainingLow-Rank Compression

    University of Science and Technology of China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences +1

  13. PRAC: Principal-Random Subspace for LLM Activation Compression and Memory-Efficient Training

    Feb 26, 2026Yanyi Li, Yimu Zhang, Cong FangLLM CompressionMemory-Efficient Training

    School of Intelligence Science and Technology, Peking University, China · Institute for Artificial Intelligence, Peking University, China

  14. Stabilizing Native Low-Rank LLM Pretraining

    Feb 12, 2026Paul Janson, Edouard Oyallon, Eugene BelilovskyLanguage Model PretrainingLow-Rank Matrix Decomposition

    Concordia University, Montreal, Canada · Mila Quebec AI Institute, Montreal, Canada · Sorbonne University, CNRS, Paris, France

  15. SkipCat: Rank-Maximized Low-Rank Compression of Large Language Models via Shared Projection and Block Skipping

    Dec 15, 2025Yu-Chen Lu, Sheng-Feng Yu, Hui-Hsien Weng +5LLM CompressionLow-Rank Compression

    National Yang Ming Chiao Tung University · Macronix International Co., Ltd. · Skymizer Taiwan Inc. +1

  16. Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM

    Oct 7, 2025Ryan Solgi, Parsa Madinei, Jiayi Tian +4Model CompressionLLM Compression

    University of California-Santa Barbara, USA · Amazon, USA

  17. BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression

    Sep 29, 2025David González-MartínezLow-Rank Matrix DecompositionLow-Rank Compression

    Department of Computer Science and Artificial Intelligence, Universidad de Sevilla · Tübingen AI Center

  18. Optimizing Singular Spectrum for Large Language Model Compression

    Feb 20, 2025Dengjie Li, Tiancheng Shen, Yao Zhou +3Model CompressionLLM Compression