Low-Rank Compression

Ask about this topic

Overview

Low-rank compression represents large matrices and tensors with smaller factors, reducing storage and sometimes the computation needed for training or inference. Common approaches include truncated or randomized SVD, activation- and loss-aware factorization, adaptive rank allocation, and combinations with quantization or sparsity; some methods instead train models or optimizer states directly in low-rank form. Work spans vision and language models, with growing attention to long-context key-value caches, where benchmarks such as LongBench and RULER test whether compressed representations preserve retrieval and reasoning. A central tension is that low reconstruction error does not always preserve task behavior, and fewer parameters do not guarantee faster execution without suitable kernels and workload-aware evaluation.

Latest papers

CardsList

108 papers22 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

  1. EinSort: Sorting is All We Need for Tensorizing LLM

    Jun 7, 2026Toshiaki Koike-Akino, Jing Liu, Ye WangTensor NetworksLLM Compression

    Mitsubishi Electric Research Laboratories (MERL), 20 Broadway, Cambridge, MA 02139, USA.

  2. STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control

    Jun 7, 2026Priyansh Bhatnagar, Ashkan Moradifirouzabadi, Se-Hyun Yang +3Mixed-Precision QuantizationKV Caching

    University of California San Diego, San Diego, CA, USA · Dnotitia Inc., Seoul, Republic of Korea · Hanyang University, Seoul, Republic of Korea

  3. SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices

    Jun 5, 2026Ernests Lavrinovics, Marco Letizia, Roy Janco +3Model CompressionLLM Compression

    Department of Computer Science, Aalborg University Copenhagen, Denmark · MaLGa-DIBRIS, University of Genoa, Genoa, Italy · INFN, Sezione di Genova, Genoa, Italy +2

  4. Learned Subspace Compression for Communication-Efficient Pipeline Parallelism

    Jun 3, 2026Paul Janson, Edouard Oyallon, Eugene BelilovskyLLM CompressionCommunication-Efficient Distributed Training

    Concordia University · Mila Quebec AI Institute · CNRS, Sorbonne University

  5. Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter

    Jun 2, 2026Zhengbao He, Ruiqi Ding, Zhehao Huang +3Adapter TuningLow-Rank Adaptation

    Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China · Shanghai Key Laboratory of Flexible Medical Robotics, Tongren Hospital, Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China

  6. GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation

    May 31, 2026Shihao Zhang, Rayan SaabLLM QuantizationLow-Rank Adaptation

    Department of Mathematics, University of California San Diego

  7. LSVD: Loss-Aware Low-Rank Approximation for Efficient Low-Precision Vision-Language Models

    May 30, 2026Haiyu Wang, Yutong Wang, Leshu Li +2Vision-Language ModelsEfficient VLM Inference

    Tandon School of Engineering, New York University · Courant Institute of Mathematical Sciences, New York University · Brookhaven National Laboratory

  8. Cross-Layer Subspace Coupling for LLM Compression: A Unifying Framework and Its Empirical Limits

    May 29, 2026Snigdha Chandan KhilarModel CompressionLLM Compression

    Independent Researcher

  9. Efficient Pre-Training of LLMs through Truncated SVD Representations

    May 27, 2026Kaivan Kamali, Kajetan Schweighofer, Hormoz Shahrzad +3Language Model PretrainingLow-Rank Matrix Decomposition

    Cognizant AI Lab

  10. SAFE-SVD: Sensitivity-Aware Fidelity-Enforcing SVD for Physics Foundation Models

    May 18, 2026Chengjie Hong, Feixiang He, Yiheng Zeng +2Model CompressionPhysical Foundation Models

    University College London, UK · Central South University, China · University of Massachusetts at Amherst, US +1

  11. IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression

    May 15, 2026Ali Abbasi, Chayne Thrash, Haoran Qin +2LLM CompressionLow-Rank Matrix Decomposition

    Vanderbilt University · University of California, Davis

  12. LRCP: Low-Rank Compressibility Guided Visual Token Pruning for Efficient LVLMs

    May 15, 2026Hongyu Lu, Feng Zhang, Wenwei Jin +5Efficient VLM InferenceLarge Vision-Language Models

    1Xiaohongshu · 2Harbin Institute of Technology · 3Fudan University

  13. Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compression

    May 9, 2026Hengyi Zhu, Zhendong Mi, Grace Li Zhang +1LLM CompressionLow-Rank Matrix Decomposition

    1Stevens Institute of Technology, Hoboken, USA · 2Technical University of Darmstadt, Darmstadt, Germany

  14. FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast

    May 8, 2026Wenhao Wu, Zishan Shao, Kangning Cui +5Efficient Transformer InferenceLLM Compression

    Department of Electrical & Computer Engineering, Duke University · Department of Statistical Science, Duke University · Department of Computer Science, Wake Forest University +1

  15. Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization

    May 7, 2026Tatsuhiro Nakamori, Laura Gomezjurado Gonzalez, Ganesh Talluri +5Communication-Efficient Distributed TrainingDistributed Optimization

    1Keio University · 2Stanford University · 3Midwestern University +3

  16. Budgeted LoRA: Distillation as Structured Compute Allocation for Efficient Inference

    May 5, 2026Mohammed Sabry, Anya BelzLLM CompressionLow-Rank Adaptation

    ADAPT Centre, Dublin City University, Ireland

  17. ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity

    May 5, 2026Jiaxi Li, Lu Yin, Li Shen +5Language Model PretrainingActivation Sparsity

    University of Surrey · Sun Yat-sen University · Bytedance +2

  18. Compress Then Adapt? No, Do It Together via Task-aware Union of Subspaces

    May 4, 2026Jingze Ge, Yun Liu, Xue Geng +4Model CompressionAdapter Tuning

    National University of Singapore · Nankai University · Institute for Infocomm Research (I2R), A*STAR

  19. Importance-Guided Basis Selection for Low-Rank Decomposition of Large Language Models

    May 2, 2026Daniel Agyei Asante, Ernie Chang, Yang LiLLM PruningLLM Compression

    Iowa State University · Meta

  20. Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm

    May 2, 2026Wen-Da Wei, Han-Bin Fang, Yang-Di Liu +3LLM CompressionGradient Compression

    Nanjing University, Nanjing, China · Tsinghua University, Beijing, China · Huazhong University of Science and Technology, Wuhan, China +1

  21. Post-Optimization Adaptive Rank Allocation for LoRA

    Apr 30, 2026Vishnuprasadh Kumaravelu, Sunil Gupta, P. K. SrijithLow-Rank AdaptationLow-Rank Compression

    1Indian Institute of Technology Hyderabad · 2Deakin University

  22. Hierarchical Spatio-Channel Clustering for Efficient Model Compression in Medical Image Analysis

    Apr 25, 2026Sisipho Hamlomo, Marcellin Atemkeng, Habte Tadesse Likassa +5Low-Rank CompressionLow-Rank Approximation

    Department of Mathematics, Rhodes University, Makhanda 6140, South Africa · Department of Statistics, Rhodes University, Makhanda 6140, South Africa · National Institute for Theoretical and Computational Sciences, Stellenbosch 7600, South Africa +8

  23. Predicting LLM Compression Degradation from Spectral Statistics

    Apr 20, 2026Mingxue XuLLM CompressionLow-Rank Matrix Decomposition

    Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom

  24. LASER: Low-Rank Activation SVD for Efficient Recursion

    Apr 19, 2026Ege Çakar, Ketan Ali Raghu, Lia ZhengLow-Rank Matrix DecompositionRecurrent Neural Networks

    Harvard University Cambridge, MA, USA

  25. ResBM: Residual Bottleneck Models for Low-Bandwidth Pipeline Parallelism

    Apr 13, 2026Alan Aboudib, Rodrigo Lopez Portillo A., Kalei Brady +1Activation CompressionCommunication-Efficient Distributed Training

    Macrocosmos AI

  26. Attention-Weighted Value Projection for KV-Cache Compression

    Apr 13, 2026Samuel SalfatiGrouped-Query AttentionKV Caching

    fraQtl AI Research

  27. SLaB: Sparse-Lowrank-Binary Decomposition for Efficient Large Language Models

    Apr 6, 2026Ziwei Li, Yuang Ma, Yi KangBinary QuantizationLLM Compression

    School of Microelectronics, University of Science and Technology of China, Hefei, Anhui, China

  28. eOptShrinkQ: Near-Lossless KV Cache Compression Through Optimal Spectral Denoising and Quantization

    Apr 6, 2026Pei-Chun SuKV-Cache CompressionKV-Cache Quantization

    Department of Applied and Computational Mathematics, Yale University, New Haven, CT, USA

  29. Low-Rank Compression of Pretrained Models via Randomized Subspace Iteration

    Apr 3, 2026Farhad Pourkamali-AnarakiLow-Rank CompressionLow-Rank Approximation

    Department of Mathematical and Statistical Sciences, University of Colorado Denver

  30. WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language Models

    Apr 2, 2026Haiyu Wang, Yutong Wang, Jack Jiang +1Efficient VLM InferenceLow-Rank Compression

    Tandon School of Engineering, New York University · Courant Institute of Mathematical Sciences, New York University