Low-Rank Compression

Ask about this topic

Overview

Low-rank compression represents large matrices and tensors with smaller factors, reducing storage and sometimes the computation needed for training or inference. Common approaches include truncated or randomized SVD, activation- and loss-aware factorization, adaptive rank allocation, and combinations with quantization or sparsity; some methods instead train models or optimizer states directly in low-rank form. Work spans vision and language models, with growing attention to long-context key-value caches, where benchmarks such as LongBench and RULER test whether compressed representations preserve retrieval and reasoning. A central tension is that low reconstruction error does not always preserve task behavior, and fewer parameters do not guarantee faster execution without suitable kernels and workload-aware evaluation.

Latest papers

CardsList

108 papers22 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

  1. Hierarchical Low-Rank Reparameterization for Parameter-Efficient

    Oct 27, 2026Rong Huang, Yi Xiong, Yuan-Yuan Cheng +1LLM Fine-TuningParameter-Efficient Fine-Tuning

    University of Science and Technology of China, Hefei, Anhui 230026, China

  2. LoRA on a Budget: Function-Aware Post-Hoc Compression for Adapter Fleets

    Oct 26, 2026Shenghui Li, Thiemo voigtLow-Rank CompressionAdaptive Rank Allocation

    Uppsala University, Uppsala, Sweden · Research Institutes of Sweden, Stockholm, Sweden

  3. Low-rank MERA Decomposition of 3D OCT Image Volume for Compression, Denoising, Feature Extraction and Diagnosis of Retinal Diseases

    Oct 26, 2026Ivica Kopriva, Fei Shi, Marija Štanfel +4Optical Coherence TomographyMedical Image Representation Learning

    Ivica Kopriva is with Division of Computing and Data Science, Rudjer Boskovich Institute, Zagreb,Croatia · Fei Shi and Gaowei Li are with the School of Electronics and Information Engineering, Soochow University, Suzhou, China, and the Jiangsu Key Laboratory of Artificial Intelligence and Medical Imaging, Soochow University, Suzhou, China · Marija Štanfel is with Department of Ophthalmology, University Hospital Center Zagreb, Zagreb, Croatia +2

  4. Subspace Crystallization in Grokking: Rank Compression Tracks the Phase Transition

    Oct 26, 2026Shlok Mehendale, Paras ChopraGrokkingNeural Network Training Dynamics

    Lossfunk

  5. Advancing SVD-based LLM Compression via Layer-Wise Error Model Search

    Oct 23, 2026Moritz Thoma, Maximilian Groezinger, Maximilian Forstenhäusler +8LLM CompressionLow-Rank Compression

    Technical University of Munich, Germany · BMW Group, Germany · University of Glasgow, United Kingdom +1

  6. BiLoCo: Binary Low-Rank Corrections for Post-Training Quantization

    Oct 23, 2026David Jin, Beshr IslamBouli, Tarushii Goel +2LLM QuantizationPost-Training Quantization

    Massachusetts Institute of Technology

  7. GS-MoE: Training-Free MoE Compression via Groupwise Subspace Sharing and Pruning

    Oct 23, 2026Carson Cary, Yukai Jin, Yihao Huang +5Mixture-of-Experts ModelsMixture-of-Experts Pruning

    Florida State University · University of Pennsylvania · University of Notre Dame

  8. Training-free Acceleration for Blind Compressed Image Restoration via Activation-Calibrated Singular Value Decomposition

    Oct 21, 2026Suhang Yao, Xin Li, Xijun Wang +3Inference AccelerationImage Restoration

    University of Science and Technology of China, Hefei, China · SparcAI Inc. · National University of Singapore, Singapore

  9. A KAN-based Non-linear Factorization for Embedding Layer Compression

    Oct 21, 2026Oliver Giudice, Anna Russo RussoEmbedding CompressionLow-Rank Compression

    Applied Research Team, IT Dept. Banca d’Italia *−

  10. LoRASpace: A Pool-Wide Shared Substrate for Static and Dynamic Multi-LoRA Composition

    Oct 20, 2026Thamirawaran Sathiyalogeswaran, Navindu De Silva, Chamikara Siriwardane +4Low-Rank AdaptationLLM Inference Acceleration

    University of Moratuwa, Sri Lanka · University of Michigan, USA

  11. BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation

    Oct 13, 2026Yaoyang Liu, Shuwei Shen, Jincheng Tang +5KV-Cache CompressionLow-Rank Compression

    Peking University · Tencent · BeiHang University

  12. GA-SVD: Generation-aware Singular Value Decomposition for LLM Compression

    Oct 11, 2026Shengkun Tang, Cong Zeng, Zongfang Liu +2LLM CompressionLow-Rank Compression

    MBZUAI · Technology Innovation Institute UAE · Zhejiang Univeristy +2

  13. LightVAE: Towards Compact and Efficient Video Autoencoders

    Oct 10, 2026Shiqiao Gu, Yang Yong, Fengjun Zhong +5Variational AutoencodersVideo Compression

    SenseTime · Beihang University · Nanyang Technological University

  14. Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights

    Oct 8, 2026Dmitrii Andriianov, Andrey Veprikov, Aleksandr BeznosikovWeight DecayNuclear Norm Minimization

    Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · SB AI Lab · Innopolis University

  15. D-SLR: The Disjoint Row-Sparse plus Low-Rank Decomposition

    Oct 7, 2026Vincent SzolnokyLow-Rank Matrix DecompositionLow-Rank Compression

    Department of Mathematical Sciences Chalmers University of Technology

  16. LRCC: Generalizing Low-Rank Compression with Conditional Computation

    Oct 5, 2026Thomas Vaitses Fontanari, Maximo Eduardo Rulli, Federico Alvetreti +2LLM CompressionLow-Rank Compression

    Sapienza University of Rome

  17. The Functional Structure of Post-Compression Recovery in Low-Rank LLMs

    Oct 4, 2026Zishan Shao, Liang Tian, Georgiy Zemlevskiy +10LLM CompressionLow-Rank Compression

    Duke University · Wake Forest University · Carnegie Mellon University +2

  18. AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD

    Oct 3, 2026Sara Abdali, Jongwoo Ko, Pashmina CameronKV-Cache CompressionLow-Rank Compression

    Microsoft Applied Sciences Group (ASG)

  19. Preserving Mathematical Reasoning in Compressed Diffusion Language Models via Trajectory-Aware Low-Rank Approximation

    Oct 2, 2026Tian Liang, Zishan Shao, Yiran ChenDiffusion Model CompressionLow-Rank Compression

    Department of Electrical and Computer Engineering, Duke University

  20. SlimKV: Joint Token-Feature KV Cache Compression with Reconstruction-Free Beacon Attention

    Oct 2, 2026Zihan Teng, Jiayu Zhao, Wentao Ren +4KV-Cache CompressionLLM Inference Acceleration

    University of Science and Technology of China · Nanyang Technological University · Tencent

  21. iS-KV: Online Low-Rank KV Cache Compression via Block-Incremental SVD

    Oct 2, 2026Yiren Zhao, Guanghui Song, Tianrui Qin +3KV-Cache CompressionLow-Rank Compression

    The Hong Kong University of Science and Technology (Guangzhou) · Guangdong OPPO Mobile Telecommunications Corp., Ltd. · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences +2

  22. Sequential Functional Structured Tucker Compression for Large Language Model Attentions

    Sep 30, 2026Jiangfeng Chen, Xinyu Wang, Tianshuo Yan +4Tucker DecompositionLLM Compression

    University of Manitoba · McGill University · Simpleway +2

  23. Learning Functional Subspaces for Neural Network Compression

    Sep 30, 2026Massimo Bini, Anders Christensen, Stephan Alaniz +5LLM CompressionLow-Rank Matrix Decomposition

    Helmholtz Munich · Technical University of Munich · MCML +6

  24. Beyond Uniform Compression: Budgeted Transmission Allocation for Extreme Federated Learning

    Sep 30, 2026Pengfei Li, Mohammad KhalilCommunication-Efficient Distributed TrainingLow-Rank Compression

    Centre for the Science of Learning & Technology (SLATE), University of Bergen, Bergen, Norway

  25. KV-Kaizen: Learning Context-Adaptive Cache Compression Choices

    Sep 29, 2026Joao Monteiro, Louis Béthune, Anastasiia Filippova +3LLM Inference AccelerationLow-Rank Compression

    Apple

  26. Pre-registered tests of solid-state-physics-inspired LLM compression: a cluster-level negative result at small-language-model scale

    Sep 28, 2026Jun-qiang LuTensor NetworksLLM Compression

    Department of Physics, University of Puerto Rico, Mayagüez, PR 00680, USA

  27. Freeze the Decoder, Heal the Encoder: Parameter-Efficient Adaptation for SVD-Based KV-Cache Compression

    Sep 25, 2026Yufeng WangKV-Cache CompressionParameter-Efficient Fine-Tuning

  28. MILO: Efficient Many-shot In-Context Learning with Block-wise Low-rank Compression

    Sep 24, 2026Youpeng Zhao, Tian Tan, Liqian Peng +2Memory-Efficient InferenceLLM Inference Acceleration

    Google · University of Central Florida

  29. Low-Rank Friction for Memory-Efficient Transformer Pretraining

    Sep 24, 2026Rajit Rajpal, Benedict LeimkuhlerDeep Learning OptimizationLow-Rank Compression

    School of Mathematics University of Edinburgh Edinburgh, UK EH9 3FD

  30. GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression

    Sep 22, 2026Baher Mohammad, Ammar Ali, Stamatios LefkimmiatisModel CompressionLow-Rank Compression

    MWS AI, ITMO University · MWS AI