Low-Rank Matrix Decomposition

Follow this topic Ask about this topic

Low-rank matrix decomposition approximates a matrix using a small set of latent factors, revealing shared structure and reducing storage or computation. Common approaches include singular value decomposition, matrix and tensor factorization, robust principal component analysis, and matrix completion, with rank chosen to balance simplicity and fidelity. Applications range from recovering missing or corrupted data to compressing language and vision models, where results are assessed with measures such as reconstruction error, perplexity, and downstream accuracy. A central challenge is that the best rank and factorization depend on the data and task: compact models can be faster and smaller, but approximation errors may compound and harm performance.

Latest papers

63 papers11 papers in the last four weeks, up 120% on the four weeks before. 0.1% of all new papers.

Stabilizing Native Low-Rank LLM Pretraining
Feb 12, 2026Paul Janson, Edouard Oyallon, Eugene BelilovskyLanguage Model Pretraining Low-Rank Matrix Decomposition
Concordia University, Montreal, Canada · Mila Quebec AI Institute, Montreal, Canada · Sorbonne University, CNRS, Paris, France
Transfer Denoising of Low-Rank Plus Sparse Matrices under Ambient and Rank Growth
Jan 29, 2026Elynn Chen, Jinhang Chai, Xuyuan Liu +1Transfer Learning Low-Rank Representation Learning
Department of Technology, Operations, & Statistics, New York University · Department of Computer Science, Dartmouth College
BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression
Sep 29, 2025David González-MartínezLow-Rank Matrix Decomposition Low-Rank Compression
Department of Computer Science and Artificial Intelligence, Universidad de Sevilla · Tübingen AI Center

Feb 12, 2026J/K move · Enter open · S save

Stabilizing Native Low-Rank LLM Pretraining

Paul Janson, Edouard Oyallon, Eugene Belilovsky

Concordia University, Montreal, Canada · Mila Quebec AI Institute, Montreal, Canada · Sorbonne University, CNRS, Paris, France

Language Model Pretraining Low-Rank Matrix Decomposition Language Model Scaling Laws Efficient Language Model Training Low-Rank Compression Low-Rank Approximation

Open paper AskPDF arXiv

Foundation models have achieved remarkable success, yet their growing parameter counts pose significant computational and memory challenges. Low-rank factorization offers a promising route to reduce training and inference costs, but the community lacks a stable recipe for training models from scratch using exclusively low-rank weights while matching the performance of the dense model. We demonstrate that Large Language Models (LLMs) can be trained from scratch using exclusively low-rank factorized weights for all non-embedding matrices without auxiliary "full-rank" guidance required by prior methods. While native low-rank training often suffers from instability and loss spikes, we identify uncontrolled growth in the spectral norm (largest singular value) of the weight matrix update as the dominant factor. To address this, we introduce Spectron: Spectral renormalization with orthogonalization, which dynamically bounds the resultant weight updates based on the current spectral norms of the factors. Our method enables stable, end-to-end factorized training with negligible overhead. Finally, we establish compute-optimal scaling laws for natively low-rank transformers, demonstrating predictable power-law behavior and improved inference efficiency relative to dense models.

Latest papers

Stabilizing Native Low-Rank LLM Pretraining

Transfer Denoising of Low-Rank Plus Sparse Matrices under Ambient and Rank Growth

BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression