Low-Rank Approximation

Low-rank approximation studies how to represent data or computations using a small number of latent components, making tasks such as compression, matrix completion, recommendation, and scientific modeling more efficient. Common approaches include truncated or randomized singular value decomposition, tensor factorization, nuclear-norm regularization, and adaptive rank selection. In neural networks and language models, researchers use activation- or loss-aware decompositions to compress weights and attention caches, and low-rank parameterizations to reduce training costs. A central tension is that reconstruction error may not reflect downstream task quality: choosing ranks that preserve useful behavior while reducing memory and computation remains difficult, especially when structure varies across layers, inputs, or training steps.