Low Rank Approximation

Low-rank approximation aims to represent high-dimensional data or matrices using a significantly smaller number of parameters, thereby reducing computational cost and storage needs while preserving essential information. Current research focuses on improving the accuracy and efficiency of low-rank techniques, particularly within the context of large language models (LLMs) and vision transformers, employing methods like singular value decomposition, Hadamard low-rank quantization, and various sketching algorithms. These advancements are crucial for deploying large models on resource-constrained devices and accelerating training processes, impacting fields ranging from machine learning and signal processing to natural language processing and healthcare.

Papers

February 20, 2024

A Bound on the Maximal Marginal Degrees of Freedom
Paul Dommel
Low Rank Approximation Kernel Ridge Regression Dimensional Approximation Freedom Matter Maximal Marginal Relevance

February 6, 2024

Operator SVD with Neural Networks via Nested Low-Rank Approximation
J. Jon Ryu, Xiangxiang Xu, H. S. Melihcan Erol, Yuheng Bu, Lizhong Zheng, Gregory W. Wornell
Neural Network High Dimensional Average Approximation Low Rank Approximation Singular Value Decomposition Nested Entity Eigenfunction Decomposition Truncated Singular Value Decomposition

February 5, 2024

A Random Matrix Approach to Low-Multilinear-Rank Tensor Approximation
Hugo Lebeau, Florent Chatelain, Romain Couillet
Low Rank Low Rank Approximation Random Matrix High Dimensional Regime Tensor Approximation Truncated Singular Value Decomposition Spiked Tensor

February 4, 2024

LQER: Low-Rank Quantization Error Reconstruction for LLMs
Cheng Zhang, Jianyi Cheng, George A. Constantinides, Yiren Zhao
Large Language Model Low Rank Approximation Quantization Error W4A8 Quantization Low Rank Quantization

December 8, 2023

StructComp: Substituting Propagation with Structural Compression in Training Graph Contrastive Learning
Shengzhong Zhang, Wenjie Yang, Xinyuan Cao, Hongwei Zhang, Zengfeng Huang
Contrastive Learning Contrastive Loss Graph Contrastive Learning Low Rank Approximation Fast Propagation Scalable Training Structured Compression

November 26, 2023

Hessian Aware Low-Rank Perturbation for Order-Robust Continual Learning
Jiaqi Li, Yuanhao Lai, Rui Wang, Changjian Shui, Sabyasachi Sahoo, Charles X. Ling, Shichun Yang, Boyu Wang, Christian Gagné, Fan Zhou
Continual LEArning Hessian Matrix Low Rank Approximation Small Perturbation Robust Continual Learning Robust Task

November 17, 2023

Optimal Embedding Dimension for Sparse Subspace Embeddings
Shabarish Chenakkod, Michał Dereziński, Xiaoyu Dong, Mark Rudelson
Low Rank Approximation Box Dimension Sparse Embeddings

November 6, 2023

Spatial Process Approximations: Assessing Their Necessity
Hao Zhang
Community Need Maximum Likelihood Low Rank Approximation Kernel Matrix

November 3, 2023

Hardness of Low Rank Approximation of Entrywise Transformed Matrix Products
Tamas Sarlos, Xingyou Song, David Woodruff, Qiuyi, Zhang
Low Rank Low Rank Approximation Hardness Result Approximation Algorithm Fast Algorithm Sparse Vector Matrix Vector Product

October 20, 2023

Accelerated sparse Kernel Spectral Clustering for large scale data clustering problems
Mihaly Novak, Rocco Langone, Carlos Alzate, Johan Suykens
Low Rank Approximation Kernel Matrix Kernel Principal Component Analysis Kernel Spectral Clustering

October 19, 2023

Compression of Recurrent Neural Networks using Matrix Factorization
Lucas Maison, Hélion du Mas des Bourboux, Thomas Courtat
Recurrent Neural Network Matrix Factorization Linear Compression Low Rank Approximation High Compression

September 29, 2023

Tight Bounds for Volumetric Spanners and Applications
Aditya Bhaskara, Sepideh Mahabadi, Ali Vakilian
Financial Application Low Rank Approximation Tight Bound Determinant Maximization

August 22, 2023

Patient Clustering via Integrated Profiling of Clinical and Digital Data
Dongjin Choi, Andy Xiang, Ozgur Ozturk, Deep Shrestha, Barry Drake, Hamid Haidarian, Faizan Javed, Haesun Park
Raw Data Low Rank Approximation Patient Data Clinical Data Clinical Language Patient Demographic Patient Level Dataset Specific Profiling

July 16, 2023

A Novel Truncated Norm Regularization Method for Multi-channel Color Image Denoising
Yiwen Shan, Dong Hu, Zhi Wang
Gradient Norm Low Rank Approximation Real World Image Denoising Color Image Denoising Nuclear Norm Frobenius Norm Minimization

July 11, 2023

Making the Nystr\"om method highly accurate for low-rank approximations
Jianlin Xia
Low Rank Approximation Kernel Function Singular Value Nystr\"om Approximation

June 20, 2023

LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation
Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, Tuo Zhao
Language Understanding Transformer Model Model Compression Low Rank Approximation Sparse Approximation

June 14, 2023

Graph-Based Matrix Completion Applied to Weather Data
Benoît Loucheur, P. -A. Absil, Michel Journée
Matrix Completion Low Rank Approximation Weather Data Low Rank Matrix Completion Weighted Graph

June 11, 2023

Learning the Positions in CountSketch
Yi Li, Honghao Lin, Simin Liu, Ali Vakilian, David P. Woodruff
Low Rank Approximation Product Specific Position Information Sketch Based Matrix Sketching

June 7, 2023

Efficient Alternating Minimization with Applications to Weighted Low Rank Approximation
Zhao Song, Mingquan Ye, Junze Yin, Lichen Zhang
Financial Application Low Rank Approximation Weight Matrix Robust Framework Numerical Linear Algebra Alternating Minimization

May 3, 2023

Multi-dimensional Signal Recovery using Low-rank Deconvolution
David Reixach
Low Rank Approximation Low Rank Tensor Feature Map Convolutional Sparse Spatiotemporal Signal