Low Rank Constraint

Low-rank constraints are increasingly used in machine learning to reduce computational complexity and memory usage while maintaining model performance. Current research focuses on developing efficient algorithms, such as those based on proximal gradient descent and Riemannian optimization, to impose these constraints effectively within various model architectures, including neural networks and tensor-based methods. This approach is particularly relevant for large-scale applications like large language model training and federated learning, where resource limitations are significant. The resulting improvements in efficiency and scalability have broad implications across numerous fields, from computer vision and natural language processing to healthcare and robotics.

Papers

October 2, 2024

Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?
Xi Chen, Kaituo Feng, Changsheng Li, Xunhao Lai, Xiangyu Yue, Ye Yuan, Guoren Wang
Large Language Model Low Rank Grand Challenge Low Rank Constraint Low Rank Training

December 15, 2023

ParsNets: A Parsimonious Orthogonal and Low-Rank Linear Networks for Zero-Shot Learning
Jingcai Guo, Qihua Zhou, Ruibing Li, Xiaocheng Lu, Ziming Liu, Junyang Chen, Xin Xie, Jie Zhang
Zero Shot Learning Post Hoc Orthogonalization Linear Network Low Rank Constraint Low Rank Property

November 15, 2023

Learning Hyperplanes for Multi-Agent Collision Avoidance in Space
Fernando Palafox, Yue Yu, David Fridovich-Keil
Collision Avoidance Multi Robot Deep Space Expert Trajectory Truthfulness Hyperplane Low Rank Constraint

October 13, 2023

Identifiability of Product of Experts Models
Spencer L. Gordon, Manav Kant, Eric Ma, Leonard J. Schulman, Andrei Staicu
Neural Network Architecture Novel Identifiability Test Expert Model Product Attribute Low Rank Constraint Binary Latent Variable

July 4, 2023

Strictly Low Rank Constraint Optimization -- An Asymptotically $\mathcal{O}(\frac{1}{t^2})$ Method
Mengyuan Zhang, Kai Liu
Regularization Model Sparsity Increase Proximal Gradient Descent Convex Problem Low Rank Constraint

June 4, 2023

Riemannian Low-Rank Model Compression for Federated Learning with Over-the-Air Aggregation
Ye Xue, Vincent Lau
Model Compression Low Rank Low Dimensional Manifold Riemannian Geometry Approach Low Rank Compression Low Rank Constraint Air Aggregation

May 29, 2023

Statistically Optimal K-means Clustering via Nonnegative Low-rank Semidefinite Programming
Yubo Zhuang, Xiaohui Chen, Yun Yang, Richard Y. Zhang
K Mean K$ Mean Semidefinite Programming Low Rank Constraint Burer Monteiro

April 20, 2023

Learning Sparse and Low-Rank Priors for Image Recovery via Iterative Reweighted Least Squares Minimization
Stamatios Lefkimmiatis, Iaroslav Koshelev
Many Sparse Least Square Reconstruction Network SAM Prior Sparse Recovery Image Recovery Low Rank Constraint

October 21, 2022

Learning Graphical Factor Models with Riemannian Optimization
Alexandre Hippert-Ferrer, Florent Bouchard, Ammar Mian, Titouan Vayer, Arnaud Breloy
Graph Learning Low Rank Graphical Model Riemannian Optimization Low Rank Constraint

May 21, 2022

Semi-Supervised Subspace Clustering via Tensor Low-Rank Representation
Yuheng Jia, Guanxing Lu, Hui Liu, Junhui Hou
Individual Representation Tensor Type Calculus Semi Supervised Clustering Affinity Matrix Low Rank Constraint

March 29, 2022

Coarse to Fine: Image Restoration Boosted by Multi-Scale Low-Rank Tensor Completion
Rui Lin, Cong Chen, Ngai Wong
Image Restoration Coarse to Fine Tensor Type Calculus Tensor Completion Low Rank Constraint Low Rank Tensor Completion

February 3, 2022

SparGE: Sparse Coding-based Patient Similarity Learning via Low-rank Constraints and Graph Embedding
Xian Wei, See Kiong Ng, Tongtong Zhang, Yingjie Liu
Graph Drawing Many Sparse Similarity Measure Healthcare Datasets Low Rank Constraint Sparse Coding Patient Similarity

December 10, 2021

Leveraging Joint-Diagonalization in Transform-Learning NMF
Sixin Zhang, Emmanuel Soubies, Cédric Févotte
Matrix Factorization Low Rank Constraint Matrix Diagonalization Learning Transformation

November 4, 2021

Stable and Compact Face Recognition via Unlabeled Data Driven Sparse Representation-Based Classification
Xiaohui Yang, Zheng Wang, Huan Wu, Licheng Jiao, Yiming Xu, Haolin Chen
Classification Code Representation Learning Many Sparse Unlabeled Data ADMM Algorithm Low Rank Constraint Lightweight Face

Low Rank Constraint

Papers

Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?

ParsNets: A Parsimonious Orthogonal and Low-Rank Linear Networks for Zero-Shot Learning

Learning Hyperplanes for Multi-Agent Collision Avoidance in Space

Identifiability of Product of Experts Models

Strictly Low Rank Constraint Optimization -- An Asymptotically $\mathcal{O}(\frac{1}{t^2})$ Method

Riemannian Low-Rank Model Compression for Federated Learning with Over-the-Air Aggregation

Statistically Optimal K-means Clustering via Nonnegative Low-rank Semidefinite Programming

Learning Sparse and Low-Rank Priors for Image Recovery via Iterative Reweighted Least Squares Minimization

Learning Graphical Factor Models with Riemannian Optimization

Semi-Supervised Subspace Clustering via Tensor Low-Rank Representation

Coarse to Fine: Image Restoration Boosted by Multi-Scale Low-Rank Tensor Completion

SparGE: Sparse Coding-based Patient Similarity Learning via Low-rank Constraints and Graph Embedding

Leveraging Joint-Diagonalization in Transform-Learning NMF

Stable and Compact Face Recognition via Unlabeled Data Driven Sparse Representation-Based Classification