Subspace Learning

Subspace learning focuses on identifying lower-dimensional representations of high-dimensional data that capture essential information, aiming to improve efficiency and robustness in various machine learning tasks. Current research emphasizes efficient algorithms like online subspace descent for large language model training and novel subspace optimization techniques for neural network training, including the exploration of low-rank adaptations and meta-learning frameworks for multi-task learning. These advancements are significant because they enhance the performance and scalability of machine learning models across diverse applications, from computer vision and natural language processing to anomaly detection and medical image analysis.

Papers

November 6, 2024

Adaptive Consensus Gradients Aggregation for Scaled Distributed Training
Yoni Choukroun, Shlomi Azoulay, Pavel Kisilev
Deep Subspace Subspace Learning Aggregated Gradient Scalable Training

November 1, 2024

Exploring the Precise Dynamics of Single-Layer GAN Models: Leveraging Multi-Feature Discriminators for High-Dimensional Subspace Learning
Andrew Bond, Zafer Dogan
GAN Model Dynamic Model Subspace Learning State of the Art GAN Single GAN Dimensional Random Vector

October 22, 2024

Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models
Yuheng Lu, Bingshuo Qian, Caixia Yuan, Huixing Jiang, Xiaojie Wang
Large Language Model Natural Language Processing Training Data Catastrophic Forgetting Adaptation Concern Subspace Learning Finetuning Method LoRA Based

October 18, 2024

A Statistical Machine Learning Approach for Adapting Reduced-Order Models using Projected Gaussian Process
Xiao Liu, Xinchao Liu
Full Model Gaussian Process Machine Learning Approach Reduced Order Subspace Learning Proper Orthogonal Decomposition Reduced Basis Polynomial Basis

October 15, 2024

Subspace Optimization for Large Language Models with Convergence Guarantees
Yutong He, Pengrui Li, Yipeng Hu, Chuyan Chen, Kun Yuan
Large Language Model Early Stage Convergence Convergence Guarantee Formal Guarantee Subspace Learning Sparse Optimization

September 25, 2024

Parameter-efficient Bayesian Neural Networks for Uncertainty-aware Depth Estimation
Richard D. Paul, Alessio Quercia, Vincent Fortuin, Katharina Nöh, Hanno Scharr
Vision Transformer Monocular Depth Estimation Parameter Efficient Fine Tuning Bayesian Neural Network Subspace Learning

August 23, 2024

Memory-Efficient LLM Training with Online Subspace Descent
Kaizhao Liang, Bo Liu, Lizhang Chen, Qiang Liu
Subspace Learning Low Rank Structure Online Gradient Descent Low Rank Training

August 13, 2024

Optimal Bound for PCA with Outliers using Higher-Degree Voronoi Diagrams
Sajjad Hashemian, Mohammad Saeed Arvenaghi, Ebrahim Ardeshir-Larijani
Principal Component Analysis Subspace Learning Voronoi Diagram

June 25, 2024

Why Line Search when you can Plane Search? SO-Friendly Neural Networks allow Per-Iteration Optimization of Learning and Momentum Rates for Every Layer
Betty Shea, Mark Schmidt
Neural Network Gradient Descent Subspace Learning Plane Feature Line Search Momentum Transformer

June 6, 2024

Subspace Clustering in Wavelet Packets Domain
Ivica Kopriva, Damir Sersic
Subspace Clustering Subspace Learning Wavelet Domain Transform Domain

May 28, 2024

HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning
Shengchao Hu, Ziqing Fan, Li Shen, Ya Zhang, Yanfeng Wang, Dacheng Tao
Reinforcement Learning Offline Reinforcement Learning Optimal Policy Decision Transformer Subspace Learning Diverse Policy

May 2, 2024

Few Shot Class Incremental Learning using Vision-Language models
Anurag Kumar, Chinmay Bharti, Saikat Dutta, Srikrishna Karanam, Biplab Banerjee
Vision Language Model Incremental Learning Subspace Learning Incremental Training Shot Class

March 24, 2024

Subspace Defense: Discarding Adversarial Perturbations by Learning a Subspace for Clean Signals
Rui Zheng, Yuhao Zhou, Zhiheng Xi, Tao Gui, Qi Zhang, Xuanjing Huang
Adversarial Attack Adversarial Example Adversarial Perturbation Clean Speech Subspace Learning Subspace Prototype Effective Adversarial

March 1, 2024

Large Convolutional Model Tuning via Filter Subspace
Wei Chen, Zichen Miao, Qiang Qiu
Fine Tuning Convolutional Filter Subspace Learning Spatial Convolution Multi View Subspace

February 3, 2024

Trustworthiness of $\mathbb{X}$ Users: A One-Class Classification Approach
Tanveer Khan, Fahad Sohrab, Antonis Michalas, Moncef Gabbouj
One Class Classification Subspace Learning User Trust

January 5, 2024

Krylov Cubic Regularized Newton: A Subspace Second-Order Method with Dimension-Free Convergence Rate
Ruichen Jiang, Parameswaran Raman, Shoham Sabach, Aryan Mokhtari, Mingyi Hong, Volkan Cevher
Convergence Rate Second Order Subspace Learning Newton Method Subspace Prototype Random Subspace

December 11, 2023

Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusion
Anke Tang, Li Shen, Yong Luo, Liang Ding, Han Hu, Bo Du, Dacheng Tao
Task Specific Model Model Merging Multi Task Model Subspace Learning Interference Cancellation Signal Subspace

November 28, 2023

SubZero: Subspace Zero-Shot MRI Reconstruction
Heng Yu, Yamin Arefeen, Berkin Bilgic
Zero Shot Self Supervised Learning Zero Shot Learning Magnetic Resonance Imaging Subspace Learning

August 20, 2023

An alternative to SVM Method for Data Classification
Lakhdar Remaki
Support Vector Machine Classification Problem Subspace Learning

July 25, 2023

Do algorithms and barriers for sparse principal component analysis extend to other structured settings?
Guanyi Wang, Mengqi Lou, Ashwin Pananjady
Practical Algorithm Knowledge Barrier Structured Document Subspace Learning Sparse Principal Component Analysis Graph Sparsity

Subspace Learning

Papers

Adaptive Consensus Gradients Aggregation for Scaled Distributed Training

Exploring the Precise Dynamics of Single-Layer GAN Models: Leveraging Multi-Feature Discriminators for High-Dimensional Subspace Learning

Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models

A Statistical Machine Learning Approach for Adapting Reduced-Order Models using Projected Gaussian Process

Subspace Optimization for Large Language Models with Convergence Guarantees

Parameter-efficient Bayesian Neural Networks for Uncertainty-aware Depth Estimation

Memory-Efficient LLM Training with Online Subspace Descent

Optimal Bound for PCA with Outliers using Higher-Degree Voronoi Diagrams

Why Line Search when you can Plane Search? SO-Friendly Neural Networks allow Per-Iteration Optimization of Learning and Momentum Rates for Every Layer

Subspace Clustering in Wavelet Packets Domain

HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning

Few Shot Class Incremental Learning using Vision-Language models

Subspace Defense: Discarding Adversarial Perturbations by Learning a Subspace for Clean Signals

Large Convolutional Model Tuning via Filter Subspace

Trustworthiness of $\mathbb{X}$ Users: A One-Class Classification Approach

Krylov Cubic Regularized Newton: A Subspace Second-Order Method with Dimension-Free Convergence Rate

Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusion

SubZero: Subspace Zero-Shot MRI Reconstruction

An alternative to SVM Method for Data Classification

Do algorithms and barriers for sparse principal component analysis extend to other structured settings?