Covariance Regularization

Covariance regularization is a technique used to improve the performance and generalizability of machine learning models, primarily by controlling the relationships between learned features. Current research focuses on applying this to various model architectures, including deep neural networks and Gaussian graphical models, often employing techniques like variance-covariance regularization (VCReg) or sparse covariance estimation to enhance transfer learning and address issues like neural collapse and gradient starvation. These methods aim to learn more robust and informative representations, leading to improved performance in tasks such as image classification, audio embedding, and few-shot learning. The impact of this work is seen in advancements across diverse applications, particularly where data is limited or domain shifts are prevalent.

Papers

October 25, 2024

Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning
Shentong Mo, Shengbang Tong
Self Supervised Learning Representation Learning Contrastive Example Unsupervised Visual Representation Learning Joint Embedding Predictive Architecture Predictive Architecture Covariance Regularization

February 28, 2024

Deep Neural Network Models Trained With A Fixed Random Classifier Transfer Better Across Domains
Hafiz Tiomoko Ali, Umberto Michieli, Ji Joong Moon, Daehyun Kim, Mete Ozay
Deep Neural Network DNN Model Neural Collapse Domain Name Covariance Regularization Classifier Transfer

September 23, 2023

Learning Large-Scale MTP$_2$ Gaussian Graphical Models via Bridge-Block Decomposition
Xiwen Wang, Jiaxi Ying, Daniel P. Palomar
Sparse Graph Multiscale Problem Gaussian Graphical Model Covariance Regularization

August 3, 2023

Online covariance estimation for stochastic gradient descent under Markovian sampling
Abhishek Roy, Krishnakumar Balasubramanian
Gradient Descent Stochastic Gradient Descent Stochastic Way Covariance Matrix Markovian Sampling Covariance Regularization

June 23, 2023

Variance-Covariance Regularization Improves Representation Learning
Jiachen Zhu, Katrina Evtimova, Yubei Chen, Ravid Shwartz-Ziv, Yann LeCun
Transfer Learning Self Supervised Representation Learning Covariance Regularization Covariate Representation

June 5, 2023

Exploring the Role of the Bottleneck in Slot-Based Models Through Covariance Regularization
Andrew Stange, Robert Lo, Abishek Sridhar, Kousik Rajesh
Integral Role Major Challenge Bottleneck Object Mask Slot Attention Feature Reconstruction Covariance Regularization Slot Based

March 7, 2023

Approach to Learning Generalized Audio Representation Through Batch Embedding Covariance Regularization and Constant-Q Transforms
Ankit Shah, Shuyi Chen, Kejun Zhou, Yue Chen, Bhiksha Raj
Constructive Approach Zero Shot Learning Audio Representation Batch Greenkhorn Covariance Regularization Audio Task

March 1, 2023

An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization
Ravid Shwartz-Ziv, Randall Balestriero, Kenji Kawaguchi, Tim G. J. Rudner, Yann LeCun
Self Supervised Learning Mutual Information Information Theoretic Covariance Regularization

February 20, 2023

Sparse PCA Beyond Covariance Thresholding
Gleb Novikov
Gaussian Distribution Sparse Principal Component Analysis Sparse Vector Covariance Regularization

December 6, 2022

Covariance Regularization for Probabilistic Linear Discriminant Analysis
Zhiyuan Peng, Mingjie Shao, Xuanji He, Xu Li, Tan Lee, Ke Ding, Guanglu Wan
Domain Adaptation Speaker Verification Speaker Embeddings Probabilistic Linear Discriminant Analysis Covariance Regularization

September 29, 2022

Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations
Grégoire Mialon, Randall Balestriero, Yann LeCun
Self Supervised Learning Learned Representation Independent Component Analysis MLP Layer Covariance Regularization

August 18, 2022

Meta Sparse Principal Component Analysis
Imon Banerjee, Jean Honorio
Sample Complexity Sparse Matrix Sparse Principal Component Analysis Covariance Regularization Dimensional Random Vector

April 11, 2022

GDC- Generalized Distribution Calibration for Few-Shot Learning
Shakti Kumar, Hussain Zaidi
LeArning Abstract Shot Class Covariance Regularization

Covariance Regularization

Papers

Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning

Deep Neural Network Models Trained With A Fixed Random Classifier Transfer Better Across Domains

Learning Large-Scale MTP$_2$ Gaussian Graphical Models via Bridge-Block Decomposition

Online covariance estimation for stochastic gradient descent under Markovian sampling

Variance-Covariance Regularization Improves Representation Learning

Exploring the Role of the Bottleneck in Slot-Based Models Through Covariance Regularization

Approach to Learning Generalized Audio Representation Through Batch Embedding Covariance Regularization and Constant-Q Transforms

An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization

Sparse PCA Beyond Covariance Thresholding

Covariance Regularization for Probabilistic Linear Discriminant Analysis

Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations

Meta Sparse Principal Component Analysis

GDC- Generalized Distribution Calibration for Few-Shot Learning