Isotropic Gaussian

Isotropic Gaussian distributions are a cornerstone in various machine learning and statistical analyses, serving as a foundational assumption in many models and algorithms. Current research focuses on leveraging isotropic Gaussian noise for regularization in neural network training, particularly to improve generalization performance and mitigate overfitting, often employing Hessian-based methods. This focus extends to analyzing the impact of isotropic Gaussian data on the sample complexity and convergence rates of stochastic gradient descent (SGD) in learning specific model classes, such as single index models. These investigations contribute to a deeper understanding of optimization landscapes and the theoretical underpinnings of various machine learning techniques.

Papers

October 13, 2024

Gradient Span Algorithms Make Predictable Progress in High Dimension
Felix Benning, Leif Döring
Machine Learning Model RSD Difference of Gaussian High Dimension Gradient Method Continuous Improvement Isotropic Gaussian Random Function

May 25, 2024

DefSent+: Improving sentence embeddings of language models by projecting definition sentences into a quasi-isotropic or isotropic vector space of unlimited dictionary entries
Xiaodong Liu
Language Model Word Embeddings Sentence Embeddings Pre Trained Embeddings Social Entity Embeddings Isotropic Gaussian Sparse Dictionary

June 14, 2023

Noise Stability Optimization for Finding Flat Minima: A Hessian-based Regularization Approach
Hongyang R. Zhang, Dongyue Li, Haotian Ju
Regularization Model Hessian Matrix Sharpness Aware Minimization Flat Minimum Optimal Rate Sharpness Aware Training Isotropic Gaussian

May 18, 2023

Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Models
Alex Damian, Eshaan Nichani, Rong Ge, Jason D. Lee
Stochastic Gradient Descent Sample Complexity Abnormal Signal Landscape Image O$ Sample Single Index Model Isotropic Gaussian

February 21, 2023

SGD learning on neural networks: leap complexity and saddle-to-saddle dynamics
Emmanuel Abbe, Enric Boix-Adsera, Theodor Misiakiewicz
Neural Network Stochastic Gradient Descent Fully Connected Neural Network Gradient Complexity Saddle to Saddle Dynamic Isotropic Gaussian Complexity Scaling

June 27, 2022

Wasserstein Distributionally Robust Estimation in High Dimensions: Performance Analysis and Optimal Hyperparameter Tuning
Liviu Aolaritei, Soroosh Shafiee, Florian Dörfler
Robust Optimization High Dimension Performance Analysis Robust Estimation Convex Concave Wasserstein Distributionally Robust Isotropic Gaussian

May 30, 2022

Leave-one-out Singular Subspace Perturbation Analysis for Spectral Clustering
Anderson Y. Zhang, Harrison H. Zhou
Spectral Clustering Gaussian Mixture Isotropic Gaussian

Isotropic Gaussian

Papers

Gradient Span Algorithms Make Predictable Progress in High Dimension

DefSent+: Improving sentence embeddings of language models by projecting definition sentences into a quasi-isotropic or isotropic vector space of unlimited dictionary entries

Noise Stability Optimization for Finding Flat Minima: A Hessian-based Regularization Approach

Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Models

SGD learning on neural networks: leap complexity and saddle-to-saddle dynamics

Wasserstein Distributionally Robust Estimation in High Dimensions: Performance Analysis and Optimal Hyperparameter Tuning

Leave-one-out Singular Subspace Perturbation Analysis for Spectral Clustering