Generalization Performance

Generalization performance in machine learning focuses on a model's ability to accurately predict outcomes on unseen data, a crucial aspect for real-world applications. Current research investigates this through various lenses, including mitigating overfitting in self-supervised and federated learning, improving the robustness of models to out-of-distribution data (e.g., using dropout or orthogonal regularization), and enhancing the efficiency of fine-tuning large pre-trained models (e.g., via low-rank adaptation). Understanding and improving generalization is vital for building reliable and adaptable AI systems across diverse domains, impacting fields from image recognition and natural language processing to control of biological neural networks.

Papers

November 29, 2022

Disentangling the Mechanisms Behind Implicit Regularization in SGD
Zachary Novack, Simran Kaur, Tanya Marwah, Saurabh Garg, Zachary C. Lipton
Stochastic Gradient Descent Generalization Performance Implicit Regularization Functional Mechanism

November 25, 2022

A Survey of Learning Curves with Bad Behavior: or How More Data Need Not Lead to Better Performance
Marco Loog, Tom Viering
Timely Survey Training Data Generalization Performance Performance Improvement Learning Curve Parametric Curve

November 23, 2022

November 22, 2022

Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions
Satwik Bhattamishra, Arkil Patel, Varun Kanade, Phil Blunsom
Transformer Megatron Decepticons Generalization Performance Recurrent Network Critique Ability Simplicity Bias Random Transformation Sparse Function

November 20, 2022

Frozen Overparameterization: A Double Descent Perspective on Transfer Learning of Deep Neural Networks
Yehuda Dar, Lorenzo Luzi, Richard G. Baraniuk
Deep Neural Network Transfer Learning Generalization Performance Double Descent Overparametrization Bound

November 16, 2022

Learning with Noisy Labels over Imbalanced Subpopulations
MingCai Chen, Yu Zhao, Bing He, Zongbo Han, Bingzhe Wu, Jianhua Yao
LeArning Abstract Pseudo Label Noisy Label Generalization Performance Distributionally Robust Optimization Imbalanced Dataset

November 12, 2022

A Generalized Doubly Robust Learning Framework for Debiasing Post-Click Conversion Rate Prediction
Quanyu Dai, Haoxuan Li, Peng Wu, Zhenhua Dong, Xiao-Hua Zhou, Rui Zhang, Rui zhang, Jie Sun
Generalization Performance Generalization Bound Doubly Robust Conversion Rate Prediction Robustness Bias Post Click

October 26, 2022

SimpleDG: Simple Domain Generalization Baseline without Bells and Whistles
Zhi Lv, Bo Lin, Siyuan Liang, Lihua Wang, Mochen Yu, Yao Tang, Jiajun Liang
Domain Generalization Generalization Performance Bell and Whistle Context Generalization

October 23, 2022

K-SAM: Sharpness-Aware Minimization at the Speed of SGD
Renkun Ni, Ping-yeh Chiang, Jonas Geiping, Micah Goldblum, Andrew Gordon Wilson, Tom Goldstein
Stochastic Gradient Descent Generalization Performance Natural Gradient Sharpness Aware Minimization Speed Effect Gradient Computation Vanilla Gradient Descent

October 22, 2022

Probing Transfer in Deep Reinforcement Learning without Task Engineering
Andrei A. Rusu, Sebastian Flennerhag, Dushyant Rao, Razvan Pascanu, Raia Hadsell
Deep Reinforcement Learning Generalization Performance Deep Reinforcement Linear Probing Task Specification Game Design Deep Reinforcement Learning Agent ATARI Game

October 19, 2022

Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clipping
Chenghao Yang, Xuezhe Ma
Fine Tuning Pre Trained Language Model Generalization Performance Core Stability Gradient Norm

October 18, 2022

Rethinking Value Function Learning for Generalization in Reinforcement Learning
Seungyong Moon, JunYeong Lee, Hyun Oh Song
Reinforcement Learning Strong Generalization Self Supervised Generalization Performance Value Function Policy Network Policy Gradient Algorithm

October 13, 2022

GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalization
Zhiyuan Zhang, Ruixuan Luo, Qi Su, Xu Sun
Adaptive Importance Generalization Performance Natural Gradient Sharpness Aware Minimization Improved Generalization

October 9, 2022

Using Gradient to Boost the Generalization Performance of Deep Learning Models for Fluid Dynamics
Eduardo Vital Brasil
Deep Learning Deep Learning Model Generalization Performance Natural Gradient Computational Fluid Dynamic Fluid Dynamic Performance Bottleneck Shape Optimization

October 5, 2022

Spectral Regularization Allows Data-frugal Learning over Combinatorial Spaces
Amirali Aghazadeh, Nived Rajaraman, Tony Tu, Kannan Ramchandran
Generalization Performance Improved Generalization Learned Function Spectral Regularization Optimal Generalization Frugal Learning High Dimensional Combinatorial

September 30, 2022

Identify ambiguous tasks combining crowdsourced labels by weighting Areas Under the Margin
Tanguy Lefort, Benjamin Charlier, Alexis Joly, Joseph Salmon
Supervised Learning Generalization Performance Margin Maximization Area Usage Task Ambiguity Crowdsourcing Study Crowdsourced Label Scoring Task

Generalization Performance

Papers

Disentangling the Mechanisms Behind Implicit Regularization in SGD

A Survey of Learning Curves with Bad Behavior: or How More Data Need Not Lead to Better Performance

Using Focal Loss to Fight Shallow Heuristics: An Empirical Analysis of Modulated Cross-Entropy in Natural Language Inference

Relating Regularization and Generalization through the Intrinsic Dimension of Activations

Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions

Frozen Overparameterization: A Double Descent Perspective on Transfer Learning of Deep Neural Networks

Learning with Noisy Labels over Imbalanced Subpopulations

A Generalized Doubly Robust Learning Framework for Debiasing Post-Click Conversion Rate Prediction

SimpleDG: Simple Domain Generalization Baseline without Bells and Whistles

K-SAM: Sharpness-Aware Minimization at the Speed of SGD

Probing Transfer in Deep Reinforcement Learning without Task Engineering

Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clipping

Rethinking Value Function Learning for Generalization in Reinforcement Learning

GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalization

GULP: a prediction-based metric between representations

Guaranteed Conservation of Momentum for Learning Particle-based Fluid Dynamics

Optimizing Evaluation Metrics for Multi-Task Learning via the Alternating Direction Method of Multipliers

Using Gradient to Boost the Generalization Performance of Deep Learning Models for Fluid Dynamics

Spectral Regularization Allows Data-frugal Learning over Combinatorial Spaces

Identify ambiguous tasks combining crowdsourced labels by weighting Areas Under the Margin