Distribution Generalization

Distribution generalization in machine learning focuses on developing models that maintain high performance when encountering data significantly different from their training data. Current research emphasizes techniques like invariant learning, multicalibration, and ensemble methods, often applied within transformer, graph neural network, and other architectures, to improve robustness against various distribution shifts (covariate, label, concept shifts). Successfully addressing this challenge is crucial for deploying reliable machine learning systems in real-world applications, where data distributions are inherently complex and dynamic, impacting fields such as autonomous driving, medical diagnosis, and scientific discovery.

Papers

December 19, 2022

Exploring Optimal Substructure for Out-of-distribution Generalization via Feature-targeted Model Pruning
Yingchun Wang, Jingcai Guo, Song Guo, Weizhan Zhang, Jie Zhang
Distribution Generalization Model Pruning Structural Generalization Structural Optimization

December 18, 2022

On the Connection between Invariant Learning and Adversarial Training for Out-of-Distribution Generalization
Shiji Xin, Yifei Wang, Jingtong Su, Yisen Wang
Adversarial Training Distribution Generalization Domain Invariant Feature New Connection Domain Adversarial Invariant Learning Invariant Risk Minimization Domain Variation

December 6, 2022

Causal Inference via Style Transfer for Out-of-distribution Generalisation
Toan Nguyen, Kien Do, Duc Thanh Nguyen, Bao Duong, Thin Nguyen
Strong Generalization Causal Inference Style Transfer Distribution Generalization Unobserved Confounders

November 29, 2022

Penalizing Confident Predictions on Largely Perturbed Inputs Does Not Improve Out-of-Distribution Generalization in Question Answering
Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
Strong Generalization Question Answering Distribution Generalization Shuffle Model Low Confidence Input Perturbation Overconfident Prediction

November 28, 2022

Malign Overfitting: Interpolation Can Provably Preclude Invariance
Yoav Wald, Gal Yona, Uri Shalit, Yair Carmon
Distribution Generalization Model Overfitting Interpolation Regime Alpha Invariance Benign Overfitting Non Parametric Learning Classifier Invariance Regularization

November 21, 2022

Exploiting Personalized Invariance for Better Out-of-distribution Generalization in Federated Learning
Xueyang Tang, Song Guo, Jie Zhang
Distribution Generalization Better Out of Distribution Generalization

November 18, 2022

Decorr: Environment Partitioning for Invariant Learning and OOD Generalization
Yufan Liao, Qi Wu, Zhaodi Wu, Xing Yan
Distribution Generalization Invariant Learning Consistent Prediction Data Subset Selection Invariant Learning Method

November 15, 2022

November 2, 2022

Hierarchies over Vector Space: Orienting Word and Graph Embeddings
Xingzhi Guo, Steven Skiena
Distribution Generalization Part Whole Hierarchy Graph Embeddings Vector Space Unaddressed Directionality

October 23, 2022

Functional Indirection Neural Estimator for Better Out-of-distribution Generalization
Kha Pham, Hung Le, Man Ngo, Truyen Tran
Strong Generalization Distribution Generalization Better Out of Distribution Generalization Functional Learning

October 17, 2022

ZooD: Exploiting Model Zoo for Out-of-Distribution Generalization
Qishi Dong, Awais Muhammad, Fengwei Zhou, Chuanlong Xie, Tianyang Hu, Yongxin Yang, Sung-Ho Bae, Zhenguo Li
Pre Trained Model Distribution Generalization Large Scale Pre Training Model Zoo Leave One Out Cross Validation Input Ranking Model

October 13, 2022

Improving Out-of-Distribution Generalization by Adversarial Training with Structured Priors
Qixun Wang, Yifei Wang, Hong Zhu, Yisen Wang
Adversarial Training Adversarial Perturbation Distribution Generalization Deep Model Structural Prior Training Domain

October 11, 2022

C-Mixup: Improving Generalization in Regression
Huaxiu Yao, Yiping Wang, Linjun Zhang, James Zou, Chelsea Finn
Strong Generalization Distribution Generalization Novel Regression Mixup Method C Mixup

October 10, 2022

SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in Fine-tuned Source Code Models
Hossein Hajipour, Ning Yu, Cristian-Alexandru Staicu, Mario Fritz
Fine Tuning Distribution Generalization Fine Tuned Model Pre Trained Code Model Source Code Model

October 9, 2022

Are All Vision Models Created Equal? A Study of the Open-Loop to Closed-Loop Causality Gap
Mathias Lechner, Ramin Hasani, Alexander Amini, Tsun-Hsuan Wang, Thomas A. Henzinger, Daniela Rus
Study Feature Distribution Generalization Vision Model

October 7, 2022

September 29, 2022

EiHi Net: Out-of-Distribution Generalization Paradigm
Qinglai Wei, Beiming Yuan, Diancheng Chen
Deep Learning Distribution Generalization Deep Model Generalization Problem

September 4, 2022

Generalization in Neural Networks: A Broad Survey
Chris Rohlfs
Neural Network Strong Generalization Distribution Generalization General Framework Task Generalization

Distribution Generalization

Papers

Exploring Optimal Substructure for Out-of-distribution Generalization via Feature-targeted Model Pruning

On the Connection between Invariant Learning and Adversarial Training for Out-of-Distribution Generalization

Causal Inference via Style Transfer for Out-of-distribution Generalisation

Penalizing Confident Predictions on Largely Perturbed Inputs Does Not Improve Out-of-Distribution Generalization in Question Answering

Malign Overfitting: Interpolation Can Provably Preclude Invariance

Exploiting Personalized Invariance for Better Out-of-distribution Generalization in Federated Learning

Decorr: Environment Partitioning for Invariant Learning and OOD Generalization

Empirical Study on Optimizer Selection for Out-of-Distribution Generalization

GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective

Hierarchies over Vector Space: Orienting Word and Graph Embeddings

Functional Indirection Neural Estimator for Better Out-of-distribution Generalization

ZooD: Exploiting Model Zoo for Out-of-Distribution Generalization

Improving Out-of-Distribution Generalization by Adversarial Training with Structured Priors

C-Mixup: Improving Generalization in Regression

SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in Fine-tuned Source Code Models

Are All Vision Models Created Equal? A Study of the Open-Loop to Closed-Loop Causality Gap

Using Interventions to Improve Out-of-Distribution Generalization of Text-Matching Recommendation Systems

Achieving and Understanding Out-of-Distribution Generalization in Systematic Reasoning in Small-Scale Transformers

EiHi Net: Out-of-Distribution Generalization Paradigm

Generalization in Neural Networks: A Broad Survey