Distribution Generalization

Distribution generalization in machine learning focuses on developing models that maintain high performance when encountering data significantly different from their training data. Current research emphasizes techniques like invariant learning, multicalibration, and ensemble methods, often applied within transformer, graph neural network, and other architectures, to improve robustness against various distribution shifts (covariate, label, concept shifts). Successfully addressing this challenge is crucial for deploying reliable machine learning systems in real-world applications, where data distributions are inherently complex and dynamic, impacting fields such as autonomous driving, medical diagnosis, and scientific discovery.

Papers

May 2, 2024

Invariant Risk Minimization Is A Total Variation Model
Zhao-Rong Lai, Weiwen Wang
Distribution Generalization Invariant Feature Total Variation Invariant Risk Minimization

April 27, 2024

Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization
Dang Nguyen, Paymon Haddad, Eric Gan, Baharan Mirzasoleiman
Raw Data Gradient Descent Generalization Performance Distribution Generalization Sharpness Aware Minimization Tiny ImageNet Training Data Distribution

March 29, 2024

An Information-Theoretic Framework for Out-of-Distribution Generalization with Applications to Stochastic Gradient Langevin Dynamics
Wenliang Liu, Guanding Yu, Lele Wang, Renjie Liao
Distribution Generalization Distribution Data Generalization Bound Information Theoretic Framework Information Theoretic Generalization Bound F$ Divergence

March 26, 2024

On the Benefits of Over-parameterization for Out-of-Distribution Generalization
Yifan Hao, Yong Lin, Difan Zou, Tong Zhang
Neural Network Distribution Generalization Distribution Data Complementary Benefit Parameterized Model Benign Over Parameterization

March 18, 2024

March 11, 2024

Towards Robust Out-of-Distribution Generalization Bounds via Sharpness
Yingtian Zou, Kenji Kawaguchi, Yingnan Liu, Jiashuo Liu, Mong-Li Lee, Wynne Hsu
Strong Generalization Distribution Generalization Node Feature Sharpness

March 4, 2024

February 14, 2024

Transformers Can Achieve Length Generalization But Not Robustly
Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, Denny Zhou
Language Model Transformer Megatron Decepticons Distribution Generalization Length Generalization Sequence Length Large Scale Transformer

February 13, 2024

Investigating Out-of-Distribution Generalization of GNNs: An Architecture Perspective
Kai Guo, Hongzhi Wen, Wei Jin, Yaming Guo, Jiliang Tang, Yi Chang
Graph Neural Network Distribution Generalization GNN Architecture Architecture Perspective GNN Backbone Graph Out of Distribution Generalization

February 9, 2024

On the Out-Of-Distribution Generalization of Multimodal Large Language Models
Xingxuan Zhang, Jiansheng Li, Wenjing Chu, Junjia Hai, Renzhe Xu, Yuqing Yang, Shikai Guan, Jiazheng Xu, Peng Cui
Zero Shot Strong Generalization Multimodal Large Language Model Distribution Generalization Generalization Problem

February 7, 2024

On Provable Length and Compositional Generalization
Kartik Ahuja, Amin Mansouri
Distribution Generalization Compositional Generalization Sequence to Sequence Model Error Bound Length Generalization

January 29, 2024

Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization
Yuhang Zang, Hanlin Goh, Josh Susskind, Chen Huang
Zero Shot Vision Language Model Vision Language Distribution Generalization Common Pitfall Visual Concept

January 8, 2024

InvariantOODG: Learning Invariant Features of Point Clouds for Out-of-Distribution Generalization
Zhimin Zhang, Xiang Gao, Wei Hu
Point Cloud Domain Generalization 3D Point Cloud Distribution Generalization Invariant Feature Input Point Cloud Point Cloud Representation Learning

January 7, 2024

Few-Shot Causal Representation Learning for Out-of-Distribution Generalization on Heterogeneous Graphs
Pengfei Ding, Yan Wang, Guanfeng Liu, Nan Wang, Xiaofang Zhou
Shot Learning Distribution Generalization Heterogeneous Graph Causal Representation Learning Heterogeneous Graph Representation

December 29, 2023

Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift
Benjamin Eyre, Elliot Creager, David Madras, Vardan Papyan, Richard Zemel
Pre Trained Distribution Generalization Novel Regression Least Square Spectral Feature Covariate Shift Distribution Performance

December 26, 2023

Unraveling the Key Components of OOD Generalization via Diversification
Harold Benoit, Liangze Jiang, Andrei Atanov, Oğuzhan Fatih Kar, Mattia Rigotti, Amir Zamir
Distribution Generalization Expert Diversification Depth Hypothesis Key Component

December 25, 2023

Mixture Data for Training Cannot Ensure Out-of-distribution Generalization
Songming Zhang, Yuxiao Luo, Qizhou Wang, Haoang Chi, Xiaofeng Chen, Bo Han, Jinyan Li
Strong Generalization Training Data Distribution Generalization Generalization Error Generalization Problem Data Mixture

December 18, 2023

Graph Invariant Learning with Subgraph Co-mixup for Out-Of-Distribution Generalization
Tianrui Jia, Haoyang Li, Cheng Yang, Tao Tao, Chuan Shi
Graph Representation Learning Distribution Generalization Mixup Method Invariant Learning Mixed Type Graph Mixup Invariant Learning Method Invariant Graph Learning

Distribution Generalization

Papers

Invariant Risk Minimization Is A Total Variation Model

Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization

An Information-Theoretic Framework for Out-of-Distribution Generalization with Applications to Stochastic Gradient Langevin Dynamics

On the Benefits of Over-parameterization for Out-of-Distribution Generalization

Improving Out-of-Distribution Generalization of Learned Dynamics by Learning Pseudometrics and Constraint Manifolds

Transfer Learning Beyond Bounded Density Ratios

Towards Robust Out-of-Distribution Generalization Bounds via Sharpness

A Survey on Evaluation of Out-of-Distribution Generalization

Improving out-of-distribution generalization in graphs via hierarchical semantic environments

Transformers Can Achieve Length Generalization But Not Robustly

Investigating Out-of-Distribution Generalization of GNNs: An Architecture Perspective

On the Out-Of-Distribution Generalization of Multimodal Large Language Models

On Provable Length and Compositional Generalization

Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization

InvariantOODG: Learning Invariant Features of Point Clouds for Out-of-Distribution Generalization

Few-Shot Causal Representation Learning for Out-of-Distribution Generalization on Heterogeneous Graphs

Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift

Unraveling the Key Components of OOD Generalization via Diversification

Mixture Data for Training Cannot Ensure Out-of-distribution Generalization

Graph Invariant Learning with Subgraph Co-mixup for Out-Of-Distribution Generalization