Training Data Distribution

Training data distribution research focuses on how the characteristics of data used to train machine learning models affect model performance and robustness, particularly concerning out-of-distribution (OOD) generalization and federated learning scenarios. Current research explores techniques like logit scaling and weight perturbations to improve OOD detection, and methods such as data augmentation and knowledge distillation to enhance in-distribution generalization and address data heterogeneity in decentralized settings. These efforts are crucial for building reliable and adaptable AI systems, improving their performance in real-world applications where data distributions are often non-uniform and may shift over time.

Papers

September 2, 2024

Logit Scaling for Out-of-Distribution Detection
Andrija Djurisic, Rosanne Liu, Mladen Nikolic
Distribution Detection Distribution Datasets Training Data Distribution Logit Calibration

July 2, 2024

Equidistribution-based training of Free Knot Splines and ReLU Neural Networks
Simone Appella, Simon Arridge, Chris Budd, Teo Deveney, Lisa Maria Kreusser
Function Approximation Shallow Neural Network Rectified Linear Unit Training Data Distribution

May 27, 2024

WeiPer: OOD Detection using Weight Perturbations of Class Projections
Maximilian Granz, Manuel Heurich, Tim Landgraf
Distribution Detection Distribution Data Training Data Distribution Weight Perturbation

April 27, 2024

Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization
Dang Nguyen, Paymon Haddad, Eric Gan, Baharan Mirzasoleiman
Raw Data Gradient Descent Generalization Performance Distribution Generalization Sharpness Aware Minimization Tiny ImageNet Training Data Distribution

September 30, 2023

FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregation
Xiang Liu, Liangxi Liu, Feiyang Ye, Yunheng Shen, Xia Li, Linshan Jiang, Jialin Li
Training Data Distribution One Shot Federated Learning One Shot Aggregation

September 21, 2023

Exploring the Impact of Training Data Distribution and Subword Tokenization on Gender Bias in Machine Translation
Bar Iluz, Tomasz Limisiewicz, Gabriel Stanovsky, David Mareček
Machine Translation Global Impact Gender Bias Subword Tokenization Gender Prediction Training Data Distribution Gender Balance

April 9, 2023

Homogenizing Non-IID datasets via In-Distribution Knowledge Distillation for Decentralized Learning
Deepak Ravikumar, Gobinda Saha, Sai Aparna Aketi, Kaushik Roy
Knowledge Distillation Data Set Decentralized Learning Heterogeneous Data Training Data Distribution

January 28, 2023

Master's Thesis: Out-of-distribution Detection with Energy-based Models
Sven Elflein
Deep Learning Adversarial Example Distribution Detection Deep Network Expert Knowledge Energy Based Model Training Data Distribution Electronic Thesis Energy Based Prior

December 4, 2022

Domain Adaptation and Generalization on Functional Medical Images: A Systematic Survey
Gita Sarafraz, Armin Behnamnia, Mehran Hosseinzadeh, Ali Balapour, Amin Meghrazi, Hamid R. Rabiee
Domain Adaptation Strong Generalization Comprehensive Survey Functional Magnetic Resonance Imaging Brain Signal Training Data Distribution

November 15, 2022

Empirical Study on Optimizer Selection for Out-of-Distribution Generalization
Hiroki Naganuma, Kartik Ahuja, Shiro Takagi, Tetsuya Motokawa, Rio Yokota, Kohta Ishikawa, Ikuro Sato, Ioannis Mitliagkas
Empirical Study Distribution Shift Distribution Generalization Training Data Distribution Class Out of Distribution Distribution Performance

November 3, 2022

FedGen: Generalizable Federated Learning for Sequential Data
Praveen Venkateswaran, Vatche Isahagian, Vinod Muthusamy, Nalini Venkatasubramanian
Federated Learning Better Generalization Sequential Data Invariant Risk Minimization Training Distribution Training Data Distribution Generalization Strategy

September 30, 2022

Rethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarks
Mahdi Morafah, Saeed Vahidian, Chen Chen, Mubarak Shah, Bill Lin
Pseudo Label Data Heterogeneity Model Aggregation Statistical Heterogeneity Training Data Distribution Novel Concept Human Aligned Benchmark

August 20, 2022

FLIS: Clustered Federated Learning via Inference Similarity for Non-IID Data Distribution
Mahdi Morafah, Saeed Vahidian, Weijia Wang, Bill Lin
Data Distribution Training Data Distribution

June 7, 2022

Generalized Data Distribution Iteration
Jiajun Fan, Changnan Xiao
Deep Reinforcement Learning Policy Iteration Training Data Distribution Iterative Stochastic

March 1, 2022

Low-Cost On-device Partial Domain Adaptation (LoCO-PDA): Enabling efficient CNN retraining on edge devices
Aditya Rajagopal, Christos-Savvas Bouganis
Convolutional Neural Network Domain Adaptation Efficient CNN Training Data Distribution Partial Domain Adaptation

November 23, 2021

Jointly Learning from Decentralized (Federated) and Centralized Data to Mitigate Distribution Shift
Sean Augenstein, Andrew Hard, Kurt Partridge, Rajiv Mathews
LeArning Abstract Distribution Shift Federated Prompt Cooperation Decentralized FL Device Training DIstribution Estimation Private Training Training Data Distribution Centralized Data

Training Data Distribution

Papers

Logit Scaling for Out-of-Distribution Detection

Equidistribution-based training of Free Knot Splines and ReLU Neural Networks

WeiPer: OOD Detection using Weight Perturbations of Class Projections

Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization

FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregation

Exploring the Impact of Training Data Distribution and Subword Tokenization on Gender Bias in Machine Translation

Homogenizing Non-IID datasets via In-Distribution Knowledge Distillation for Decentralized Learning

Master's Thesis: Out-of-distribution Detection with Energy-based Models

Domain Adaptation and Generalization on Functional Medical Images: A Systematic Survey

Empirical Study on Optimizer Selection for Out-of-Distribution Generalization

FedGen: Generalizable Federated Learning for Sequential Data

Rethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarks

FLIS: Clustered Federated Learning via Inference Similarity for Non-IID Data Distribution

Generalized Data Distribution Iteration

Low-Cost On-device Partial Domain Adaptation (LoCO-PDA): Enabling efficient CNN retraining on edge devices

Jointly Learning from Decentralized (Federated) and Centralized Data to Mitigate Distribution Shift