Better Generalization

Better generalization in machine learning aims to improve a model's ability to perform well on unseen data, a crucial aspect for real-world applications. Current research focuses on techniques like weight decay, sharpness minimization, and various continual learning strategies, often applied to neural networks including transformers and convolutional models, to achieve this goal. These efforts are driven by the need for more robust and reliable AI systems across diverse domains, impacting fields ranging from medical image analysis and natural language processing to climate modeling and robotics. Improved generalization ultimately leads to more trustworthy and effective AI deployments.

Papers

January 21, 2024

Understanding the Generalization Benefits of Late Learning Rate Decay
Yinuo Ren, Chao Ma, Lexing Ying
Learning Rate Better Generalization Large Learning Rate Optimal Generalization Generalization Benefit

January 16, 2024

lpNTK: Better Generalisation with Less Data via Sample Interaction During Learning
Shangmin Guo, Yi Ren, Stefano V. Albrecht, Kenny Smith
Neural Network LeArning Abstract Strong Generalization Generalization Performance Neural Tangent Kernel Better Generalization Less Data Interaction Understanding Empirical Neural Tangent Kernel

November 27, 2023

Should We Learn Most Likely Functions or Parameters?
Shikai Qiu, Tim G. J. Rudner, Sanyam Kapoor, Andrew Gordon Wilson
Estimation Task Many Parameter Posterior Inference Better Generalization Bayesian Posterior Maximum a Posteriori Function Prediction

November 23, 2023

Risk Bounds of Accelerated SGD for Overparameterized Linear Regression
Xuheng Li, Yihe Deng, Jingfeng Wu, Dongruo Zhou, Quanquan Gu
Strong Generalization Generalization Performance Better Generalization Faster Convergence Parameterized Model Accelerated Gradient Descent Risk Bound

November 13, 2023

Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains
Joshua Clymer, Garrett Baker, Rohan Subramani, Sam Wang
Reward Function Reward Model Internal Representation Better Generalization Domain Name Test Bed Human Like Generalization

September 30, 2023

Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learning
Mingde Zhao, Safa Alver, Harm van Seijen, Romain Laroche, Doina Precup, Yoshua Bengio
Reinforcement Learning Task Planning Better Generalization Hierarchical Planning Temporal Abstraction

September 25, 2023

Better Generalization of White Matter Tract Segmentation to Arbitrary Datasets with Scaled Residual Bootstrap
Wan Liu, Chuyang Ye
Better Generalization White Matter White Matter Tract Segmentation Residual Bootstrap

September 20, 2023

Incorporating Singletons and Mention-based Features in Coreference Resolution via Multi-task Learning for Better Generalization
Yilun Zhu, Siyao Peng, Sameer Pradhan, Amir Zeldes
Multi Task Learning Coreference Resolution Better Generalization Mention Detection Neural Coreference Resolution

September 19, 2023

Multi-fidelity climate model parameterization for better generalization and extrapolation
Mohamed Aziz Bhouri, Liran Peng, Michael S. Pritchard, Pierre Gentine
Machine Learning Better Generalization Extrapolation Framework Climate Projection Turbulence Simulation

August 2, 2023

Teaching Smaller Language Models To Generalise To Unseen Compositional Questions
Tim Hartill, Neset Tan, Michael Witbrock, Patricia J. Riddle
Language Model Zero Shot Question Answering Reasoning Ability Better Generalization Smaller Language Model Compositional Question

July 20, 2023

Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalization
Kaiyue Wen, Zhiyuan Li, Tengyu Ma
Neural Network Strong Generalization Sharpness Aware Minimization Better Generalization Two Layer ReLU Node Feature Sharpness

June 27, 2023

GenerTTS: Pronunciation Disentanglement for Timbre and Style Generalization in Cross-Lingual Text-to-Speech
Yahuan Cong, Haoyu Zhang, Haopeng Lin, Shichao Liu, Chunfeng Wang, Yi Ren, Xiang Yin, Zejun Ma
Text to Speech Speech Synthesis Better Generalization Speech Representation Disentanglement Multilingual Speech Timbre Perception Speaking Style

June 15, 2023

Building blocks for complex tasks: Robust generative event extraction for radiology reports under domain shifts
Sitong Zhou, Meliha Yetisgen, Mari Ostendorf
Generative Model Domain Adaptation Domain Shift Generative Approach Radiology Report Better Generalization Complex Task Honeycomb Block

June 13, 2023

Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations
Anima Singh, Trung Vu, Nikhil Mehta, Raghunandan Keshavan, Maheswaran Sathiamoorthy, Yilin Zheng, Lichan Hong, Lukasz Heldt, Li Wei, Devansh Tandon, Ed H. Chi, Xinyang Yi
Case Study Movie Recommendation Recommendation Model Partial Ranking Better Generalization Item Representation Hash Code Semantic ID

June 7, 2023

Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning
Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan, Mikhail Belkin
Neural Network Strong Generalization Gradient Descent Stochastic Gradient Descent Feature Learning Better Generalization Large Learning Rate Training Loss

May 25, 2023

Sharpness-Aware Minimization Revisited: Weighted Sharpness as a Regularization Term
Yun Yue, Jiadi Jiang, Zhiling Ye, Ning Gao, Yongchao Liu, Ke Zhang
Strong Generalization Sharpness Aware Minimization Better Generalization Improved Generalization Flat Minimum Regularization Term Node Feature Sharpness

May 15, 2023

GeNAS: Neural Architecture Search with Better Generalization
Joonhyun Jeong, Joonsang Yu, Geondo Park, Dongyoon Han, YoungJoon Yoo
Neural Architecture Search Neural Architecture Better Generalization

May 1, 2023

Towards the Flatter Landscape and Better Generalization in Federated Learning under Client-level Differential Privacy
Yifan Shi, Kang Wei, Li Shen, Yingqi Liu, Xueqian Wang, Bo Yuan, Dacheng Tao
Better Generalization Differentially Private Perturbation Robustness Differentially Private Federated Learning Client Level Differential Privacy Return Landscape

April 27, 2023

Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization
Hamza Kheddar, Yassine Himeur, Somaya Al-Maadeed, Abbes Amira, Faycal Bensaali
Deep Learning Automatic Speech Recognition Transfer Learning Better Generalization Deep Transfer Learning Deep Transfer Training Datasets

April 10, 2023

Simulated Annealing in Early Layers Leads to Better Generalization
Amirmohammad Sarfi, Zahra Karimpour, Muawiz Chaudhary, Nasir M. Khalid, Mirco Ravanelli, Sudhir Mudur, Eugene Belilovsky
Transfer Learning Shot Learning Better Generalization Improved Generalization Simulated Annealing Last Layer Early Layer

Better Generalization

Papers

Understanding the Generalization Benefits of Late Learning Rate Decay

lpNTK: Better Generalisation with Less Data via Sample Interaction During Learning

Should We Learn Most Likely Functions or Parameters?

Risk Bounds of Accelerated SGD for Overparameterized Linear Regression

Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains

Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learning

Better Generalization of White Matter Tract Segmentation to Arbitrary Datasets with Scaled Residual Bootstrap

Incorporating Singletons and Mention-based Features in Coreference Resolution via Multi-task Learning for Better Generalization

Multi-fidelity climate model parameterization for better generalization and extrapolation

Teaching Smaller Language Models To Generalise To Unseen Compositional Questions

Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalization

GenerTTS: Pronunciation Disentanglement for Timbre and Style Generalization in Cross-Lingual Text-to-Speech

Building blocks for complex tasks: Robust generative event extraction for radiology reports under domain shifts

Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations

Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning

Sharpness-Aware Minimization Revisited: Weighted Sharpness as a Regularization Term

GeNAS: Neural Architecture Search with Better Generalization

Towards the Flatter Landscape and Better Generalization in Federated Learning under Client-level Differential Privacy

Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization

Simulated Annealing in Early Layers Leads to Better Generalization