Generalization Behavior

Generalization behavior in machine learning investigates how well models trained on one dataset perform on unseen data, a crucial aspect for real-world applications. Current research focuses on understanding this behavior across various architectures, including recurrent neural networks, transformers, and generative flow networks, examining factors like model complexity, training dynamics (e.g., grokking, neural collapse), and the role of pretraining and knowledge distillation. These studies aim to improve model robustness and efficiency by identifying and controlling factors that influence generalization, ultimately leading to more reliable and adaptable AI systems. The insights gained are relevant to both theoretical advancements in understanding learning and to practical improvements in the design and deployment of machine learning models.

Papers

October 4, 2024

Measuring and Controlling Solution Degeneracy across Task-Trained Recurrent Neural Networks
Ann Huang, Satpreet H. Singh, Kanaka Rajan
Recurrent Neural Network Measurement System Neural Dynamic Model Degeneracy Generalization Behavior

October 2, 2024

Revisiting Optimism and Model Complexity in the Wake of Overparameterized Machine Learning
Pratik Patil, Jin-Hong Du, Ryan J. Tibshirani
Model Complexity Prediction Error Machine Learning Problem Generalization Behavior Arbitrary Machine Learning Model

June 12, 2024

Strategies for Pretraining Neural Operators
Anthony Zhou, Cooper Lorsung, AmirPouya Hemmasian, Amir Barati Farimani
Neural Operator General Strategy Physic Aware Generalization Behavior Physic Prediction

May 29, 2024

Deep Grokking: Would Deep Neural Networks Generalize Better?
Simin Fan, Razvan Pascanu, Martin Jaggi
Generalization Performance Multi Stage Neural Network Generalization Grokking Phenomenon Generalization Behavior

April 25, 2024

Learning Syntax Without Planting Trees: Understanding When and Why Transformers Generalize Hierarchically
Kabir Ahuja, Vidhisha Balachandran, Madhur Panwar, Tianxing He, Noah A. Smith, Navin Goyal, Yulia Tsvetkov
Transformer Megatron Decepticons Hierarchical Learning Syntactic Structure Generalization Behavior Individual Coniferous Tree Syntax Acquisition

February 7, 2024

Investigating Generalization Behaviours of Generative Flow Networks
Lazar Atanackovic, Emmanuel Bengio
Generative Model Generative Flow Network Generalization Property Generative Framework Energy Based GFlowNets Generalization Behavior

October 12, 2023

Towards Demystifying the Generalization Behaviors When Neural Collapse Emerges
Peifeng Gao, Qianqian Xu, Yibo Yang, Peisong Wen, Huiyang Shao, Zhiyong Yang, Bernard Ghanem, Qingming Huang
Neural Collapse Improved Generalization Margin Based Generalization Behavior

October 4, 2023

Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data
Zhiwei Xu, Yutong Wang, Spencer Frei, Gal Vardi, Wei Hu
Gradient Descent Model Overfitting ReLU Network Benign Overfitting Grokking Phenomenon Optimal Generalization Generalization Behavior

August 16, 2023

It Ain't That Bad: Understanding the Mysterious Performance Drop in OOD Generalization for Generative Transformer Models
Xingcheng Xu, Zihao Pan, Haipeng Zhang, Yanqing Yang
Strong Generalization Transformer Based Model Improved Generalization Performance Degradation Generalization Behavior Generative Transformer Model

January 28, 2023

Supervision Complexity and its Role in Knowledge Distillation
Hrayr Harutyunyan, Ankit Singh Rawat, Aditya Krishna Menon, Seungyeon Kim, Sanjiv Kumar
Knowledge Distillation Integral Role Online Distillation Generalization Behavior Co Supervision Explicit Supervision

May 19, 2022

Dataset Pruning: Reducing Training Data by Examining Generalization Influence
Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, Ping Li
Deep Learning Training Data Generalization Gap Training Datasets Generalization Behavior Dataset Pruning

March 1, 2022

On the Generalization of Representations in Reinforcement Learning
Charline Le Lan, Stephen Tu, Adam Oberman, Rishabh Agarwal, Marc G. Bellemare
Reinforcement Learning Strong Generalization Meaningful Representation State Representation Successor Representation Generalization Behavior

November 20, 2021

Building Object-based Causal Programs for Human-like Generalization
Bonan Zhao, Christopher G. Lucas, Neil R. Bramley
Domain Generalization Causal Structure Building PCC Causal Variable Human Like Generalization Generalization Behavior