High Probability Generalization

High-probability generalization in machine learning focuses on developing theoretical guarantees for the performance of models on unseen data, moving beyond empirical observations. Current research investigates this through various lenses, including analyzing the stability and generalization bounds of algorithms like stochastic gradient descent and triplet learning, exploring the role of architectural design and input variations in models such as graph neural networks and transformer-based multimodal models, and examining the limitations of relying solely on statistical generalization for understanding complex models like large language models. These efforts aim to improve the reliability and robustness of machine learning systems across diverse applications by providing a deeper understanding of their generalization capabilities.

Papers

August 14, 2024

Learning Decisions Offline from Censored Observations with {\epsilon}-insensitive Operational Costs
Minxia Chen, Ke Fu, Teng Huang, Miao Bai
Data Driven Decision Censored Time to Event Price Sensitivity High Probability Generalization

July 4, 2024

Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Tasks
Amit Parekh, Nikolas Vitsakis, Alessandro Suglia, Ioannis Konstas
Integral Role Multimodal Model Manipulation Task Multimodal Transformer Multimodal Input Task Complexity Diverse Instruction High Probability Generalization

June 7, 2024

A Manifold Perspective on the Statistical Generalization of Graph Neural Networks
Zhiyang Wang, Juan Cervino, Alejandro Ribeiro
Graph Neural Network Graph Convolution Generalization Gap Training Graph Manifold Dimension High Probability Generalization

May 3, 2024

Position: Understanding LLMs Requires More Than Statistical Generalization
Patrik Reizinger, Szilvia Ujváry, Anna Mészáros, Anna Kerekes, Wieland Brendel, Ferenc Huszár
Medical LLM Inductive Bias Model Generalization Product Specific Position Information Deep Learning Theory High Probability Generalization Non Identifiability

February 20, 2023

On the Stability and Generalization of Triplet Learning
Jun Chen, Hong Chen, Xue Jiang, Bin Gu, Weifu Li, Tieliang Gong, Feng Zheng
Strong Generalization Stochastic Gradient Descent Core Stability Generalization Guarantee Triplet Learning High Probability Generalization

December 14, 2022

Cross-Modal Similarity-Based Curriculum Learning for Image Captioning
Hongkuan Zhang, Saku Sugawara, Akiko Aizawa, Lei Zhou, Ryohei Sasano, Koichi Takeda
Vision Language Model Image Captioning Curriculum Learning Modal Similarity High Probability Generalization

December 1, 2022

Task Discovery: Finding the Tasks that Neural Networks Generalize on
Andrei Atanov, Andrei Filatov, Teresa Yeo, Ajay Sohmshetty, Amir Zamir
Deep Learning Model Inductive Bias New Task Neural Network Generalization High Probability Generalization

October 12, 2022

A New Family of Generalization Bounds Using Samplewise Evaluated CMI
Fredrik Hellström, Giuseppe Durisi
Generalization Bound Information Theoretic KL Divergence Conditional Mutual Information Information Theoretic Generalization Bound High Probability Generalization New Family

October 3, 2022

On Stability and Generalization of Bilevel Optimization Problem
Meng Ding, Mingxi Lei, Yunwen Lei, Di Wang, Jinhui Xu
Strong Generalization Core Stability Bilevel Optimization Hyper Parameter Optimization Strongly Convex High Probability Generalization

May 27, 2022

Generalization Bounds for Gradient Methods via Discrete and Continuous Prior
Xuanyuan Luo, Luo Bei, Jian Li
Gradient Descent Generalization Bound Langevin Dynamic Gradient Method Discrete Environment Prior Model High Probability Generalization Algorithm Dependent Generalization

February 26, 2022

Symmetric Convolutional Filters: A Novel Way to Constrain Parameters in CNN
Harish Agrawal, Sumana T., S. K. Nandy
State of the Art CNN Network Pruning Method Symmetric Neural Network Novel Way High Probability Generalization