Learning Curve

Learning curves, which plot a machine learning model's performance against training data size, are crucial for understanding model behavior and optimizing training processes. Current research focuses on improving learning curve estimation techniques, particularly for deep learning models like convolutional neural networks (CNNs) and language models, to predict performance, reduce training time, and identify optimal hyperparameters. This work addresses issues like overoptimism in published results and the unexpected phenomenon of "grokking," where performance suddenly improves after a long period of stagnation, ultimately aiming to enhance the efficiency and reliability of machine learning across various applications.

Papers

October 23, 2024

A Comprehensive Analysis on the Learning Curve in Kernel Ridge Regression
Tin Sum Cheng, Aurelien Lucchi, Anastasis Kratsios, David Belius
Generalization Performance Novel Regression Comprehensive Analysis Ridge Regression Learning Curve Eigenvalue Decay

October 11, 2024

Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structure
Jihao Andreas Lin, Sebastian Ament, Maximilian Balandat, Eytan Bakshy
Gaussian Process Learning Rate Efficient Inference Hyper Parameter Intermediate Latent Kronecker Product Learning Curve

October 10, 2024

Meta-Learning from Learning Curves for Budget-Limited Algorithm Selection
Manh Hung Nguyen, Lisheng Sun-Hosoya (LISN), Isabelle Guyon
Practical Algorithm Algorithm Selection Aggregated Curve Known Algorithm Learning Curve

May 23, 2024

Unraveling overoptimism and publication bias in ML-driven science
Pouria Saidi, Gautam Dasarathy, Visar Berisha
Machine Learning Machine Learning Model Overconfident Prediction Learning Curve Publication Bias Reward Overoptimization

October 12, 2023

Strategies and impact of learning curve estimation for CNN-based image classification
Laura Didyk, Brayden Yarish, Michael A. Beck, Christopher P. Bidinosti, Christopher J. Henry
Training Data Machine Learning Model Global Impact General Strategy Image Classification Task Learning Curve CNN Classifier Curve Modeling

August 29, 2023

Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability
Tyler A. Chang, Zhuowen Tu, Benjamin K. Bergen
Language Model LeArning Abstract Core Stability Anti Forgetting Autoregressive Language Model N Gram Learning Curve Language Model Pre Training

July 6, 2023

Learning Curves for Noisy Heterogeneous Feature-Subsampled Ridge Ensembles
Benjamin S. Ruben, Cengiz Pehlevan
Linear Classifier Learning Curve Heterogeneous Feature Ridge Ensemble

June 23, 2023

Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok
Pascal Jr. Tikeng Notsawo, Hattie Zhou, Mohammad Pezeshki, Irina Rish, Guillaume Dumas
Neural Network Full Model Model Overfitting Glance Annotation Loss Landscape Grokking Phenomenon Spectral Signature Learning Curve

March 1, 2023

Learning curves for deep structured Gaussian feature models
Jacob A. Zavatone-Veth, Cengiz Pehlevan
Random Feature Deep Learning Theory Random Weight Learning Curve

November 25, 2022

A Survey of Learning Curves with Bad Behavior: or How More Data Need Not Lead to Better Performance
Marco Loog, Tom Viering
Timely Survey Training Data Generalization Performance Performance Improvement Learning Curve Parametric Curve

August 31, 2022

Fine-Grained Distribution-Dependent Learning Curves
Olivier Bousquet, Steve Hanneke, Shay Moran, Jonathan Shafer, Ilya Tolstikhin
Fine Grained Learning Rate PAC Learning Hypothesis Class PAC Bayes Bound Learning Curve

August 4, 2022

Meta-learning from Learning Curves Challenge: Lessons learned from the First Round and Design of the Second Round
Manh Hung Nguyen, Lisheng Sun, Nathan Grinsztajn, Isabelle Guyon
Product Design Meta Learning Challenge Task Critical Lesson Meta Dataset Learning Curve Multi Round

May 30, 2022

Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression
Lechao Xiao, Hong Hu, Theodor Misiakiewicz, Yue M. Lu, Jeffrey Pennington
High Dimensional Kernel Ridge Regression Learning Curve Scaling Limit Dot Product Kernel Scaling Relation Large Sample

May 22, 2022

Investigating classification learning curves for automatically generated and labelled plant images
Michael A. Beck, Christopher P. Bidinosti, Christopher J. Henry, Manisha Ajmani
Supervised Learning Learning Rate Classification Task Classification Performance Learnable Parameter Learning Curve Leaf Image

March 22, 2022

Learning curves for the multi-class teacher-student perceptron
Elisabetta Cornacchia, Francesca Mignacco, Rodrigo Veiga, Cédric Gerbelot, Bruno Loureiro, Lenka Zdeborová
High Dimensional Teacher Network Random Weight Learning Curve

January 28, 2022

Learning Curves for Decision Making in Supervised Machine Learning -- A Survey
Felix Mohr, Jan N. van Rijn
Machine Learning Timely Survey Decision Making Model Selection Supervised Machine Learning Learning Curve Parametric Curve

December 3, 2021

Learning Curves for Continual Learning in Neural Networks: Self-Knowledge Transfer and Forgetting
Ryo Karakida, Shotaro Akaho
Neural Network Strong Generalization Transfer Learning Continual LEArning Generalization Performance Sequential Learning Learning Curve

November 5, 2021

NAS-Bench-x11 and the Power of Learning Curves
Shen Yan, Colin White, Yash Savani, Frank Hutter
Neural Architecture Search Real Power Surrogate Model Learning Curve Single Fidelity

Learning Curve

Papers

A Comprehensive Analysis on the Learning Curve in Kernel Ridge Regression

Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structure

Meta-Learning from Learning Curves for Budget-Limited Algorithm Selection

Unraveling overoptimism and publication bias in ML-driven science

Strategies and impact of learning curve estimation for CNN-based image classification

Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability

Learning Curves for Noisy Heterogeneous Feature-Subsampled Ridge Ensembles

Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok

Learning curves for deep structured Gaussian feature models

A Survey of Learning Curves with Bad Behavior: or How More Data Need Not Lead to Better Performance

Fine-Grained Distribution-Dependent Learning Curves

Meta-learning from Learning Curves Challenge: Lessons learned from the First Round and Design of the Second Round

Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression

Investigating classification learning curves for automatically generated and labelled plant images

Learning curves for the multi-class teacher-student perceptron

Learning Curves for Decision Making in Supervised Machine Learning -- A Survey

Learning Curves for Continual Learning in Neural Networks: Self-Knowledge Transfer and Forgetting

NAS-Bench-x11 and the Power of Learning Curves