Training Loss

Training loss, a measure of a model's error during learning, is central to optimizing machine learning models and understanding their behavior. Current research focuses on developing novel loss functions tailored to specific tasks and data types, such as those addressing noisy data, imbalanced classes, and strategic manipulation, often incorporating techniques like curriculum learning and adaptive regularization. These advancements aim to improve model accuracy, robustness, and efficiency, impacting diverse applications from recommendation systems and natural language processing to 3D point cloud processing and software engineering. The exploration of the relationship between training loss and emergent abilities in large language models is also a significant area of investigation.

Papers

October 29, 2024

Abrupt Learning in Transformers: A Case Study on Matrix Completion
Pulkit Gopalani, Ekdeep Singh Lubana, Wei Hu
Transformer Megatron Decepticons Case Study BERT Model Training Dynamic Matrix Completion Fast Learning Training Loss

October 14, 2024

The Epochal Sawtooth Effect: Unveiling Training Loss Oscillations in Adam and Other Optimizers
Qi Liu, Wanjing Ma
Gradient Based Training Loss Precise Inter EPOCH Feature

September 10, 2024

Loss Distillation via Gradient Matching for Point Cloud Completion with Weighted Chamfer Distance
Fangzhou Lin, Haotian Liu, Haoying Zhou, Songlin Hou, Kazunori D Yamada, Gregory S. Fischer, Yanhua Li, Haichong K. Zhang, Ziming Zhang
Loss Function Point Cloud Completion Distillation Loss Gradient Matching Training Loss Chamfer Distance

March 23, 2024

Understanding Emergent Abilities of Language Models from the Loss Perspective
Zhengxiao Du, Aohan Zeng, Yuxiao Dong, Jie Tang
Language Model Large Model Emergent Ability Training Loss

February 29, 2024

Loss-aware Curriculum Learning for Heterogeneous Graph Neural Networks
Zhen Hao Wong, Hansi Yang, Xiaoyi Fu, Quanming Yao
Graph Neural Network Curriculum Learning Heterogeneous Graph Heterogeneous Graph Neural Network Training Loss

February 22, 2024

Scaling Efficient LLMs
B. N. Kausik
Training Corpus Training Loss Efficient LLM

January 18, 2024

Keeping Deep Learning Models in Check: A History-Based Approach to Mitigate Overfitting
Hao Li, Gopi Krishnan Rajbahadur, Dayi Lin, Cor-Paul Bezemer, Zhen Ming, Jiang
Deep Learning Model Model Overfitting Faulty Negative Mitigation Training Loss

November 6, 2023

AdaFlood: Adaptive Flood Regularization
Wonho Bae, Yi Ren, Mohamad Osama Ahmed, Frederick Tung, Danica J. Sutherland, Gabriel L. Oliveira
Test Time Domain Adaptation Training Loss

August 7, 2023

Unsupervised Adversarial Detection without Extra Model: Training Loss Should Change
Chien Cheng Chyou, Hung-Ting Su, Winston H. Hsu
Adversarial Attack Full Model Adversarial Training Adversarial Robustness Cross Entropy Loss Adversarial Detection Training Loss

June 19, 2023

Scaling of Class-wise Training Losses for Post-hoc Calibration
Seungjin Jung, Seungmo Seo, Yonghyun Jeong, Jongwon Choi
Multiplicative Size Scaling Intra Class Variation Training Loss Scaling Factor

June 7, 2023

Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning
Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan, Mikhail Belkin
Neural Network Strong Generalization Gradient Descent Stochastic Gradient Descent Feature Learning Better Generalization Large Learning Rate Training Loss

May 18, 2023

Integrating Item Relevance in Training Loss for Sequential Recommender Systems
Andrea Bacciu, Federico Siciliano, Nicola Tonellotto, Fabrizio Silvestri
Recommender System Sequential Recommender System Training Loss Evaluation Protocol Item Relevance Class Relevance Learning

February 24, 2023

On the Training Instability of Shuffling SGD with Batch Normalization
David X. Wu, Chulhee Yun, Suvrit Sra
Gradient Descent Batch Normalization Inverse Divergence Training Loss Training Instability Shuffling Gradient

November 29, 2022

Neural Speech Phase Prediction based on Parallel Estimation Architecture and Anti-Wrapping Losses
Yang Ai, Zhen-Hua Ling
Online Estimation Phase Shift Training Loss Neural Speech Phase Prediction Speech Phase Prediction

August 17, 2022

CTRL: Clustering Training Losses for Label Error Detection
Chang Yue, Niraj K. Jha
Noisy Label GTR CTRL Label Error Training Loss Label Corruption Clustering Loss

July 23, 2022

A Universal Trade-off Between the Model Size, Test Loss, and Training Loss of Linear Predictors
Nikhil Ghosh, Mikhail Belkin
Model Size Overparametrization Bound Optimal Trade Training Loss Linear Predictor Test Loss

May 19, 2022

EXACT: How to Train Your Accuracy
Ivan Karpukhin, Stanislav Dereka, Sergey Kolesnikov
Stochastic Process Gradient Ascent Stochastic Model Training Loss

March 25, 2022

Learning Losses for Strategic Classification
Tosca Lechner, Ruth Urner
Transfer Learning Strategic Manipulation Manipulation Strategy Strategic Classification Training Loss Optimal Decision Rule