Adaptive Loss Weighting

Adaptive loss weighting studies how to allocate training emphasis when examples, tasks, or objectives differ in difficulty, importance, noise, or learning progress. Common approaches use gradient norms or uncertainty, track loss and prediction dynamics, align gradients, or learn weights through meta-learning and bilevel optimization. These methods appear in settings such as physics-informed neural networks, multi-task learning, long-tailed classification, diffusion, language-model fine-tuning, forecasting, and federated learning. A central tension is that weights that improve one metric or objective can harm another, so stability, task-specific performance, and evaluation beyond a single weighted loss remain important.