cs.LGNov 3, 2026

Stepsize modulation for cross entropy from local quadratic optimization

Authors: Bruno Bauwens, Viacheslav Krivodub, Bingyu Zhang

Organizations: National Research University Higher School of Economics, Faculty of Computer Science, Moscow, Russia · Luxoft Serbia

Abstract

Two methods are presented that perturb a given learning rate schedule in stochastic gradient descent with cross entropy loss. Both methods use an extra forward pass to estimate various derivatives of the minibatch loss. The first method selects a value ηη by linearising the softmax values and minimizing the squared error 12∣∣y−q−ηq′∣∣F2\tfrac 1 2 ||y - q - ηq'||_F^2, where qq is the matrix representing the softmax values of the minibatch and yy is the matrix with one-hot encodings of the labels. The second method considers a quadratic Taylor approximation of the minibatch loss under the outer approximation of the Hessian. Thus, it minimizes L(0)+ηL′(0)+12η2L′′(0)L(0) + ηL'(0) + \tfrac 1 2 η^2 L''(0). Here, the derivatives L′L' and L′′L'' are computed from the derivatives of the logits (using the outer approximation of the Hessian of the loss). We observe that both methods give very similar values of ηη. Finally, these estimated ηη are used to slightly increase or decrease the learning rate of an existing schedule. The method is evaluated on Cifar10 and Cifar100 datasets for training Resnets, where they decrease the loss faster than the schedule that is perturbed and increase monotonicity. For Cifar10 it also leads to a small improvement of test accuracy, but in Cifar100 test accuracy remains the same.

Figures & tables

Explore similar work

Mar 13, 2026

Ghosts of Softmax: Complex Singularities That Limit Safe Step Sizes in Cross-Entropy

Optimization analyses for cross-entropy training rely on local Taylor models of the loss to predict whether a proposed step will decrease the objective. These surrogates are reliable only inside the Taylor convergence radius of the true loss along the update direction. That radius is set not by real-line curvature alone but by the nearest complex singularity. For cross-entropy, the softmax partition function F=∑jexp⁡(zj)F=\sum_j \exp(z_j) has complex zeros -- ``ghosts of softmax'' -- that induce logarithmic singularities in the loss and cap this radius. To make this geometry usable, we derive closed-form expressions under logit linearization along the proposed update direction. In the binary case, the exact radius is ρ∗=δ2+π2/Δaρ^*=\sqrt{δ^2+ π^2}/Δ_a. In the multiclass case, we obtain the lower bound ρa=π/Δaρ_a=π/Δ_a, where Δa=max⁡kak−min⁡kakΔ_a=\max_k a_k-\min_k a_k is the spread of directional logit derivatives ak=∇zk⋅va_k=\nabla z_k\cdot v. This bound costs one Jacobian-vector product and reveals what makes a step fragile: samples that are both near a decision flip and highly sensitive to the proposed direction tighten the radius. The normalized step size r=τ/ρar=τ/ρ_a separates safe from dangerous updates. Across six tested architectures and multiple step directions, no model fails for r<1r<1, yet collapse appears once r≥1r\ge 1. Temperature scaling confirms the mechanism: normalizing by ρaρ_a shrinks the onset-threshold spread from standard deviation 0.9920.992 to 0.1640.164. A controller that enforces τ≤ρaτ\leρ_a survives learning-rate spikes up to 10,000×10{,} 000\times in our tests, where gradient clipping still collapses. Together, these results identify a geometric constraint on cross-entropy optimization that operates through Taylor convergence rather than Hessian curvature.
Jul 12, 2026

WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training

Standard learning rate schedules such as cosine annealing are tied to a fixed training horizon, limiting their ability to accommodate post hoc horizon extension. Warmup-stable-decay (WSD) partially addresses this issue by maintaining a long constant-rate phase before a short linear cooldown, allowing training to resume from a pre-decay checkpoint. However, its peak learning rate is still tuned based on the original training horizon and can become suboptimal when training is extended. Motivated by stochastic convex optimization, we propose WSqD (Warmup with Square-root base and linear Decay), a learning rate schedule that replaces WSD's constant stable phase with a shifted inverse-square-root base while retaining the final linear cooldown. In the stochastic convex setting, WSqD provably attains the minimax-optimal O(1/T)O(1/\sqrt{T}) last-iterate convergence rate. Importantly, its base learning rate schedule is horizon-independent, and the training horizon is needed only to determine when to begin the final cooldown. Empirically, on language-model pretraining using the SlimPajama corpus, WSqD matches or outperforms carefully tuned WSD and other baselines across multiple training horizons while reusing a single peak learning rate.
Oct 22, 2026

A Fix for Softmax and Cross-Entropy

Standard implementations of softmax, log-softmax, and cross-entropy have two numerical bugs. The first bug, Softmax Collapse, rounds the target logit's gradient to zero if that one logit is much larger than the others, resulting in gradients that no longer sum to zero. The second bug, Softmax Cancellation, occurs when student logits are close to teacher logits, and produces non-zero gradients even when the student and teacher logits are equal. We present simple implementations of log-softmax, softmax, and cross-entropy that fix both bugs. Our fix eliminates the previously observed "slingshot" phenomenon, speeds up grokking with weight decay, prevents accuracy collapse without weight decay, stops student networks from drifting away from the teacher during distillation, and shows that after memorization, logit growth is bounded by an equilibrium between weight decay and Adam's εε hyperparameter.