Aug 31, 2026 · cs.LGJ/K move · Enter open · S save
Xiaoyang Li, Runni Zhou
College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
Gradient flow is widely used as a continuous-time surrogate for gradient descent, but state convergence does not imply convergence of differentiated training maps in nonsmooth networks. We characterize the fixed-horizon, vanishing-step limit of exact automatic differentiation through hard-ReLU gradient descent. Under a stable finite itinerary of separated, same-direction transverse activation events, gradient-descent states converge at first order to the corresponding piecewise-smooth gradient flow, while the exact derivative of every nonresonant discrete program converges to an event-free regional propagator. The true flow derivative instead interleaves classical saltation matrices that encode event-time sensitivity. For globally convex objectives, any strict activation event prevents complete cancellation of these missing transfers. Moreover, minimal globally 1-strongly convex residual-ReLU risks can realize arbitrarily large reciprocal sensitivity gaps, subject to an explicit transversality-scale tradeoff, and a coupled strongly convex construction yields an open set on which the largest initialization-gradient coordinate is reversed. In a controlled 17-parameter ReLU MLP, state and regional-AD errors vanish under mesh refinement while AD-to-flow errors remain between 0.18 and 0.39; an event-aware corrected product restores convergence. Resolved smoothing likewise recovers the flow sensitivity when the transition layer is sufficiently resolved. These results show that the gradient-flow limit of hard-ReLU training need not remain valid after differentiation.