cs.LGOct 30, 2026

Gradient Descent on Two ReLU Neurons: Global Landscape and Bifurcation Dynamics

Authors: Binghua Li, Mengzhe Li, Denny Wu, Tianhao Wang

Organizations: University of California, Berkeley · École Normale Supérieure Paris-Saclay · New York University & Flatiron Institute · University of California, San Diego

Abstract

Understanding how gradient descent learns features remains a central challenge in neural network theory. We study this question in a minimal multi-index setting that already exhibits rich multi-phase gradient dynamics: a well-specified two-neuron ReLU teacher-student model under isotropic Gaussian inputs, trained by population gradient flow from small initialization. We show that the dynamics are organized by two directions: the "easy" bisector direction, which carries the leading signal, and the "hard" splitting direction, which governs specialization. To characterize the loss structure and learning behavior, we analyze the population squared-loss landscape and show that every critical point with both student weights nonzero is either a global minimum or a saddle. We then track the gradient-flow trajectory, showing that student neurons first collapse toward a bisector saddle before escaping and specializing to teacher neurons. Our results provide both a landscape and dynamics account of the multi-phase symmetry-breaking behavior that arises in a simple multi-index model and under standard algorithmic and architectural choices.

Figures & tables

Explore similar work

CardsList
  1. When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks

    Jun 3, 2026Berk Tinaz, Changzhi Xie, Mahdi SoltanolkotabiReLU Neural NetworksNeural Network Optimization

  2. Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks

    Apr 10, 2026Jie Huang, Bruno Loureiro, Stefano Sarao MannelliOverparameterizationLoss Landscape

  3. State-Space NTK Collapse Near Bifurcations

    May 12, 2026James Hazelden, Eric Shea-BrownRecurrent Neural NetworksNeural Network Training Dynamics