We study the critical points over an algebraic variety of an optimization problem defined by a quadratic objective that is degenerate. This scenario arises in machine learning when the dataset size is small with respect to the model, and is typically referred to as overparametrization. Our main result relates the degenerate optimization problem to a nondegenerate one via a projection. In the highly-degenerate regime, we find that a central role is played by the ramification locus of the projection. Additionally, we provide tools for counting the number of critical points over projective varieties, and discuss specific cases arising from deep learning. Our work bridges tools from algebraic geometry with ideas from machine learning, and it extends the line of literature around the Euclidean distance degree to the degenerate setting.
Figures & tables
Figure 1 : A slice of the neuromanifold of lightning self-attention mechanisms (adapted from [ 12 ] ).
Figure 2 : Illustration of different scenarios, where X is either a curve (a) or a surface (b), (c) or (d) in R3 .
Figure 3 : Critical points (yellow) and normal lines (dashed) on the circle w.r.t. two different quadrics Q .
Table 1 : CritY,Q(u) for a generic point u∈R6 , where Y is the singular locus of X.