The Symmetric Perceptron: a Teacher-Student Scenario
Authors: Giovanni Catania, Aurélien Decelle, Suhanee Korpe
Organizations: Departamento de Física Teórica, Universidad Complutense de Madrid, 28040 Madrid, Spain · Institute for Cross-disciplinary Physics and Complex Systems IFISC (CSIC-UIB), Campus Universitat Illes Balears, 07122 Palma de Mallorca, Spain. · Université Paris-Saclay, CNRS, INRIA Tau team, LISN, 91190 Gif-sur-Yvette, France. · GISC - Grupo Interdisciplinar de Sistemas Complejos 28040 Madrid, France. · Indian Institute of Science Education and Research, Bhopal 462066, India
We introduce and solve a teacher-student formulation of the symmetric binary Perceptron, turning a traditionally storage-oriented model into a planted inference problem with a guaranteed solution at any sample density. We adapt the formulation of the symmetric Perceptron which traditionally considers either the u-shaped potential or the rectangular one, by including labels in both regions. With this formulation, we analyze both the Bayes-optimal regime at for noise-less examples and the effect of thermal noise under two different potential/classification rules. Using annealed and quenched free-entropy calculations in the high-dimensional limit, we map the phase diagram in the three control parameters, namely the sample density α, the distance between the origin and one of the symmetric hyperplanes κ and temperature T, and identify a robust scenario where learning is organized by a second-order instability that creates teacher-correlated suboptimal states, followed by a first-order transition to full alignment. We show how this structure depends on the choice of potential, the interplay between metastability of the suboptimal solution and its melting towards the planted configuration, which is relevant for Monte Carlo-based optimization algorithms.
Figures & tables
Figure 1: Decision boundary for a symmetric Perceptron in N=2 with w=[−1/\penalty2,1/\penalty2] and κ=1 , illustrating a case where the data in the red region are labelled +1 , while the ones in the blue region are labeled −1 .
Figure 2: Left: the potential V(0) for both labels σ0=±1 . With this potential, any error has a fixed cost. Right: the linear potential V(1) for both labels σ0=±1 . This potential tends to penalize more errors that are far away from the decision boundary.
Figure 3: Snapshots of free energy profiles at 6 different values of κ , corresponding to the thin vertical lines in Fig. 4 (left panel). Each panel ( a ) → ( f ) shows several free energy profiles f(R) at different values of α . The thicker curves correspond to the values of the first/second order transition and the spinodal (when different). In blue we illustrate first order transitions, in red second order ones and in yellow the melting (spinodal) point of the sub-optimal solution 0<R<1 . The colors used here are the same as in Fig. 4 .
Figure 4: Left: Phase diagram of the symmetric binary Perceptron at T=0 , using the annealed approximation. Right: Phase diagram of the symmetric binary Perceptron at T=0 , using the quenched approximation in the Bayes-Optimal setting. The vertical lines indicate the values of κ used on Fig. 3 (resp. Fig. 5 ) for the annealed (resp. quenched) case. In both cases, the blue line indicated the first order transition, the red line the second order one and the yellow line the spinodal, that is when the sub-optimal solution disappear. In the range of κ where the second order phase transition occurs, the dashed-blue line only corresponding to when the unstable paramagnetic state and the teacher state have the same free energy.
Figure 5: Snapshots of free energy profiles at 6 different values of κ at T=0 in the Bayes-Optimal setting. We can observe a phase diagram that is qualitatively similar to the annealed case. From small values of κ , we observe first a first order phase transition, follow by a melting toward the teacher state at higher values of α . Then, we have the at larger κ a second order phase transition before the melting.
Figure 6: Left: Phase diagram of the constant potential V0 at κ=1 . Right: Phase diagram of the piece-wise constant potential V1 at κ=1 . As in the case of the Perceptron, the orange line is expected to terminate at (0,0) . In this parameter regime, the shape of the potential strongly influences the structure of the loss landscape. For V0 , and for sufficiently large T and α , there exists a phase in which the teacher dominates the equilibrium measure, while a typical experiment remains trapped in completely uncorrelated states. In contrast, for V1 , the second-order phase transition is always dominant with respect to the other transitions and provides information about the teacher well before the latter even becomes a metastable state.