Generalization in Learning: Eyring Formula and Predator-prey Model

S. V. Kozyrev; I. A. Lopatin; Alexander Pechen

cs.LGOct 25, 2026

Generalization in Learning: Eyring Formula and Predator-prey Model

Authors: S. V. Kozyrev, I. A. Lopatin, Alexander Pechen

Organizations: Steklov Mathematical Institute of Russian Academy of Sciences, Moscow, Russia · Ivannikov Institute for System Programming of the Russian Academy of Sciences, Moscow, Russia

Abstract

In this work, generalization in machine learning is investigated using a predator-prey model. We overview biological and chemical interpretation of the stochastic gradient Langevin dynamics and generative adversarial network. For a particular example of recovering an unknown function by a polynomial of fixed degree from a set of noisy data, the predator-prey model provides better approximation compared to the gradient descent method. Further, thermodynamical arguments, in particular Eyring formula, are used to explain grokking (delayed generalization) phenomenon in machine learning.

Figures & tables

Figure 1: Sampling examples of training and test datasets for the function $f(x)=\sin(\pi x)$ .

Degree $d$	1	3	5	7
Loss on train	0.171	0.072	0.027	0.026
Loss on test	0.400	0.033	0.161	0.159
Deviation from $f$	0.227	0.007	0.059	0.080

Table 1: Learning results for different degrees of polynomials. The last line of the table shows the MSE of the model predictions for the hidden function on a uniform grid on

[-1,1]

with

1000

points.

Figure 2: Solid lines show approximations to the function $f$ (dashed lines) found by gradient descent after $10^{6}$ iterations for $d=1,3,5,7$ in ( 9 ). The train points are marked by ’x’.

Figure 3: Simulation results for: (a) loss on train; (b) loss on test; (c) distance between the predator and the prey; (d) distance between the current state $u_{k}$ and the averaged state $\overline{u}_{k}$ for the prey.

Figure 4: Deviation of the model from the hidden function vs number of iterations.

Neural Network Generalization

Neural network generalization studies why trained models perform well on unseen data and how architecture, data, and training shape that ability.

See all 305 papers

Predator-Prey Model: Driven Hunt for Accelerated Grokking
Sep 10, 2025I. A. Lopatin, S. V. Kozyrev, A. N. PechenDeep Learning Optimization Grokking
Steklov Mathematical Institute of Russian Academy of Sciences, Gubkina str. 8, 119991 Moscow, Russia · Ivannikov Institute for System Programming of the Russian Academy of Sciences, Moscow, 109004 Russia
A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
Sep 7, 2026Yuqing Wang, Ioannis G. Kevrekidis, Mikhail BelkinNeural Network Generalization Neural Network Training Dynamics
Johns Hopkins University · University of California San Diego
On the Dynamics of Delayed Generalization in Linear Models
Feb 9, 2026Atreya Vedantam, Nataraj Das, Chandrashekar LakshminarayananLinear Classification Neural Network Generalization
Indian Institute of Technology Madras, Chennai.

Sep 10, 2025J/K move · Enter open · S save

Predator-Prey Model: Driven Hunt for Accelerated Grokking

I. A. Lopatin, S. V. Kozyrev, A. N. Pechen

Steklov Mathematical Institute of Russian Academy of Sciences, Gubkina str. 8, 119991 Moscow, Russia · Ivannikov Institute for System Programming of the Russian Academy of Sciences, Moscow, 109004 Russia

Deep Learning Optimization Grokking

Open paper AskPDF arXiv

A machine learning method is proposed using two agents (representing two coupled replicas of the model parameters) that simulate the biological behavior of a predator and a prey. In this method, the two agents interact with each other via a pair of potentials that simulate the dynamics when the predator chases the prey while the prey runs away from the predator - to perform an optimization on the landscape. This method allows, for optimization landscapes with narrow ravines with gentle slope or even flat bottoms, to avoid optimization getting stuck in a ravine. For some examples of grokking (i.e., delayed generalization) problems we show that this method allows for achieving up to a hundred times faster learning in gradient evaluations and up to 60 times in the wall-clock time compared to the standard learning procedure.

Abstract

Neural Network Generalization

Explore similar work

Predator-Prey Model: Driven Hunt for Accelerated Grokking

A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay

On the Dynamics of Delayed Generalization in Linear Models