cs.LGFeb 4, 2026

Laws of Learning Dynamics and the Core of Learners

Authors: Inkee Jung, Siu Cheong Lau

Organizations: Department of mathematics, Boston University, Boston, USA.

Abstract

We formulate an algorithm-independent theory of learning dynamics for individual models and ensembles. We establish a conservation law, the vanishing of predictive entropy under successful learning, quantitative entropy and core-expansion bounds, and calibration and finite-sample validation results for identifying reliable predictions. These results motivate a two-generation ensemble that uses Jensen-Shannon (JS) disagreement to route uncertain inputs to specialized learners. CIFAR-10 experiments examine controlled perturbations, distribution shifts, and adversarial attacks. On a heterogeneous test environment, routing improves accuracy over probability averaging while keeping core accuracy near 95%. For adversarial defense, we consider a query-only black-box setting: the attacker can query the overall prediction function but does not know a particular internal decomposition into learners and routing rules. Our current attacks use a fixed-initial-route gradient, which is accurate for a single-step attack when the route remains unchanged, but may deviate from actual multi-step attacks, especially for large perturbations. We evaluate our two-layer defense against this approximation; whether additional adversarially trained layers improve resistance to stronger multi-step attacks remains for future study.

Figures & tables

Explore similar work

CardsList
  1. Reliable learning in challenging environments

    Apr 6, 2023Maria-Florina Balcan, Steve Hanneke, Rattana Pukdee +1Distribution Shift RobustnessAdversarial Robustness

  2. Adversarial Frontiers: Minimum-Norm Attack Ensembles for Robustness Evaluation

    Jul 22, 2026Luca Scionis, Luca Melis, Maura Pintor +5Adversarial AttacksNeural Network Robustness