Incremental Learning in Mirror Flows

Raphaël Berthier; Loucas Pillaud-Vivien

math.OCJun 22, 2026

Incremental Learning in Mirror Flows

Authors: Raphaël Berthier, Loucas Pillaud-Vivien

Organizations: Sorbonne Université, Inria, Centre Inria de Sorbonne Université, Paris, France · CERMICS, CNRS, ENPC, Institut Polytechnique de Paris, Marne-la-Vallée, France

Abstract

We study mirror flows generated by a convex quadratic loss and a general convex lower semicontinuous mirror potential. We show that, when initialized near the boundary of the domain of the mirror potential, their rescaled trajectories converge to a limiting mirror flow whose potential is the indicator function of the domain. In this limit, the primal variable minimizes the loss over a time-dependent hypothesis set: the subdifferential of the support function of the domain, evaluated at the dual variable. This characterization provides a general mechanism for incremental learning in mirror flows.

Explore similar work

Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning
May 19, 2026Tom Jacobs, Guido MontufarFeature Learning Gradient Flow
Remembering by Reconstructing: Domain Incremental Learning With Test-Time Training on Video Streams
May 29, 2026Jonathan Swinnen, Tinne TuytelaarsDomain Incremental Learning Domain Adaptation
Establishing Boundary KKT Convergence of Mirror Descent through Reparameterization
Aug 7, 2026Kuangyu Ding, Kim-Chuan TohMirror Descent Convergence

May 19, 2026 · cs.LGJ/K move · Enter open · S save

Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning

Tom Jacobs, Guido Montufar

We study the max-margin solutions reached by mirror flow in deep neural networks with homogeneous activation functions. Extending classical results on gradient flow, we derive a novel balance equation for mirror flow from convex duality, enabling a characterization of the horizon function governing the induced margin. We further establish max-margin characterizations together with convergence rates and norm growth estimates. Finally, we support our theory through experiments on synthetic datasets and standard vision tasks. Concretely, we show that: (1) distinct non-homogeneous mirror maps can induce the same max-margin solution; (2) convergence can be extremely slow, including exponentially slow regimes; and (3) although all considered mirror maps exhibit feature learning, they can produce markedly different representations, ranging from sparse to dense neuron activations. Together, these results provide a unified perspective on sparse and dense feature learning in homogeneous neural networks, highlighting how mirror maps shape both optimization dynamics and the geometry of the learned classifiers.

Feature Learning Gradient Flow Representation Learning Maximization Implicit Bias

Open paper Ask about it PDF

Abstract

Explore similar work

Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning

Remembering by Reconstructing: Domain Incremental Learning With Test-Time Training on Video Streams

Establishing Boundary KKT Convergence of Mirror Descent through Reparameterization