stat.MLNov 9, 2026

The balance between feature learning and collapse in dynamical systems-based generative models

Authors: Julian Brandon, Bruno Loureiro, N Alex Cayco-Gajic, Arthur Pellegrino

Organizations: Ecole Normale Supérieure, PSL

Abstract

Over the past decade, a growing body of work has established that implicit biases in learning dynamics fundamentally shape the solutions found by neural networks, governing their learned features and generalisation performance. However, despite these strides in deep learning theory, far less is known about representation learning in generative models based on dynamical systems, from normalising flows to diffusion. Importantly, empirical evidence suggests that continuous-time generative models can be prone to mode collapse, yet a learning theory explaining such biases remains elusive. To address this, we use tools from dynamical systems and operator theory to express the gradient flow of a general class of stochastic differential equations as a composition of linear operators, allowing us to relate collapse to low-rank biases in generative models. In variational inference and flow-matching tasks, we show that this bias can manifest in two different mechanisms: i) where collapsed models are stable fixed points of the gradient flow, or ii) in sequential feature learning, where smaller modes take progressively longer to be learned, producing an effective collapse for a fixed training budget. Together these findings point to a trade-off between feature learning and collapse, suggesting an operator view as a step towards a unified framework for understanding the learning dynamics of dynamical systems-based generative models.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Mitigating Model Collapse through Dreaming Learning

    Oct 21, 2026Matteo Benati, Alessandro Londei, Denise Lanzieri +2Model CollapseSelf-Training for Language Models

  2. Adynamical systems view of training generativemodels and the memorization phenomenon

    May 19, 2026Siva Athreya, Chiranjib Bhattacharya, Vivek S. BorkarNeural Network MemorizationMemorization in Generative Models

  3. A Markovian View of Iterative-Feedback Loops in Image Generative Models: Neural Resonance and Model Collapse

    Feb 22, 2026Vibhas Kumar Vats, David J. Crandall, Samuel GoreeMarkov ChainsModel Collapse