Generative models trained on finite data face a fundamental tension: their score-matching or next-token objective converges to the empirical training distribution rather than the population distribution we seek to learn. Using rule-valid synthetic tasks, we trace this tension across two training timescales:
τrule, the step at which generations first become rule-valid, and
τmem, the step at which models begin reproducing training samples. Focusing on parity and extending to other binary rules and combinatorial puzzles, we characterize how these two clocks,
τrule and
τmem, depend on key aspects of the learning setup. Specifically, we show that
τrule increases with rule complexity and decreases with model capacity, while
τmem is approximately invariant to the rule and scales nearly linearly with dataset size
N. We define the \emph{innovation window} as the interval
[τrule,τmem]. This window widens with increasing
N and narrows with rule complexity, and may vanish entirely when
τrule≥τmem. The same two-clock structure arises in both diffusion (DiT) and autoregressive (GPT) models, with architecture-dependent offsets. Dissecting the learned score of DiT models reveals a corresponding evolution of the optimization landscapes, where rule-valid samples' basins expand substantially around
τrule, while training samples' basins begin to dominate around
τmem. Together, these results yield a unified and predictive account of when and how generative models exhibit genuine innovation.