Temporal Task Diversity: Inductive Biases Under Non-Stationarity in Synthetic Sequence Modelling

Afiq Abdillah Effiezal Aswadi; Oliver Britton; Ross Baker; Matthew Farrugia-Roberts

cs.LGMay 18, 2026

Temporal Task Diversity: Inductive Biases Under Non-Stationarity in Synthetic Sequence Modelling

Authors: Afiq Abdillah Effiezal Aswadi, Oliver Britton, Ross Baker, Matthew Farrugia-Roberts

Organizations: 1Independent · University of Oxford

Abstract

Modern deep learning science often assumes that neural networks learn from a fixed data distribution. However, many practically important learning problems involve data distributions that change throughout training. How does such non-stationarity impact the inductive biases of deep learning towards models with different structural, generalisation, and safety properties? A fruitful testbed for studying inductive bias is in-context linear regression sequence modelling, where small transformers display strikingly different generalisation patterns depending on the diversity of the (fixed) training task distribution. In this paper, we explore the effect of diversifying the task distribution across training time, finding that such temporal diversity leads to an increased bias towards generalisation over memorisation.

Explore similar work

The Effect of Training Task Diversity on In-Context Learning through the Lens of Low-Dimensional Subspaces
Jun 5, 2026Soo Min Kwon, Alec S. Xu, Can Yaras +3In-Context Learning Unified Transformer
Continuity Laws for Sequential Models
May 8, 2026Annan Yu, Dongwei Lyu, N. Benjamin ErichsonSequence Modeling Efficient Continual Learning
Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
Jul 20, 2026Damien Teney, Liangze Jiang, Hemanth Saratchandran +1Transformer Architectures Inductive Bias

Jun 5, 2026 · stat.MLJ/K move · Enter open · S save

The Effect of Training Task Diversity on In-Context Learning through the Lens of Low-Dimensional Subspaces

Soo Min Kwon, Alec S. Xu, Can Yaras, Dogyoon Song+2

Department of Electrical and Computer Engineering, University of Michigan · Department of Statistics, University of California, Davis

The transformer's emergent ability to perform in-context learning (ICL) has sparked a wide range of studies designed to understand its underlying mechanisms. Existing works often study how training task diversity, defined either as the number of ICL training task vectors or as the number of function classes from which the task vectors are drawn, shapes both the learning dynamics and generalization capabilities of ICL. While both definitions have uncovered many interesting phenomena, many observations under the latter definition remain theoretically unexplained. This paper presents a minimal analytical model under which these phenomena provably emerge from the properties of the training data. By modeling the training task vectors as a mixture of low-rank Gaussians, we show how training task diversity, defined by the number of non-overlapping columns between subspaces that parameterize the covariance matrices, improves both the generalization and optimization trajectory of ICL with linear attention. In particular, we show that our model can explain (i) why training with task diversity shortens the ICL plateau and (ii) why ICL appears to achieve out-of-distribution generalization. We conclude by empirically demonstrating how our results extend to nonlinear transformers and nonlinear function classes. Overall, our work presents a tractable framework to unify existing observations.

In-Context Learning Unified Transformer

Open paper Ask about it PDF

Abstract

Explore similar work

The Effect of Training Task Diversity on In-Context Learning through the Lens of Low-Dimensional Subspaces

Continuity Laws for Sequential Models

Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks