Implicit Bias in State Space Models and Linear Autoregressive Training
Organizations: Faculty of Computer Science, Hebrew University of Jerusalem, Israel · Faculty of Computer Science, Faculty of Computer and Electrical Engineering, Technion, Israel · Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Israel
Abstract
Linear State Space Models (SSMs) and linear autoregressive sequence models reuse the same parameters across a sample-dependent number of steps. As a result, different examples impose constraints through different predictors, even though all predictors share the same weights. We study this phenomenon through a general model of multiple non-homogeneous polynomial predictors with shared parameters, in a classification setting. Under standard interpolation and directional-convergence assumptions, together with a trajectory-dependent effective-degree condition, we show that gradient descent on the exponential loss converges in direction to a KKT point of an effective-degree max-margin problem. In this problem, only examples with minimal effective degree impose hard margin constraints; examples with faster-growing margins are asymptotically inactive except for feasibility. Our result provides a first-order characterization of the implicit bias in linear SSMs and linear autoregressive models. It exposes a mechanism of length bias: a longer input sequence or additional autoregressive steps may increase the example's effective degree, and the examples with the smallest effective degree dominate the limiting classifier. Our result highlights how non-homogeneity and parameter sharing alter classical homogeneous implicit-bias results.
Figures & tables
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Implicit Bias
Implicit bias studies how training algorithms and model structure favor particular solutions, representations, and generalization behaviors without explicit penalties.
See all 36 papers