cs.LGNov 6, 2026

Implicit Bias in State Space Models and Linear Autoregressive Training

Authors: Gal Katzhendler, Gilad Yehudai, Gal Vardi

Organizations: Faculty of Computer Science, Hebrew University of Jerusalem, Israel · Faculty of Computer Science, Faculty of Computer and Electrical Engineering, Technion, Israel · Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Israel

Abstract

Linear State Space Models (SSMs) and linear autoregressive sequence models reuse the same parameters across a sample-dependent number of steps. As a result, different examples impose constraints through different predictors, even though all predictors share the same weights. We study this phenomenon through a general model of multiple non-homogeneous polynomial predictors with shared parameters, in a classification setting. Under standard interpolation and directional-convergence assumptions, together with a trajectory-dependent effective-degree condition, we show that gradient descent on the exponential loss converges in direction to a KKT point of an effective-degree max-margin problem. In this problem, only examples with minimal effective degree impose hard margin constraints; examples with faster-growing margins are asymptotically inactive except for feasibility. Our result provides a first-order characterization of the implicit bias in linear SSMs and linear autoregressive models. It exposes a mechanism of length bias: a longer input sequence or additional autoregressive steps may increase the example's effective degree, and the examples with the smallest effective degree dominate the limiting classifier. Our result highlights how non-homogeneity and parameter sharing alter classical homogeneous implicit-bias results.

Figures & tables

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Minor First, Major Last: A Depth-Induced Implicit Bias of Sharpness-Aware Minimization

    Mar 9, 2026Chaewon Moon, Dongkuk Si, Chulhee YunSharpness-Aware MinimizationImplicit Bias

  2. Aligning Inductive Bias for Data-Efficient Generalization in State Space Models

    Sep 25, 2025Qiyu Chen, Guozhang ChenSequence ModelingData-Efficient Learning

  3. Implicit Bias of (Stochastic) Gradient Descent on Deep Linear Diagonal Networks

    Oct 29, 2026Wiebke Bartolomaeus, Holger RauhutImplicit BiasImplicit Regularization