cs.LGOct 14, 2025

Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learning

Authors: Junsoo Oh, Wei Huang, Taiji Suzuki

Organizations: KAIST AI

Abstract

Mamba, a recently proposed linear-time sequence model, has attracted significant attention for its computational efficiency and strong empirical performance. However, a rigorous theoretical understanding of its underlying mechanisms remains limited. In this work, we provide a theoretical analysis of Mamba's in-context learning (ICL) capability by focusing on tasks defined by low-dimensional nonlinear target functions. Specifically, we study in-context learning of a single-index model y≈g∗(⟨β,x⟩)y \approx g_*(\langle \boldsymbolβ, \boldsymbol{x} \rangle), which depends on only a single relevant direction β\boldsymbolβ, referred to as feature. We prove that Mamba, pretrained by gradient-based methods, can achieve efficient ICL via test-time feature learning, extracting the relevant direction directly from context examples. Consequently, we establish a test-time sample complexity that improves upon linear Transformers -- analyzed to behave like kernel methods -- and is comparable to nonlinear Transformers, which have been shown to surpass the Correlational Statistical Query (CSQ) lower bound and achieve near information-theoretically optimal rate in previous works. Our analysis reveals the crucial role of the nonlinear gating mechanism in Mamba for feature extraction, highlighting it as the fundamental driver behind Mamba's ability to achieve both computational efficiency and high performance.

Figures & tables

Explore similar work

CardsList
  1. How Can Mamba Learn In Context with Outliers and Generalize Provably?

    Oct 1, 2025Hongkang Li, Songtao Lu, Xiaodong Cui +2Selective SSMsIn-Context Learning

  2. From Markov to Laplace: How Mamba In-Context Learns Markov Chains

    Feb 14, 2025Marco Bondaschi, Nived Rajaraman, Xiuying Wei +5MambaSelective SSMs

  3. Test time training enhances in-context learning of nonlinear functions

    Sep 30, 2025Kento Kuwataka, Taiji SuzukiSingle-Index ModelsTest-Time Training