Single-Index Models

Follow this topic Ask about this topic

A single-index model compresses an input into one direction, then uses a possibly nonlinear function to predict its outcome, offering a structured alternative to fitting an unrestricted model. Researchers study spectral and Stein-based estimators, stochastic gradient descent, neural networks, and kernel methods to recover the direction and learn the link, including when labels are noisy or data are collected adaptively. These methods are used in regression, contextual bandits, robust learning, and in-context learning, where the model must infer a new task from examples. A central tension is that direction and link recovery may be statistically possible but computationally difficult, especially in high dimensions or with non-monotone links; understanding when efficient algorithms reach optimal rates remains open.

Latest papers

16 papers2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners
Oct 1, 2026Haotian Gu, Yizhou Xu, Lenka ZdeborováRepresentation Learning Single-Index Models
Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · University of Chinese Academy of Sciences · Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)
Elicitation and Decision Geometry in Single-Index Bandits
Sep 28, 2026Sakshi Arya, Cheng Soon OngMulti-Armed Bandits Single-Index Models
Case Western Reserve University, USA · CSIRO, Australia
Low-Rank Single-Index Bandits with Unknown Links: From Matrices to Tensors
Sep 26, 2026Zhongxuan Liu, Yue Kang, Thomas C. M. LeeMulti-Armed Bandits Single-Index Models
Department of Statistics University of California, Davis Davis, CA 95616 · Microsoft Redmond, WA 98052
Active Regression for Single-Index Models with Unknown Link Functions
Aug 2, 2026Chansophea Wathanak In, Yi Li, Wai Ming Tai +1Single-Index Models Active Learning
School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore · Independent Researcher · John Hopcroft Center for Computer Science, Shanghai Jiao Tong University, China
The Multiscale Single-Index Model: A Stylized Model for Hierarchical Feature Learning
Jul 3, 2026Joan BrunaSingle-Index Models Hierarchical Representation Learning
Courant Institute School of Mathematics, Computing and Data Science, New York University
Can Neural Networks Achieve Optimal Computational-statistical Tradeoff? An Analysis on Single-Index Model
Jun 13, 2026Siyu Chen, Beining Wu, Miao Lu +2Representation Learning Single-Index Models
Department of Statistics and Data Science, Yale University · Department of Statistics, University of Chicago · Department of Management Science and Engineering, Stanford University +1
Deep Single-Index Fréchet Regression
Jun 5, 2026Muqing Cui, Yidong Zhou, Su I Iao +1Single-Index Models
Department of Statistics, University of California, Davis, CA, United States.
Convex Basins in Single-Index Model Loss Landscapes: Applications to Robust Recovery under Strong Adversarial Corruption
May 28, 2026Santanu Das, Sagnik Chatterjee, Jatin BatraSingle-Index Models Adversarial Robustness
School of Technology and Computer Science, Tata Institute of Fundamental Research, Mumbai, India.
How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysis
May 23, 2026Rei Higuchi, Ryotaro Kawata, Akifumi Wachi +3Reward Modeling Representation Learning
The University of Tokyo · RIKEN AIP · LY Corporation
Optimal Regret for Single Index Bandits
May 10, 2026Devdan Dey, Sujoy Bhore, Avishek GhoshMulti-Armed Bandits Single-Index Models
Department of Computer Science and Engineering, Indian Institute of Technology Bombay.
Kernel Single-Index Bandits: Estimation, Inference, and Learning
Mar 19, 2026Sakshi Arya, Satarupa Bhattacharjee, Bharath K. SriperumbudurContextual Bandits Single-Index Models
Department of Mathematics, Applied Mathematics, and Statistics, Case Western Reserve University · Department of Statistics, University of Florida · Department of Statistics, The Pennsylvania State University
A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Models
Mar 18, 2026Leonardo Defilippis, Florent Krzakala, Bruno Loureiro +1Statistical Learning Theory Multi-Index Models
Departement d’Informatique, École Normale Supérieure, PSL & CNRS · Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · INRIA Paris
Improved high-dimensional estimation with Langevin dynamics and stochastic weight averaging
Mar 6, 2026Stanley Wei, Alex Damian, Jason D. LeeLangevin Dynamics Single-Index Models
Princeton University · Harvard University · University of California, Berkeley
Interactive Learning of Single-Index Models via Stochastic Gradient Descent
Feb 19, 2026Nived Rajaraman, Yanjun HanSingle-Index Models Generalized Linear Bandits
Microsoft Research NYC · Courant Institute of Mathematical Sciences and Center for Data Science, New York University
Limitations of SGD for Multi-Index Models Beyond Statistical Queries
Feb 5, 2026Daniel Barzilai, Ohad ShamirSingle-Index Models Statistical Learning Theory
Weizmann Institute of Science · University of Toronto
Test time training enhances in-context learning of nonlinear functions
Sep 30, 2025Kento Kuwataka, Taiji SuzukiSingle-Index Models Test-Time Training
Department of Mathematical Engineering and Information Physics The University of Tokyo Hongo 7-3-1, Bunkyo-ku, Tokyo

Oct 1, 2026J/K move · Enter open · S save

In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners

Haotian Gu, Yizhou Xu, Lenka Zdeborová

Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · University of Chinese Academy of Sciences · Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)

Representation Learning Single-Index Models In-Context Learning Kernel Methods

Open paper AskPDF arXiv

In-context learning (ICL) enables a pretrained model to infer a task from demonstrations without updating its parameters. While much of the existing theory focuses on linear target functions, in this paper we study nonlinear cases by comparing two one-layer attention architectures on the same family of single-index tasks. A kernel learner first maps inputs through a fixed nonlinear feature map and then applies linear attention, whereas a feature learner applies attention to the original input, followed by a learned nonlinear readout. We derive predictions for their memorization and generalization errors using the replica method, retaining the effects of pretraining size, task-pool diversity, and training and inference context lengths. The resulting predictions closely match numerical experiments across a broad range of regimes. Our analysis yields phase diagrams that characterize when each architecture is advantageous as the amount of pretraining data, task diversity, and context lengths vary. We further identify qualitatively different context-length scalings for the two learners. Together, these results clarify how architectural choices interact with the dataset and govern nonlinear in-context learning.