cs.LGNov 16, 2026

Searching for Autoregressive Neural Architectures from Fundamental Operations

Authors: Thomas Woehrle, Linus Ericsson, Henry Gouk, Elliot J. Crowley

Organizations: University of Edinburgh · University of Glasgow

Abstract

Expressive Neural Architecture Search (NAS) search spaces are desirable as they do not restrict the space of potential architectures to what is already established. However, few search spaces are expressive enough to represent a broad range of architecture families, and those that are have focused primarily on image classification problems. The task of autoregressive sequence modelling has become ubiquitous in modern machine learning, but no existing search space supports the construction of and search over sequence models from fundamental operations, instead imposing strong inductive biases on the structure of potential architectures. This work describes how the expressive einspace search space can be extended to yield a search space capable of representing autoregressive models using fundamental operations. We perform a series of experiments showing that the novel einspaceAR search space can find capable autoregressive sequence models from scratch--i.e., without imposing strong inductive biases. We believe that our einspaceAR search space will be a useful part of the NAS toolbox and that the techniques described in this work can serve as a blueprint for the integration of autoregressive modelling in other expressive search spaces.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Transferrable Surrogates in Expressive Neural Architecture Search Spaces

    Apr 17, 2025Shiwen Qin, Gabriela Kadlecová, Martin Pilát +5Neural Architecture SearchNeural Surrogate Modeling

  2. Evolutionary Architecture Search through Grammar-Based Sequence Alignment

    Dec 4, 2025Adri Gómez Martín, Felix Möller, Steven McDonagh +5Evolutionary NASNeural Architecture Search

  3. AgenticRS-EnsNAS: Ensemble-Decoupled Self-Evolving Architecture Search

    Mar 20, 2026Yun Chen, Moyu Zhang, Jinxin Hu +2Neural Architecture SearchEnsemble Learning