Bayesian Optimization in Sequence-to-Architecture Latent Space for Zero-Shot NAS

Ondrej Tybl; Lukas Neumann

cs.CVOct 5, 2026

Bayesian Optimization in Sequence-to-Architecture Latent Space for Zero-Shot NAS

Authors: Ondrej Tybl, Lukas Neumann

Organizations: Department of Cybernetics FEE Czech Technical University

Abstract

Zero-shot Neural Architecture Search removes the prohibitive cost of traditional NAS, but its search process is typically based on the evolutionary algorithm (EA); lacking an explicit model of the objective, it often resorts to a near-random search through mutation. Bayesian Optimization offers a principled alternative by modeling the objective and aggregating information across iterations, but scales poorly to the high-dimensional, discrete, graph-structured spaces of modern NAS, restricting its use to only small networks. In this paper, we bring Bayesian Optimization to zero-shot NAS for large-scale architectures by learning a latent space via a Variational Autoencoder trained to reconstruct a novel prefix encoding of architectures and propose a proxy scalarization that combines several zero-shot proxies into a single Bayesian Optimization objective. After only 10,000 iterations of the proposed search algorithm (8 hours on a single GPU), our method found a network architecture which under the given model parameter count constraints achieves state-of-the-art results on three separate tasks -- image classification, object detection and semantic segmentation.

Figures & tables

Figure 1: We first train a variational autoencoder (VAE) to create a smooth, low-dimensional representation of large-scale heterogeneous architectures (left). Then, the latent space is used for Bayesian Optimization with zero-shot accuracy proxies aggregate as the objective function (right).

Model	Params	FLOPs	Top-1 (%)
ResNet He et al. (2016)	24.8M	9.0G	75.19
EfficientNet Tan and Le (2019)	24.0M	6.0G	80.52
ViT Dosovitskiy et al. (2020) ; Dai et al. (2021)	26.4M	18.4G	80.64
CoAtNet Dai et al. (2021)	26.9M	8.44G	80.58
ZenNAS Lin et al. (2021)	26.3M	18.6G	80.48
AZ-NAS Lee and Ham (2024)	26.7M	19.0G	80.39

Table 1: Top-1 image classification accuracy on ImageNet-1K ( Deng et al., 2009 ) . All models have a similar number of parameters and were trained using the UniNAS training protocol.

Method	Search strategy	Top-1 (%)	GPU days
$\beta$ -DARTS† Ye et al. (2022)	Differentiable (DARTS space)	76.1	0.4
GP-NAS† Li et al. (2020)	Bayesian Opt. w/ hand-crafted encoding	73.4	0.9
BONAS† Shi et al. (2020)	Bayesian Opt. w/ learned encoding	75.4	10.0
UniNAS-A Tybl and Neumann (2026)	Evolutionary Algorithm	81.15	0.5
ours	Bayesian Opt. w/ seq-to-arch latent enc.	81.51	0.33

Table 2: Comparison of zero-shot (training-free) NAS optimization strategies by best reported top-1 accuracy on ImageNet-1K ( Deng et al., 2009 ) . † denotes methods where optimal architecture was found on a smaller dataset (CIFAR) and then upscaled before training on ImageNet-1K.

		Object Detection and Seg.				Semantic Segmentation
Model	Params	AP b	AP m	FPS	FLOPs	mIoU	FPS	FLOPs
ResNet He et al. (2016)	24.8M	37.7	34.7	102.8	184G	39.3	481.2	47G
EfficientNet Tan and Le (2019)	24.0M	39.0	35.8	43.1	124G	37.0	152.6	32G
CoAtNet Dai et al. (2021)	26.9M	41.3	38.4	14.4	296G	42.4	61.7	51G
UniNAS-A Tybl and Neumann (2026)	26.8M	42.4	39.0	14.2	297G	45.6	88.2	51G
ours	27.0M	42.9	39.8	13.5	305G	46.1	62.4	51G

Table 3: Accuracy of object detection & instance segmentation on MS-COCO Lin et al. (2014) and semantic segmentation on ADE20K Zhou et al. (2017) .

\text{AP}^{\text{b}}

\text{AP}^{\text{m}}

and mIoU denote box and mask average precision and mean intersection over union, respectively.

Variant	Search	Latent space	Proxy	Top-1 (%)	$\Delta$
Ours (full)	BO	prefix-based	scalarized	81.5	–
w/o latent space	EA	none	scalarized	80.0	$-1.5$
w/o proxy scalarization	BO	prefix-based	VKDNW (single)	77.2	$-4.3$
w/o prefix encoding	BO	graph-based (VGAE)	scalarized	diverged †	–

Table 4: Ablation of the main components. We remove each component and replace it with the closest alternative. † BO did not improve the proxy, and the final architecture training diverged.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Network architecture reconstruction error (Levenshtein distance) of a variational autoencoder, comparing four encoder–decoder combinations where each component is either graph-based or prefix-sequence-based. The traditional VGAE as well as graph-based methods fail to converge, motivating our use of a prefix-sequence decoder.

Letter	Node	Arity	Status	Description
Leaf nodes (no children) — 13 retained from UniNAS
$b$	BatchNorm	–	used	batch normalisation
$j$	LayerNorm	–	used	layer normalisation
$c$	Conv1	–	used	$1{\times}1$ convolution
$d$	Conv3	–	used	$3{\times}3$ convolution
$u$	ConvDepth3	–	used	$3{\times}3$ depthwise convolution

Appendix

Table 5: Node vocabulary and its relation to the original set of UniNAS nodes Tybl and Neumann [2026] .

Explore similar work

Structuring Open-Ended NAS: Semi-Automated Design Knowledge Structuring with LLMs for Efficient Neural Architecture Search
May 19, 2026Yuiko Sakuma, Masakazu Yoshimura, Marcel Gröpl +4Neural Architecture Search Lattice Mutation
Sony Group Corporation, Tokyo, Japan · ETH Zurich, Switzerland
Agentic Neural Architecture Search
Jul 8, 2026Seokhoon Jeong, Mijung Kim, Taehwan KimNeural Architecture Search Agent2Agent
Artificial Intelligence Graduate School, Ulsan National Institute of Science and Engineering, Ulsan, Republic of Korea · Department of Computer Science and Engineering, Ulsan National Institute of Science and Engineering, Ulsan, Republic of Korea
LESS: Lightweight Evolutionary Supernet Search in Minutes
Oct 1, 2026Aviral Gandhi, Jinglue Xu, Jialong Li +1Neural Architecture Search Search Algorithms
The University of Tokyo · Sakana AI · Waseda University

May 19, 2026, cs.CVJ/K move · Enter open · S save

Structuring Open-Ended NAS: Semi-Automated Design Knowledge Structuring with LLMs for Efficient Neural Architecture Search

Yuiko Sakuma, Masakazu Yoshimura, Marcel Gröpl, Zitang Sun+3

Sony Group Corporation, Tokyo, Japan · ETH Zurich, Switzerland

Neural Architecture Search Lattice Mutation

Open paper AskPDF arXiv

Current neural architecture search (NAS) methods are often limited by their predefined, restrictive search spaces. While recent large language model (LLM)-assisted NAS methods enable open-ended search spaces, they often suffer from inefficient exploration due to biased or low-quality design ideas. To address these issues, we propose to semi-automatically structure model design knowledge to guide the search process. Our approach first defines a high-level structural template of architectural attributes. An LLM then populates this template by analyzing papers, creating a rich and diverse search space that embodies this structured design knowledge. To efficiently explore this vast space, we introduce FairNAD, using a multi-type mutation that enables broad exploration through mutation with fair idea sampling, Pareto-aware mutation, LLM-driven iterative mutation, and a fine-grained feedback loop. We demonstrate the effectiveness of FairNAD in discovering high-performing architectures that yield 0.84, 2.17, and 2.35 points improvement on CIFAR-10, CIFAR-100, and ImageNet16-120, respectively, compared to current state-of-the-art methods.

Abstract

Explore similar work

Structuring Open-Ended NAS: Semi-Automated Design Knowledge Structuring with LLMs for Efficient Neural Architecture Search

Agentic Neural Architecture Search

LESS: Lightweight Evolutionary Supernet Search in Minutes