Transferrable Surrogates in Expressive Neural Architecture Search Spaces

Shiwen Qin; Gabriela Kadlecová; Martin Pilát; Shay B. Cohen; Roman Neruda; Elliot J. Crowley; Jovita Lukasik; Linus Ericsson

cs.LGApr 17, 2025

Transferrable Surrogates in Expressive Neural Architecture Search Spaces

Authors: Shiwen Qin, Gabriela Kadlecová, Martin Pilát, Shay B. Cohen, Roman Neruda, Elliot J. Crowley, Jovita Lukasik, Linus Ericsson

Organizations: University of Edinburgh · Charles University, Faculty of Mathematics and Physics · The Czech Academy of Sciences, Institute of Computer Science · University of Siegen

Abstract

Neural architecture search (NAS) faces a challenge in balancing the exploration of expressive, broad search spaces that enable architectural innovation with the need for efficient evaluation of architectures to effectively search such spaces. We investigate surrogate model training for improving search in highly expressive NAS search spaces based on context-free grammars. We show that i) surrogate models trained either using zero-cost-proxy metrics and neural graph features (GRAF) or by fine-tuning an off-the-shelf LM have high predictive power for the performance of architectures both within and across datasets, ii) these surrogates can be used to filter out bad architectures when searching on novel datasets, thereby significantly speeding up search and achieving better final performances, and iii) the surrogates can be further used directly as the search objective for huge speed-ups.

Figures & tables

Figure 1: The encodings used by our language models (left) and our random forest and XGBoost models (right). A derivation tree is obtained from the grammar by sampling or mutation, and encoded into a string representation. The derivation tree can be compiled into a neural network, from which we can extract ZCP scores and GRAF encodings that are ultimately concatenated to form a descriptor for the RF and XGBoost models.

Figure 2: Search results using different surrogates. We plot the validation accuracies of the best models at each iteration of search, with the mean and standard error across three seeds.

Model	Input	Spearman	Kendall
RF	ZCP + GRAF	0.663	0.514
XGBRegressor	ZCP + GRAF	0.600	0.449
RoBERTa-base	String Encoding	0.723	0.575
RoBERTa-large	String Encoding	0.679	0.523
ModernBERT-base	String Encoding	0.746	0.597
ModernBERT-large	String Encoding	0.745	0.612

Table 1: Rank correlation between surrogate reward and ground truth accuracy on CIFAR10 architectures evaluation set for different surrogate models. Correlation averaged across three different seeds are given for few-shot learning surrogates with standard error. Correlation of ZCPs can be found in Table 7 .

	CIFAR10	AddNIST	Language	MultNIST	CIFARTile	Gutenberg	Isabella	GeoClassing	Chesseract	Avg
CIFAR10	0.612 / 0.648	0.470 / 0.516	0.408 / 0.326	0.604 / 0.447	0.470 / 0.374	0.577 / 0.525	0.189 / 0.239	0.581 / 0.284	0.280 / 0.424	0.466 / 0.420
CIFAR10(1k)	0.524 / 0.582	0.479 / 0.370	0.374 / 0.380	0.507 / 0.434	0.320 / 0.379	0.439 / 0.586	0.254 / 0.236	0.360 / 0.375	0.157 / 0.414	0.379 / 0.417
AddNIST	0.331 / 0.526	0.589 / 0.577	0.271 / 0.342	0.460 / 0.478	0.387 / 0.336	0.362 / 0.577	0.223 / 0.163	0.459 / 0.230	0.179 / 0.412	0.362 / 0.405
Language	0.370 / 0.486	0.443 / 0.229	0.388 / 0.200	0.463 / 0.128	0.393 / 0.126	0.467 / 0.133	0.268 / 0.268	0.262 / 0.219	0.271 / 0.430	0.369 / 0.246
MultNIST	0.371 / 0.497	0.369 / 0.280	0.264 / 0.401	0.394 / 0.445	0.285 / 0.307	0.423 / 0.500	0.229 / 0.241	0.245 / 0.307	0.300 / 0.335	0.320 / 0.368
CIFARTile	0.233 / 0.553	0.522 / 0.575	0.114 / 0.236	0.336 / 0.365	0.397 / 0.332	0.286 / 0.495	0.212 / -0.056	0.484 / 0.219	0.256 / 0.257	0.316 / 0.331

Table 2: Kendall-Tau correlation on the Unseen NAS datasets for surrogate ModernBERT-large and random forest . For CIFAR10 we use 7k architectures to train the surrogate on, while on the other datasets we use 1 000. For a fair comparison, one seed is randomly chosen from CIFAR10 training set to be comparable to other tasks.

	ModernBERT-large		Random Forest
Eval Tasks	Spearman	Kendall	Spearman	Kendall
AddNIST	0.773	0.625	0.671	0.500
Language	0.610	0.433	0.623	0.450
MultNIST	0.798	0.625	0.722	0.533
CIFARTile	0.629	0.444	0.558	0.374
Gutenberg	0.835	0.658	0.816	0.643

Table 3: Correlation on the Unseen NAS datasets for Leave-one-out surrogates with percentile normalisation.

	CIFAR10	AddNIST	Langauge	MultNIST	CIFARTile	Gutenberg	Isabella	GeoClassing	Chesseract
Evolution	0.624(0.096)	0.706 (0.160)	0.899 (0.020)	0.841 (0.080)	0.358 (0.002)	0.406(0.039)	0.449(0.038)	0.725 (0.030)	0.595(0.016)
Evolution(RF)	0.735 (0.025)	0.591(0.197)	0.881(0.006)	0.690(0.160)	0.331(0.004)	0.433 (0.009)	0.470 (0.013)	0.497(0.117)	0.601 (0.014)
Evolution(RF Transfer)	0.680(0.063)	0.656(0.229)	0.860(0.015)	0.877 (0.070)	0.336(0.012)	0.422(0.006)	0.451(0.029)	0.687(0.023)	0.595(0.012)
Evolution(BERT)	0.828 (0.006)	0.846 (0.088)	0.910(0.017)	0.765(0.019)	0.341 (0.010)	0.425 (0.005)	0.512 (0.040)	0.735 (0.024)	0.606 (0.008)
Evolution(BERT as obj)	0.656(0.046)	0.479(0.165)	0.857(0.013)	0.557(0.169)	0.290(0.007)	0.359(0.038)	0.440(0.029)	0.512(0.061)	0.597(0.028)

Table 4: Search results using regularised evolution on the Unseen NAS datasets. For each version we run on 3 random seeds and report the average best accuracy found along with the standard error of the mean. Refer to Appendix B.4 for detailed evolution settings.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: We apply four forms of augmentation to the architecture dataset to increase the effective data size for LM training. Three of our augmentations preserve the same exact network, while the last causes minor changes: (1) We swap the nesting order of sequential models; (2) We swap the two branches in a branching(2) module; (3) We add components that reduce to the identity operation; and (4) We change the output dimensionality to a neighbouring option in the list [16, 32, 64, 128, 256, 512, 1024, 2048] . For each augmented architecture, we adjust the corresponding accuracy by adding Gaussian noise from

\mathcal{N}(0,0.005^{2})

Encoding	Spearman	Kendall
Derivation tree str	0.621	0.448
Derivation tree str (+ shape)	0.745	0.612
PyTorch str	0.698	0.535

Appendix

Table 5: Correlation on CIFAR10 architectures evaluation set for different architecture encodings.

		BERT		Random Forest		XGB Regressor
		Spearman	Kendall	Spearman	Kendall	Spearman	Kendall
Dataset	Normalisation
AddNIST	minmax	0.768	0.617	0.599	0.433	0.541	0.397
CIFARTile	minmax	0.609	0.433	0.494	0.327	0.563	0.393
AddNIST	percentile	0.773	0.624	0.671	0.500	0.551	0.409
CIFARTile	percentile	0.629	0.444	0.558	0.374	0.557	0.388

Appendix

Table 6: Leave-one-out surrogate correlations for different normalization methods.

Figure 4: Extended runs for ModernBERT-large surrogates.

ZCP	Spearman	Kendall
grad_norm	0.040	0.036
snip	0.094	0.078
grasp	-0.072	-0.053
fisher	-0.037	-0.020
jacob_cov	0.381	0.243
plain	-0.189	-0.126

Appendix

Table 7: Rank correlation between ZCP reward and ground truth accuracy on CIFAR10 architectures evaluation set.

Model	Train	AddNIST	Langauge	MultNIST	CIFARTile	Gutenberg	Isabella	GeoClassing	Chesseract	Avg
BERT	✓	0.203	0.412	0.288	0.273	0.416	0.233	0.383	0.390	0.325
BERT + Cifar10	✓	0.386	0.557	0.510	0.380	0.553	0.423	0.581	0.541	0.491
BERT + Cifar10	✗	0.258	0.313	0.396	0.274	0.350	0.052	0.381	0.199	0.278
BERT + 8 Others	✓	0.394	0.467	0.513	0.398	0.558	0.391	0.582	0.494	0.475
BERT + 8 Others	✗	0.318	0.296	0.418	0.264	0.431	0.111	0.259	0.284	0.298
RF	✓	0.246	0.551	0.393	0.360	0.550	0.472	0.521	0.561	0.457

Appendix

Table 8: Continual training results with “✓” indicates the model was trained during evaluation; “✗” means it was not. We use BERT in short for ModernBERT-large and report average the correlation across each 100 iterations across 2 different seeds.

	ModernBERT-large		Random Forest		XGB Regressor
Eval Tasks	Spearman	Kendall	Spearman	Kendall	Spearman	Kendall
AddNIST	0.773	0.625	0.671	0.500	0.551	0.409
Language	0.610	0.433	0.623	0.450	0.622	0.456
MultNIST	0.798	0.625	0.722	0.533	0.749	0.561
CIFARTile	0.629	0.444	0.558	0.374	0.557	0.388
Gutenberg	0.835	0.658	0.816	0.643	0.839	0.669

Appendix

Table 9: Correlation on the Unseen NAS datasets for Leave-one-out surrogates with percentile normalisation.

	CIFAR10	AddNIST	Language	MultNIST	CIFARTile	Gutenberg	Isabella	GeoClassing	Chesseract	Avg
CIFAR10	0.639	0.466	0.349	0.459	0.381	0.599	0.281	0.174	0.405	0.417
CIFAR10(1k)	0.535	0.187	0.365	0.409	0.391	0.506	0.254	0.354	0.332	0.370
AddNIST	0.488	0.596	0.337	0.489	0.423	0.503	0.248	0.261	0.421	0.418
Language	0.443	0.125	0.277	0.216	0.087	0.205	0.255	0.306	0.386	0.255
MultNIST	0.495	0.330	0.366	0.416	0.301	0.450	0.183	0.153	0.304	0.333
CIFARTile	0.523	0.508	0.247	0.368	0.280	0.437	-0.082	0.105	0.222	0.290

Appendix

Table 10: Kendall-Tau correlation on the Unseen NAS datasets for the XGBoost regressor surrogate

Task	Model	Train	1	2	3	4	5	6	7	8	9	Avg
AddNIST	BERT	✓	0.11	0.41	0.16	0.02	0.46	0.13	0.06	0.21	0.09	0.18
	BERT + Cifar10	✓	0.66	0.62	0.42	0.37	0.66	0.36	0.27	0.38	0.28	0.45
	BERT + Cifar10	✗	0.54	0.60	0.31	0.27	0.44	0.12	-0.01	0.28	0.14	0.30
	BERT + 8 Others	✓	0.66	0.62	0.39	0.34	0.58	0.40	0.27	0.40	0.28	0.44
	BERT + 8 Others	✗	0.71	0.62	0.36	0.15	0.56	0.18	0.14	0.34	0.24	0.37
Chesseract	BERT	✓	0.41	0.33	0.26	0.38	0.45	0.65	0.38	0.40	0.51	0.42

Appendix

Table 11: Results of the ModernBERT-large models in simulated surrogate runs, seed=0

Task	Model	Train	100	200	300	400	500	600	700	800	900	Avg
AddNIST	BERT	✓	0.19	0.36	0.13	0.16	0.24	0.10	0.12	0.39	0.31	0.22
	BERT + Cifar10	✓	0.11	0.44	0.39	0.29	0.36	0.26	0.33	0.46	0.28	0.32
	BERT + Cifar10	✗	0.35	0.49	0.14	0.21	0.34	0.04	0.01	0.39	-0.02	0.22
	BERT + 8 Others	✓	0.45	0.53	0.29	0.30	0.33	0.26	0.23	0.47	0.29	0.35
	BERT + 8 Others	✗	0.60	0.51	0.20	0.17	0.30	0.07	0.11	0.38	0.09	0.27
Chesseract	BERT	✓	0.36	0.24	0.36	0.22	0.39	0.28	0.44	0.45	0.51	0.36

Appendix

Table 12: Results of the ModernBERT-large models in simulated surrogate runs, seed=1

Task	Model	Train	100	200	300	400	500	600	700	800	900	AVG
AddNIST	RF	✓	0.41	0.58	0.27	0.19	0.50	0.26	0.24	0.30	0.16	0.32
	RF + CIFAR10	✗	0.40	0.22	0.29	0.15	0.49	0.13	0.21	0.17	0.02	0.23
	RF + CIFAR10	✓	0.40	0.56	0.25	0.23	0.56	0.30	0.28	0.29	0.20	0.34
	RF + 8 Others	✗	0.45	0.19	0.28	0.15	0.39	0.21	0.10	0.04	0.01	0.20
	RF + 8 Others	✓	0.40	0.47	0.28	0.36	0.52	0.30	0.25	0.24	0.17	0.33
Chesseract	RF	✓	0.32	0.49	0.58	0.61	0.62	0.71	0.57	0.56	0.56	0.56

Appendix

Table 13: Results of the RF Regressor models in simulated surrogate runs, seed=0

Task	Model	Train	100	200	300	400	500	600	700	800	900	AVG
AddNIST	RF	✓	0.35	0.33	0.25	0.11	0.03	0.17	0.06	0.36	0.25	0.21
	RF + CIFAR10	✗	0.13	0.52	0.11	0.21	0.35	0.25	0.30	0.23	0.26	0.26
	RF + CIFAR10	✓	0.41	0.42	0.24	0.20	0.01	0.15	0.19	0.33	0.28	0.25
	RF + 8 Others	✗	0.19	0.16	-0.03	0.01	0.28	0.01	0.09	0.25	0.17	0.13
	RF + 8 Others	✓	0.20	0.07	0.29	0.22	0.09	0.21	0.12	0.29	0.27	0.19
Chesseract	RF	✓	0.50	0.48	0.46	0.53	0.48	0.52	0.60	0.64	0.66	0.54

Appendix

Table 14: Results of the RF Regressor models in simulated surrogate runs, seed=1

Task	Model	Train	100	200	300	400	500	600	700	800	900	AVG
AddNIST	XGB	✓	0.35	0.56	0.29	0.27	0.53	0.22	0.27	0.36	0.16	0.33
	XGB + CIFAR10	✗	0.40	0.22	0.28	0.10	0.39	0.10	0.18	0.07	0.01	0.19
	XGB + CIFAR10	✓	0.38	0.56	0.26	0.27	0.57	0.22	0.29	0.32	0.13	0.33
	XGB + 8 Others	✗	0.43	0.34	0.25	0.03	0.39	0.14	0.26	0.14	-0.00	0.22
	XGB + 8 Others	✓	0.45	0.53	0.30	0.41	0.59	0.25	0.27	0.27	0.16	0.36
Chesseract	XGB	✓	0.38	0.47	0.60	0.53	0.60	0.67	0.55	0.53	0.55	0.54

Appendix

Table 15: Results of the XGB Regressor models in simulated surrogate runs, seed=0

Task	Model	Train	100	200	300	400	500	600	700	800	900	AVG
AddNIST	XGB	✓	0.34	0.42	0.24	0.11	-0.02	0.26	0.23	0.37	0.26	0.24
	XGB + CIFAR10	✗	-0.13	0.50	0.11	0.22	0.35	0.26	0.32	0.26	0.24	0.24
	XGB + CIFAR10	✓	0.37	0.45	0.27	0.21	0.09	0.25	0.34	0.37	0.27	0.29
	XGB + 8 Others	✗	0.26	0.29	0.16	0.28	0.39	0.28	0.22	0.29	0.22	0.26
	XGB + 8 Others	✓	0.24	0.36	0.28	0.21	0.19	0.22	0.31	0.38	0.29	0.28
Chesseract	XGB	✓	0.52	0.51	0.50	0.50	0.48	0.52	0.60	0.60	0.65	0.54

Appendix

Table 16: Results of the XGB Regressor models in simulated surrogate runs, seed=1

RF
	limit	100	200	300	400	500	600	700	800	900
Remove zero acc.	Aggregation
False	minmax	0.39	0.42	0.44	0.50	0.46	0.44	0.47	0.54	0.49
	none	0.39	0.42	0.44	0.50	0.46	0.44	0.47	0.54	0.49
	percentile	0.37	0.42	0.44	0.51	0.46	0.45	0.46	0.54	0.51
True	minmax	0.39	0.41	0.43	0.49	0.45	0.43	0.45	0.54	0.51

Appendix

Table 17: Comparison of RF and XGBoost without transfer. Average Kendall tau over all Unseen datasets and seeds when trained on data from first iterations. Only the given dataset is used for training.

RF
	limit	100	200	300	400	500	600	700	800	900
Remove zero acc.	Aggregation
False	minmax	0.28	0.31	0.28	0.31	0.33	0.31	0.33	0.36	0.37
	none	0.31	0.30	0.27	0.32	0.33	0.31	0.31	0.35	0.35
	percentile	0.35	0.33	0.31	0.36	0.36	0.35	0.37	0.40	0.41
True	minmax	0.37	0.33	0.31	0.35	0.34	0.33	0.33	0.37	0.39

Appendix

Table 18: Comparison of RF and XGBoost with transfer. Average Kendall tau over all Unseen datasets and seeds when trained on data from first iterations. Data from the initial part of the current dataset and Cifar10, the other Unseen datasets, or both are used for training.

Neural Architecture Search

Neural Architecture Search automates neural network design for target tasks, resource budgets, and hardware constraints.

See all 136 papers

Towards Code-Oriented LM Embeddings for Surrogate-Assisted Neural Architecture Search
May 15, 2026Pranav Somu, Advay Balakrishnan, Stepan Kravtsov +2Neural Surrogate Modeling Representation Learning
Georgia Institute of Technology Atlanta, Georgia, USA · Georgia Tech Research Institute Atlanta, Georgia, USA
QuaNAS: Uncertainty-Aware Reuse of Earlier Searches for Query-Efficient Neural Architecture Search
Oct 22, 2026Brama Seta Putra Satria, Dwi Sunaryono, Misbakhul Munir Irfan Subakti +2Neural Architecture Search Uncertainty-Aware Exploration
Department of Informatics, Institut Teknologi Sepuluh Nopember, Surabaya, Indonesia · Department of Information Technology, Institut Teknologi Sepuluh Nopember, Surabaya, Indonesia · Department of Electrical Engineering, Institut Teknologi Sepuluh Nopember, Surabaya, Indonesia
ZAPS: Zero-Cost Active Proxy Search for Neural Architecture Search
Sep 12, 2026Hassan Touayouch, Rabie Najem, Mohammed BenjellounNeural Architecture Search Surrogate-Assisted Optimization
ENSICAEN · ISIA Lab, Université de Mons

May 15, 2026J/K move · Enter open · S save

Towards Code-Oriented LM Embeddings for Surrogate-Assisted Neural Architecture Search

Pranav Somu, Advay Balakrishnan, Stepan Kravtsov, Aaron McDaniel+1

Georgia Institute of Technology Atlanta, Georgia, USA · Georgia Tech Research Institute Atlanta, Georgia, USA

Neural Surrogate Modeling Representation Learning Neural Architecture Performance Prediction Neural Architecture Search

Open paper AskPDF arXiv

Developing effective surrogates (performance predictors) for Neural Architecture Search (NAS) typically requires expensive fine-tuning or the engineering of complex representations. We propose a low-cost embedding strategy that leverages the inductive bias of Language Models (LMs) to eliminate these overheads. By representing architectures as PyTorch class definition text, we demonstrate that off-the-shelf LMs act as competitive feature extractors without NAS-specialized fine-tuning. The final predictor is constructed by passing the extracted Code-Oriented LM Embeddings (COLE) through a lightweight regression head. We also investigate strategies to improve embedding quality and utilization. Our experiments on the NAS-Bench-201 and einspace search spaces reveal that raw code inputs yield higher predictive performance than other text-based encodings (e.g., ONNX-to-text encodings) when using frozen LMs. We also observe COLE drives superior surrogate-assisted search using the BANANAS algorithm in NAS-Bench-201. When optimizing for CIFAR-100 performance, replacing structural path encodings with COLE for architecture representation allows for a 34% decrease in the evaluation budget required to reach within 1% of the fittest architecture in the search space (by test accuracy). As any neural architecture can be represented as code, these findings establish COLE as a versatile and efficient foundation for advancing NAS.

Abstract

Neural Architecture Search

Explore similar work

Towards Code-Oriented LM Embeddings for Surrogate-Assisted Neural Architecture Search

QuaNAS: Uncertainty-Aware Reuse of Earlier Searches for Query-Efficient Neural Architecture Search

ZAPS: Zero-Cost Active Proxy Search for Neural Architecture Search