Neural Architecture Search (NAS) automates network design, but evaluating a single candidate requires training it to convergence, making exhaustive search intractable. Zero-cost proxies estimate architecture quality at initialization in seconds, yet a single proxy is noisy, and combining several does not straightforwardly help: proxies are strongly correlated, so naive aggregation compounds their shared errors instead of averaging them out. Existing methods exploit either proxy signals or architectural topology - never both within a single active-learning framework. We introduce ZAPS (Zero-cost Active Proxy Search), a four-stage pipeline that closes this gap. ZAPS (i) selects a compact, non-redundant proxy subset offline via ProxyFit, a greedy anti-redundancy criterion; (ii) seeds the search with a hybrid K-means strategy that balances exploitation and exploration; (iii) re-selects proxies at every iteration by a bootstrapped vote as the labeled set grows; and (iv) ranks candidates with an XGBoost ensemble trained jointly on proxy ranks and one-hot topological encodings, queried through an Upper Confidence Bound (UCB) acquisition function. On NAS-Bench-201 under a budget of B=200 evaluations, ZAPS recovers 52.3% of the true top-100 architectures on CIFAR-10 and 65.8% on CIFAR-100, ahead of every baseline we consider - Random Search, Local Search, REA, BANANAS and TPE - and, on CIFAR-10, with less than half the run-to-run standard deviation of the strongest of them. The advantage is largest where evaluations are scarce: on NAS-Bench-201 it narrows as the budget grows, whereas on the harder NAS-Bench-101, which no method comes close to saturating, it widens instead. All methods are scored by a single criterion: how much of the true top-100 lies among the architectures they actually evaluated.
Figures & tables
Figure 1: The four stages of ZAPS . ProxyFit runs once offline and returns a proxy subset; the hybrid K-means seeds the labeled set; re-selection, surrogate fitting and UCB querying then alternate for T iterations, each one spending part of the budget on newly evaluated architectures.
Figure 2: Absolute Spearman correlation between proxy pairs on NAS-Bench-201 / CIFAR-10, with proxies ordered by decreasing relevance—the order in which ProxyFit examines them. Redundancy is assessed on ∣ρ∣ , since two strongly anti-correlated proxies are as redundant as two strongly correlated ones. Highly correlated pairs such as flops – params ( ∣ρ∣=0.996 ) are the redundancies ProxyFit is designed to eliminate.
Figure 3: P@100 vs. budget B on NAS-Bench-201 (top: CIFAR-10, bottom: CIFAR-100); means over 200 seeds.
Figure 4: Regret vs. budget B , same setting; geometric means over 200 seeds, floored at 10−5 .
Figure 5: Convergence of ZAPS alone: P@100 vs. budget B on CIFAR-10 and CIFAR-100; means and interquartile bands over 200 seeds.
Figure 6: Regret of ZAPS alone, same setting; geometric means over 200 seeds, floored at 10−5 .
Figure 7: Spearman correlation between the surrogate ranking and the true ranking of all 15,625 architectures of NAS-Bench-201, on CIFAR-10 and CIFAR-100; means and interquartile bands over 200 seeds.
Figure 8: Ablation of ProxyFit across the full budget range on CIFAR-10 and CIFAR-100: the ProxyFit subset of six proxies against the full pool of thirteen. Lines are means over 200 seeds and shaded bands interquartile ranges. Table 6 reports the values at B=200 .
Figure 9: Ablation of the initialization across the full budget range on CIFAR-10 and CIFAR-100: hybrid K-means initialization against uniform random sampling. Lines are means over 200 seeds and shaded bands interquartile ranges. Table 6 reports the values at B=200 .
Surrogate input
Dim.
CIFAR-10
CIFAR-100
Proxy ranks + topology
37 – 40
52.3±6.1
65.8±11.0
proxy ranks only
7 – 10
47.8±8.4
58.7±14.0
topology only
30
31.1±8.0
43.4±11.0
Table 7: Fusion ablation on NAS-Bench-201 at B=200 , 200 seeds. Only the surrogate’s input changes; ProxyFit , initialization, re-selection and UCB acquisition are identical across rows. The proxy block is the online re-selection’s output, whose size grows with the labeled set: 7 proxies at the first fit, 10 thereafter.
Figure 10: Fusion ablation, 200 seeds; only the surrogate’s input differs. Topology-only never closes its gap; proxies-only converges to the full pipeline. Regret on a linear scale, since ZAPS reaches exactly zero.
Figure 11: P@100 (top) and regret (bottom) on NAS-Bench-101 / CIFAR-10; means over 200 seeds, with interquartile bands for P@100 .
CIFAR-100
ImageNet16-120
B
native
transferred
native
transferred
100
24.2
32.7
16.7
14.8
200
65.8
62.0
26.4
27.5
300
76.1
76.5
44.9
48.9
500
88.1
87.1
68.1
70.7
Table 9: P@100 (%) when ProxyFit is calibrated on the target dataset ( native ) versus transferred unchanged from CIFAR-10 ( transferred ). Mean over 200 seeds. A transferred subset uses no ground-truth accuracy from the target dataset.
Figure 12: Cross-dataset transfer: the subset calibrated on CIFAR-10, applied unchanged to CIFAR-100 and ImageNet16-120 ( transferred ), against one calibrated on the target itself ( native ). Values in Table 9 .
Figure 13: Cross-space transfer: the subset calibrated on NAS-Bench-201, applied unchanged to NAS-Bench-101 ( transferred ), against one calibrated on NAS-Bench-101 ( oracle ). They differ by at most 4.2 pp in P@100 and 0.07 pp in regret.
Evaluating candidate architectures in neural architecture search (NAS) faces an inherent trade-off: on the one hand, reliable performance labels are limited because training and evaluating architectures is costly; on the other hand, zero-cost proxies (ZCPs) are cheap to compute at large scale but can be noisy. Yet, how to effectively combine these two sources of supervision remains unclear. In this paper, we propose PPNAS, a novel prediction-powered inference (PPI) approach for NAS. PPNAS fuses (1) a small set of architectures with observed performance labels and (2) a large set of architectures with ZCP information. To combine these two sources of supervision, PPNAS exploits the ordinal information provided by ZCPs to construct additional pairwise ranking supervision, while PPI debiases systematic discrepancies between ZCP-based and true performance rankings. We evaluate PPNAS in end-to-end predictor-based NAS, where it achieves state-of-the-art under limited evaluation budgets. To the best of our knowledge, PPNAS is the first prediction-powered approach for label-efficient NAS.
Pascal Janetzky, Yuxin Wang, Michael Klar +1
Bosch Center for Artificial Intelligence · LMU Munich Munich Center for Machine Learning(MCML)
Zero-cost proxies enable neural architecture search (NAS) to rank candidate networks from statistics computed at initialization, avoiding repeated training. However, different proxies capture different properties and often produce inconsistent rankings across search spaces. Ensemble proxies can combine complementary signals, but automated discovery must optimize both discrete aggregation structures and their continuous coefficients, making structural quality difficult to separate from parameter calibration. We propose Bi-EZP, a bilevel framework that decouples these decisions. At the upper level, a large language model generates executable aggregation programs over four complementary base proxies with program-specific parameter bounds. At the lower level, covariance matrix adaptation evolution strategy (CMA-ES) optimizes the continuous parameters of each fixed program on an inner training split. The calibrated programs are then evaluated using Kendall's rank correlation on a disjoint validation split, enabling evolutionary selection to favor structures that generalize beyond their calibration data. Experiments on NATS-Bench and Network Design Spaces evaluate ranking performance across heterogeneous search spaces, and DARTS experiments assess downstream architecture search. Results show that separating program discovery from numerical calibration provides an effective approach to automated ensemble zero-cost proxy construction. The source code is available at: https://anonymous.4open.science/r/Bi-EZP-318D
Yutao Lai, Kezhao Lai, Hai-Lin Liu
School of Mathematics and Statistics, Guangdong University of Technology, Guangzhou 510006, China
Efficient neural architecture search (NAS) requires accurate performance estimation under limited evaluation budgets. Existing zero-cost proxies (ZCPs) provide fast training-free signals, but each proxy usually captures only a partial property of an architecture, leading to unstable rankings across datasets and search spaces. In contrast, GNN-based predictors can improve accuracy, but often depend on labeled architectures, graph encoders, and costly representation learning, which makes them less suitable for extreme few-shot settings. We propose FENAS, a topology-agnostic few-shot prediction framework that turns complementary initialization priors into robust ranking signals. Its core component, CAST, is a feature pipeline that combines copula alignment and synergy transformation. CAST maps heterogeneous ZCPs into a compact feature manifold, enabling lightweight meta-learners to produce reliable rankings from scarce supervision. On NAS-Bench-201, with only 0.5% labeled samples, FENAS achieves a Kendall's Tau (τ) of 0.70, outperforming representative topology-based few-shot predictors in this low-label regime. In closed-domain search, FENAS evaluates the NAS-Bench-201 space in approximately 1138 seconds and achieves competitive test accuracy. By providing a fast, reliable, and structurally simple evaluation mechanism, FENAS supports the design of automated neural architectures under limited evaluation budgets.