Organizations: Ludwig-Maximilians-Universität München (LMU Munich), Germany · Munich Center for Machine Learning (MCML), Germany · Paderborn University, Germany · Faculty of Computer Science, University of Duisburg-Essen, Germany
Neural Architecture Search (NAS) has advanced the automated design of deep networks, but remains computationally expensive due to reliance on model training or performance predictors. We introduce DIFFNAS, a predictor-free NAS framework that leverages a discrete graph diffusion model guided by zero-cost performance proxies. Unlike prior diffusion-based NAS methods that require learned performance predictors, DIFFNAS conditions the generative process on training-free metrics such as MeCo and SWAP, enabling efficient architecture search with minimal computational overhead. Experiments on NAS-Bench-201 show that DIFFNAS finds the optimal architectures for CIFAR-10 and CIFAR-100 (94.37% and 73.51%) without a pretrained predictor, while reducing search cost by up to 17.5x compared to DiffusionNAG. On the MobileNetV3 search space, its accuracy is lower than that of DiffusionNAG. These results highlight the promise of diffusion models combined with zero-cost proxies for scalable and effective architecture search.
Figures & tables
Figure 1: Illustration of diffusion framework and search pipeline for DIFFNAS.
Target Dataset
NAS Method
Params
Search Time
Speed Up
Search Cost
Accuracy
(M)
(GPU h)
($)
(%)
CIFAR-10
NASWOT
0.615
0.09
22.5×
0.22
92.96
TE-NAS
1.317
0.43
107.5×
1.07
93.78
TG-NAS
0.802
0.01
2.5×
0.02
93.75
DiffusionNAG
-
0.07
17.5×
0.18
94.37
PC-DARTS
1.17
2.89
722.5×
7.22
93.66
Table 2: Comparison of NAS methods on NAS-Bench-201 across CIFAR-10, CIFAR-100, Aircraft, and Oxford-IIIT Pets. Reported metrics include model size (in millions of parameters), search time (GPU hours), monetary cost (in USD), and accuracy (%). “–” denotes missing or unreported values.
Target Dataset
Stats.
Methods
Oracle
Random
MetaD2A
Diff.
Ours
CIFAR10
Max
94.37
94.37
94.37
94.37
94.37
Mean
93.50
87.12
91.52
93.13
93.41
Min
93.18
10.00
10.00
86.44
92.08
CIFAR100
Max
73.51
72.74
73.51
73.51
73.51
Mean
70.62
61.59
67.14
70.34
69.91
Table 3: Performance comparison on CIFAR10 and CIFAR100 datasets in NB201 search space. 1000 architectures are generated for computing statistical metrics for each method.
Target Dataset
Stats.
Methods
Oracle Top-1,000
w/o MeCo
w/o scaling
SWAP
Full Frame
CIFAR10
Max
94.37
94.36
94.36
94.37
94.37
Mean
93.50
87.70
92.33
86.57
93.41
Min
93.18
10.00
86.64
10.00
92.08
Table 4: Summary of ablation studies. 1000 architectures are generated for computing statistical metrics for each method.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Algorithm
Acc. (CIFAR-10)
DiffusionNAG
97.39
DIFFNAS (Ours)
89.50
Appendix
Table 5: Comparison of neural architecture search methods evaluated on the MobileNetV3 search space on CIFAR-10.
Asset
Version
License
Source URL
NAS-Bench-201
v2.0
MIT
https://github.com/D-X-Y/NAS-Bench-201
MobileNetV3
0.1 1 1 1 According to Cai et al. [2020] version
Apache-2.0
https://github.com/mit-han-lab/once-for-all
MeCo
0d830dd 2 2 2 The latest github commit number
MIT
https://github.com/HamsterMimi/MeCo
SWAP
0853fc8 3 3 3 The latest github commit number
AFL-3.0 license
https://github.com/pym1024/SWAP
PyTorch Lightning
2.5.1
Apache-2.0
https://github.com/Lightning-AI/pytorch-lightning
Appendix
Table 6: External assets, versions, licenses, and source URLs.
Neural Architecture Search (NAS) aims to automate neural network architecture design, reducing reliance on human expertise. Among the various NAS methods, differentiable NAS has gained prominence due to its efficiency and accuracy compared to conventional NAS approaches. Since differentiable NAS relaxes the architecture search space into a continuous domain, it is possible to apply principles from continuous optimization to NAS. In this paper, we propose Linear Programming-based NAS (LP-NAS), a mathematical programming-based framework for differentiable NAS that is applicable to a wide range of continuous search spaces. LP-NAS formulates a linear program (LP) using the validation-loss gradient and the training-loss Hessian to compute an architecture update direction that improves generalization while preserving the optimality of the model parameters. By following this LP-derived descent direction, LP-NAS efficiently navigates the architecture search space, leading to faster and more effective architecture optimization. We introduce two computationally efficient variants of LP-NAS, namely S-LP-NAS and R-LP-NAS. Applying LP-NAS to the Differentiable Architecture Search (DARTS) search space results in two algorithmic variants, S-LP-DARTS and R-LP-DARTS. Both variants achieve faster convergence and significantly higher validation performance during the early search iterations than the standard DARTS algorithm. Extensive experiments on CIFAR-10 and CIFAR-100 show that LP-DARTS outperforms standard DARTS in both the architecture search and evaluation phases. Additionally, we compare our approach with several DARTS variants (P-DARTS, PC-DARTS, and STO-DARTS) on the CIFAR-10 dataset and demonstrate its effectiveness. Furthermore, we validate the transferability of the discovered architectures through experiments on the ImageNet dataset.
Abhishek Shukla, Ankur Sinha, Faiz Hamid
Department of Management Sciences, IIT Kanpur, India · Krishnamurthy Tandon School of AI, IIM Ahmedabad, India
Evaluating candidate architectures in neural architecture search (NAS) faces an inherent trade-off: on the one hand, reliable performance labels are limited because training and evaluating architectures is costly; on the other hand, zero-cost proxies (ZCPs) are cheap to compute at large scale but can be noisy. Yet, how to effectively combine these two sources of supervision remains unclear. In this paper, we propose PPNAS, a novel prediction-powered inference (PPI) approach for NAS. PPNAS fuses (1) a small set of architectures with observed performance labels and (2) a large set of architectures with ZCP information. To combine these two sources of supervision, PPNAS exploits the ordinal information provided by ZCPs to construct additional pairwise ranking supervision, while PPI debiases systematic discrepancies between ZCP-based and true performance rankings. We evaluate PPNAS in end-to-end predictor-based NAS, where it achieves state-of-the-art under limited evaluation budgets. To the best of our knowledge, PPNAS is the first prediction-powered approach for label-efficient NAS.
Pascal Janetzky, Yuxin Wang, Michael Klar +1
Bosch Center for Artificial Intelligence · LMU Munich Munich Center for Machine Learning(MCML)
Neural Architecture Search (NAS) has become a pivotal technique in automated machine learning. Evolutionary Algorithm (EA)-based methods demonstrate superior search quality but suffer from prohibitive computational costs, while gradient-based approaches like DARTS offer high efficiency but are prone to premature convergence and performance collapse. To bridge this gap, we propose G-ICSO-NAS, a hybrid framework implementing a three-stage optimization strategy. The Warm-up Phase pre-trains supernet weights (w) via differentiable methods while architecture parameters (α) remain frozen. The Exploration Phase adopts a hybrid co-optimization mechanism: an Improved Competitive Swarm Optimizer (ICSO) with diversity-aware fitness navigates the architecture space to update α, while gradient descent concurrently updates w. The Stability Phase employs fine-grained gradient-based search with early stopping to converge to the optimal architecture. By synergizing ICSO's global navigation capability with differentiable methods' efficiency, G-ICSO-NAS achieves remarkable performance with minimal cost. In the context of the DARTS search space, an accuracy of 97.46% is achieved on CIFAR-10 with a computational budget of just 0.15 GPU-Days. The method also exhibits strong transfer potential, recording accuracies of 83.1% (CIFAR-100) and 75.02% (ImageNet). Furthermore, regarding the NAS-Bench-201 benchmark, G-ICSO-NAS is shown to deliver state-of-the-art results across all evaluated datasets.
Xingbang Du, Enzhi Zhang, Rui Zhong +2
Information Systems Design Laboratory, Hokkaido University, Sapporo, Japan · Information Initiative Center, Hokkaido University, Sapporo, Japan