Neural Architecture Search (NAS) aims to automate neural network architecture design, reducing reliance on human expertise. Among the various NAS methods, differentiable NAS has gained prominence due to its efficiency and accuracy compared to conventional NAS approaches. Since differentiable NAS relaxes the architecture search space into a continuous domain, it is possible to apply principles from continuous optimization to NAS. In this paper, we propose Linear Programming-based NAS (LP-NAS), a mathematical programming-based framework for differentiable NAS that is applicable to a wide range of continuous search spaces. LP-NAS formulates a linear program (LP) using the validation-loss gradient and the training-loss Hessian to compute an architecture update direction that improves generalization while preserving the optimality of the model parameters. By following this LP-derived descent direction, LP-NAS efficiently navigates the architecture search space, leading to faster and more effective architecture optimization. We introduce two computationally efficient variants of LP-NAS, namely S-LP-NAS and R-LP-NAS. Applying LP-NAS to the Differentiable Architecture Search (DARTS) search space results in two algorithmic variants, S-LP-DARTS and R-LP-DARTS. Both variants achieve faster convergence and significantly higher validation performance during the early search iterations than the standard DARTS algorithm. Extensive experiments on CIFAR-10 and CIFAR-100 show that LP-DARTS outperforms standard DARTS in both the architecture search and evaluation phases. Additionally, we compare our approach with several DARTS variants (P-DARTS, PC-DARTS, and STO-DARTS) on the CIFAR-10 dataset and demonstrate its effectiveness. Furthermore, we validate the transferability of the discovered architectures through experiments on the ImageNet dataset.
Figures & tables
Figure 1: Optimization paths of a first-order method and our proposed approach.
(Ak+1,Vk+1)
←(Ak,Wk)+t(dAk∗,dWk∗)
Algorithm 1 LP-NAS
NAS Alg.
CIFAR-10
CIFAR-100
Val. Acc. (%)
Search Cost (CPU hours)
Val. Acc. (%)
Search Cost (CPU hours)
DARTS
81.93±0.31
26.30±0.56
49.69±0.86
26.12±0.40
S-LP-DARTS
84.94±0.31
25.30±0.22
55.51±0.39
25.84±0.66
R-LP-DARTS
84.96±0.47
26.11±0.42
55.78±0.92
25.40±0.23
Table 1: Performance metrics of NAS methods on CIFAR-10 and CIFAR-100.
Figure 2: Architecture search performance plots of the algorithms for three independent runs on the CIFAR-10 and CIFAR-100 datasets.
NAS Alg.
CIFAR-10
CIFAR-100
Test Acc. (%)
Eval. Cost (CPU hours)
Params (M)
Test Acc. (%)
Eval. Cost (CPU hours)
Params (M)
DARTS
70.83±2.09
11.55±0.48
3.40±0.10
23.84±9.48
11.85±0.44
3.64±0.12
S-LP-DARTS
89.04±1.26
11.43±0.24
1.41±0.22
63.94±3.61
11.78±0.18
1.59±0.23
R-LP-DARTS
89.87±0.82
12.08±0.18
1.66±0.28
58.92±4.56
12.02±0.30
1.75±0.41
Table 2: Evaluation metrics for the architectures on CIFAR-10 and CIFAR-100.
Figure 3: (a) Matrix plots illustrating architecture selection for the CIFAR-10 and CIFAR-100 datasets. (b) Training and validation performance of the best architectures on the CIFAR-10 and CIFAR-100 datasets.
Method
CIFAR-10
CIFAR-100
Test Acc. (%)
Eval. Cost (CPU hours)
Params (M)
Test Acc. (%)
Eval. Cost (CPU hours)
Params (M)
DARTS
82.57
19.66
3.38
49.86
19.85
3.50
S-LP-DARTS
92.56
19.91
1.66
68.50
20.04
1.33
R-LP-DARTS
92.55
19.72
1.36
65.58
19.59
1.36
Table 3: Evaluation metrics for the best architectures on CIFAR-10 and CIFAR-100.
Method
CIFAR-10
ImageNet
Test Acc. (%)
Eval. Cost (CPU hours)
Params (M)
Top-1 (5) Acc. (%)
Eval. Cost (CPU hours)
Params (M)
DARTS
82.57
19.66
3.38
39.09 (64.10)
99.36
5.81
P-DARTS
90.23
21.67
2.74
44.60 (69.38)
98.96
4.94
PC-DARTS
89.10
20.84
2.90
42.40 (67.39)
101.53
5.27
STO-DARTSv1
89.04
19.79
2.20
42.72 (67.78)
101.52
4.15
STO-DARTSv2
87.89
20.85
3.06
42.89 (68.05)
102.32
5.37
Table 4: Evaluation metrics for the best architectures on CIFAR-10 and their transferability on ImageNet under time-constrained settings.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9
Method
Search Type
Search Space
Dataset
Top-1 Acc. (%)
Params (M)
GPU Days
NASNet-A [ 29 ]
RL-based
Cell-based
CIFAR-10 (ImageNet)
97.35 (74.0)
3.3 (5.3)
2000
ENAS [ 30 ]
RL (Weight Sharing)
Cell-based
CIFAR-10
97.11
4.6
0.5
AmoebaNet-A [ 31 ]
Evolutionary
Cell-based
CIFAR-10 (ImageNet)
96.66 (74.5)
3.2 (5.1)
3150
DARTS [ 17 ]
Differentiable
Continuous
CIFAR-10 (ImageNet)
97.24 (73.3)
3.3 (4.7)
4
PC-DARTS [ 27 ]
Differentiable
Continuous
CIFAR-10 (ImageNet)
97.43 (74.9)
3.6 (5.3)
0.1
ProxylessNAS [ 32 ]
Differentiable
Hardware-aware
CIFAR-10 (ImageNet)
97.92 (75.1)
5.7 (7.1)
4
Appendix
Table 5: Comparative performance of representative NAS methods on CIFAR-10 and ImageNet benchmarks.
Neural Architecture Search (NAS) has become a pivotal technique in automated machine learning. Evolutionary Algorithm (EA)-based methods demonstrate superior search quality but suffer from prohibitive computational costs, while gradient-based approaches like DARTS offer high efficiency but are prone to premature convergence and performance collapse. To bridge this gap, we propose G-ICSO-NAS, a hybrid framework implementing a three-stage optimization strategy. The Warm-up Phase pre-trains supernet weights (w) via differentiable methods while architecture parameters (α) remain frozen. The Exploration Phase adopts a hybrid co-optimization mechanism: an Improved Competitive Swarm Optimizer (ICSO) with diversity-aware fitness navigates the architecture space to update α, while gradient descent concurrently updates w. The Stability Phase employs fine-grained gradient-based search with early stopping to converge to the optimal architecture. By synergizing ICSO's global navigation capability with differentiable methods' efficiency, G-ICSO-NAS achieves remarkable performance with minimal cost. In the context of the DARTS search space, an accuracy of 97.46% is achieved on CIFAR-10 with a computational budget of just 0.15 GPU-Days. The method also exhibits strong transfer potential, recording accuracies of 83.1% (CIFAR-100) and 75.02% (ImageNet). Furthermore, regarding the NAS-Bench-201 benchmark, G-ICSO-NAS is shown to deliver state-of-the-art results across all evaluated datasets.
Xingbang Du, Enzhi Zhang, Rui Zhong +2
Information Systems Design Laboratory, Hokkaido University, Sapporo, Japan · Information Initiative Center, Hokkaido University, Sapporo, Japan
Current neural architecture search (NAS) methods are often limited by their predefined, restrictive search spaces. While recent large language model (LLM)-assisted NAS methods enable open-ended search spaces, they often suffer from inefficient exploration due to biased or low-quality design ideas. To address these issues, we propose to semi-automatically structure model design knowledge to guide the search process. Our approach first defines a high-level structural template of architectural attributes. An LLM then populates this template by analyzing papers, creating a rich and diverse search space that embodies this structured design knowledge. To efficiently explore this vast space, we introduce FairNAD, using a multi-type mutation that enables broad exploration through mutation with fair idea sampling, Pareto-aware mutation, LLM-driven iterative mutation, and a fine-grained feedback loop. We demonstrate the effectiveness of FairNAD in discovering high-performing architectures that yield 0.84, 2.17, and 2.35 points improvement on CIFAR-10, CIFAR-100, and ImageNet16-120, respectively, compared to current state-of-the-art methods.
Yuiko Sakuma, Masakazu Yoshimura, Marcel Gröpl +4
Sony Group Corporation, Tokyo, Japan · ETH Zurich, Switzerland
Neural Architecture Search (NAS) is naturally formulated as a bilevel optimization problem, where the upper-level optimizes the architecture using validation performance and the lower-level trains network parameters using training loss. However, NAS is computationally expensive due to discrete architectural decisions, exponentially growing search spaces, and the high cost of training candidate architectures. This work develops a general bilevel optimization framework for NAS across diverse architectures, including MLPs, CNNs, RNNs, and Transformers, to identify compact architectures with strong predictive performance. We propose three scalable formulations that replace discrete neuron- and activation-level decisions with continuous relaxations, enabling differentiable optimization over otherwise combinatorial architecture spaces. These formulations give rise to three NAS methods: NAS based on Neuron Gating (NAS-NG), NAS based on Mixed Activation (NAS-MA), and NAS based on Neuron Gating and Mixed Activation (NAS-NGMA). Experiments on MLPs and CNNs using MNIST and CIFAR-10 show that the proposed methods consistently identify compact architectures with competitive or improved predictive performance. On MNIST, NAS-NGMA achieves 98.68% test accuracy with 7.69M MLP parameters, while NAS-NG achieves 99.63% accuracy with only 0.26M CNN parameters. On CIFAR-10, the proposed methods consistently outperform vanilla DARTS. Further experiments demonstrate that NAS-NG can optimize substantially over-parameterized and literature-optimal architectures, improving accuracy while reducing parameters. These results establish relaxed bilevel optimization as a scalable alternative to discrete NAS and provide a general framework for efficient neuron- and activation-level architecture optimization.
Abhishek Shukla, Ankur Sinha, Faiz Hamid
Department of Management Sciences, IIT Kanpur, India