cs.LGFeb 17, 2026

SEval-NAS: A Search-Agnostic Evaluation for Neural Architecture Search

Authors: Atah Nuh Mih, Jianzhou Wang, Truong Thanh Hung Nguyen, Hung Cao

Organizations: Analytics Everywhere Lab, University of New Brunswick, Canada

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Neural architecture search (NAS) automates the discovery of neural networks that meet specified criteria, yet its evaluation procedures are often hardcoded, limiting the ability to introduce new metrics. This issue is especially pronounced in hardware-aware NAS, where objectives depend on target devices such as edge hardware. To address this limitation, we propose SEval-NAS, a metric-evaluation mechanism that converts architectures to strings, embeds them as vectors, and predicts performance metrics. Using NATS-Bench and HW-NAS-Bench, we evaluated accuracy, latency, and memory. Kendall's ττ correlations showed stronger latency and memory predictions than accuracy, indicating the suitability of SEval-NAS as a hardware cost predictor. We further integrated SEval-NAS into FreeREA to evaluate metrics not originally included. The method successfully ranked FreeREA-generated architectures, maintained search time, and required minimal algorithmic changes. Our implementation is available at: https://github.com/Analytics-Everywhere-Lab/neural-architecture-search

Figures & tables

Figure 1. Proposed SEval-NAS methodology and its integration in a NAS pipeline
Figure 2. Plots of predicted vs true hardware cost of NATS-Bench TSS architectures ( T5-small ) for performance metrics reported on CIFAR-10, CIFAR-100, and ImageNet16-120. The strength of correlation increases as τ\tau approaches 1.
Figure 3. Plots of predicted vs true hardware cost of NATS-Bench SSS architectures ( T5-small ) for performance metrics reported on CIFAR-10, CIFAR-100, and ImageNet16-120. The strength of correlation increases as τ\tau approaches 1.
Figure 4. Kendall’s τ\tau correlation for predicted vs true accuracy in NATS-Bench TSS and SSS search spaces. Comparison includes (accuracy, latency) and (accuracy, memory) bi-objectives.
Figure 5. Plots of predicted vs true latency for NAS-Bench-201 architectures with T5-small encoder for 6 edge devices reported in the HW-NAS-Bench benchmark. The strength of correlation increases as τ\tau approaches 1.
Table 1. Test accuracy and time for various NAS algorithms for NATS-Bench.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6. Kendall’s τ\tau correlation for predicted vs true accuracy among encoders of T5-small , T5-base , and T5-large on NATS-Bench TSS and NATS-Bench SSS (accuracy, latency) and (accuracy, memory) bi-objectives.
Figure 7. Plots of predicted vs true hardware cost of NATS-Bench TSS architectures for performance metrics reported on T5-small , T5-base , and T5-large . The strength of correlation increases as τ\tau approaches 1.
Figure 8. Plots of predicted vs true hardware cost of NATS-Bench SSS architectures for performance metrics reported on T5-small , T5-base , and T5-large . The strength of correlation increases as τ\tau approaches 1.
Figure 9. Plots of predicted vs true latency for NAS-Bench-201 architectures with T5-base encoder for 6 edge devices reported in the HW-NAS-Bench benchmark. The strength of correlation increases as τ\tau approaches 1.
Figure 10. Plots of predicted vs true latency for NAS-Bench-201 architectures with T5-large encoder for 6 edge devices reported in the HW-NAS-Bench benchmark. The strength of correlation increases as τ\tau approaches 1.

Explore similar work

CardsList

  1. LESS: Lightweight Evolutionary Supernet Search in Minutes

    Oct 1, 2026Aviral Gandhi, Jinglue Xu, Jialong Li +1Evolutionary OptimizationCovariance Matrix Adaptation Evolution Strategy

    The University of Tokyo · Sakana AI · Waseda University

  2. Prediction-powered Neural Architecture Search

    Oct 1, 2026Pascal Janetzky, Yuxin Wang, Michael Klar +1Neural Architecture SearchPrediction-Powered Inference

    Bosch Center for Artificial Intelligence · LMU Munich Munich Center for Machine Learning(MCML)

  3. ENAS: An Efficient Hardware-Aware Neural Architecture Search Framework for TinyML on Resource-Constrained Microcontrollers

    Aug 10, 2026Mohd Moin Khan, Naman Srivastava, Pandarasamy ArjunanHardware-Aware NASNeural Architecture Search

    Indian Institute of Science (IISc), Bengaluru 560012, India