Rank-Reliable Teacher-Guided Fitness Approximation for Expensive Evolutionary Optimization: A TinyML Architecture Search Study

Soumen Garai; Suman Samui

cs.AISep 24, 2026

Rank-Reliable Teacher-Guided Fitness Approximation for Expensive Evolutionary Optimization: A TinyML Architecture Search Study

Authors: Soumen Garai, Suman Samui

Organizations: Department of Electronics and Communication Engineering, National Institute of Technology Durgapur, West Bengal 713209, India

Abstract

Expensive evolutionary search does not always need an exact fitness estimate for every candidate. It often needs a reliable answer to a simpler question: which candidate is better? We address this need through Teacher-Guided Learning NSGA-II (TGL-NSGA-II), a low-fidelity framework for constrained Tiny Machine Learning (TinyML) neural architecture search. A pretrained teacher organizes samples into strata defined jointly by difficulty and class. Each candidate then undergoes KD-Lite, a short and capped knowledge-distillation procedure on a compact training set, before being scored on a separate stratified evaluation set. This teacher-guided score is fused with a Gaussian-process surrogate to select candidates for full evaluation. For a fixed candidate population, we analyse evaluation variance, score concentration, pairwise rank inversion, expected Kendall- $τ$ , first-front identification, and hypervolume perturbation. We also derive a variance-aware fusion weight and a capacity-adaptive distillation rule. On keyword spotting and bird-call classification, the measured Kendall- $τ$ values are 0.74 and 0.62, exceeding the corresponding predicted lower bounds of 0.60 and 0.46. Joint stratification reduces proxy-score variance by 41% relative to random evaluation. Selective teacher mismatch, in contrast, increases differential bias and reduces Kendall- $τ$ to 0.41. Under a constrained evaluation budget, TGL-NSGA-II achieves the largest mean hypervolume and smallest generational distance on keyword spotting, records the lowest mean false-positive rate on BirdCLEF, and runs 2.2x faster than full NSGA-II. These guarantees apply to population-level low-fidelity evaluation and do not establish convergence of the complete evolutionary trajectory.

Figures & tables

Literature group	Rank-aware by design	A priori error bound	Linked to HV perturbation
Surrogate-assisted evolutionary NAS (ENAS) [ 8 , 9 , 10 , 18 ]	Surrogate only	No	No
Low-fidelity and zero-cost proxies [ 5 , 7 , 6 , 26 ]	Measured post hoc	No	No
Curriculum learning with KD [ 36 , 37 , 33 ]	Not applicable (training)	Not applicable	No
SAEAs for ECOPs [ 1 , 15 , 16 , 17 ]	Surrogate only	No	No
Hardware-aware NAS [ 4 , 47 , 48 ]	Assumed input	No	No
This work	Proxy itself	Yes (Theorem 2 )	Yes (Proposition 1 )

TABLE I: Positioning of the proposed method relative to representative literature groups. HV denotes hypervolume.

Panel of Figure 2 — Fig. 1: Overview of TGL-NSGA-II. The teacher is used once to score sample difficulty and to form the disjoint proxy-training and proxy-evaluation sets. In each generation, every offspring receives a low-cost KD-Lite score and a GP-surrogate score. Their fused score, together with GP uncertainty, determines which offspring are promoted to expensive full training. Only fully evaluated promoted offspring update the archive and enter environmental selection with the parent population.

Analysis	Evidence
Ranking fidelity	Predicted and measured Kendall- $\tau$ on the same fixed architecture set.
Component contribution	Random, difficulty-only, no-curriculum, and no-teacher ablations with variance and Kendall- $\tau$ .
Teacher robustness	Strong, uniformly weak, and selectively weak teachers with common bias, differential bias, and Kendall- $\tau$ .
Constrained search	Hypervolume, generational distance, spread, feasible rate, FPR, and set coverage under an equal full-evaluation budget.
Efficiency and selection	Wall-clock cost and Tchebycheff-ranked deployable architectures.

TABLE II: Sequence of experimental analyses and the corresponding evidence.

Fig. 2: Residual convolutional backbone used for spectrogram-based audio classification. Bold blue tokens ( filters , kernel_size , batch-normalization flags, residual-block count, dense-layer units , and dropout) are the discrete design variables optimized by the search; a global average pooling layer precedes the softmax classifier to keep the parameter count within the TinyML budget.

Parameter	NSGA- II	MOBO	SA- NSGA-II	TGL
Population $P$	40	—	40	40
Generations $G$	50	50	50	50
Crossover / mutation	0.9 / 0.1	—	0.9 / 0.1	0.9 / 0.1
SBX / mutation index	15 / 20	—	15 / 20	15 / 20
Surrogate	—	GP	GP	GP
Promotion fraction $k$	—	—	20%	20%

TABLE III: Main search configuration used for all reported runs. A dash (—) indicates that a parameter is not applicable to a method.

Dataset	$\hat{\rho}_{\mathrm{str}}$	$\hat{\sigma}^{2}$	$\hat{\bar{\xi}}$	$\tau_{\mathrm{pred}}$	$\tau_{\mathrm{meas}}$
GSC	0.42	0.0964	0.018	0.60	0.74
BirdCLEF	0.34	0.1642	0.028	0.46	0.62

TABLE IV: Predicted and measured ranking fidelity on the fixed architecture set. Values are aggregate estimates from repeated proxy evaluations.

Fig. 3: Generation-wise surrogate fidelity (average of normalized Spearman correlation and Top-5 overlap, three-generation rolling mean). The legacy label SA-KD-NSGA-II denotes the teacher-guided variant folded into TGL-NSGA-II.

Variant / Teacher	$\operatorname{Var}(S)/\operatorname{Var}(S_{\mathrm{rand}})$	$\|\hat{\bar{B}}\|$	$\hat{\bar{\xi}}$	Kendall- $\tau$
Full TGL (strong teacher)	0.59	0.042	0.021	0.68
Random evaluation	1.00	N/A	N/A	0.56
Difficulty strata only	0.71	N/A	N/A	0.63
Without curriculum	0.60	N/A	N/A	0.64
Without teacher guidance	0.83	N/A	N/A	0.49
Uniformly weak teacher	N/A	0.086	0.025	0.62

TABLE V: Component and teacher-quality ablations on the fixed architecture set. Variance is normalized by random evaluation; lower bias terms and higher Kendall-

\tau

are better. N/A indicates that a quantity is not applicable to that control.

Panel of Figure 9 — Fig. 4: Representative single-run audio search diagnostics. (a) HV rises steadily for TGL-NSGA-II; (b) TGL reaches a fully feasible population early and stays near one.

Panel of Figure 10 — Fig. 5: BirdCLEF operational analysis. (a) TGL-NSGA-II attains the lowest mean FPR (lower is better). (b) Three-objective profile with model size and FPR inverted so larger radial values are better on all axes.

Method	HV $\uparrow$	GD $\downarrow$	Spread $\downarrow$
NSGA-II [ 3 ]	$0.0049{\pm}0.0006$	$0.0032{\pm}0.0004$	$1.028{\pm}0.031$
MOBO [ 47 ]	$0.0809{\pm}0.0071$	$0.1210{\pm}0.0153$	$1.218{\pm}0.048$
Zero-Cost NSGA-II [ 5 ]	$0.0714{\pm}0.0090$	$6.290{\pm}0.812$	$1.921{\pm}0.087$
SA-NSGA-II [ 10 ]	$0.0878{\pm}0.0043$	$0.0459{\pm}0.0061$	$\mathbf{0.961{\pm}0.024}$
TGL-NSGA-II	$\mathbf{0.0935{\pm}0.0038}$	$\mathbf{0.0019{\pm}0.0003}$	$1.049{\pm}0.029$

TABLE VI: Pareto-front quality on the KWS runs (mean

\pm

standard deviation over five seeds; higher HV is better, lower GD and spread are better). Best mean per column in bold.

	TGL	NSGA-II	MOBO	SA-NSGA-II	Zero-Cost
TGL-NSGA-II	N/A	0.000	0.000	0.154	0.000
NSGA-II	0.071	N/A	0.071	0.000	0.000
MOBO	0.500	0.000	N/A	0.250	0.000
SA-NSGA-II	0.800	0.000	0.000	N/A	0.000
Zero-Cost	1.000	0.000	0.000	0.667	N/A

TABLE VII: Set-coverage

C

-matrix on the KWS runs (fronts pooled over five seeds). Entry

(i,j)

is the fraction of row-

i

solutions dominated by at least one column-

j

solution; diagonal entries are not applicable.

Method	Cost (s) $\downarrow$
NSGA-II [ 3 ]	$1389.58{\pm}42.7$
MOBO [ 47 ]	$147.10{\pm}9.8$
SA-NSGA-II [ 10 ]	$224.39{\pm}12.3$
TGL-NSGA-II	$621.71{\pm}18.5$

TABLE VIII: Measured computational cost on the audio search (seconds; mean

\pm

standard deviation over five seeds).

Dataset	Model	Accuracy	Size (MB)	FPR	Hyperparameters	Score
GSC	GSC_MOBO-1	$0.9121{\pm}0.0043$	$0.2743{\pm}0.006$	$0.0106{\pm}0.0012$	{32,3,0,1,1,1}	0.0006
	GSC_MOBO-2	$0.9024{\pm}0.0051$	$0.2743{\pm}0.006$	$0.0101{\pm}0.0011$	{32,2,0,1,1,1}	0.0034
	GSC_SA-NSGA-II-1	$0.9059{\pm}0.0047$	$0.3236{\pm}0.008$	$0.0105{\pm}0.0013$	{16,5,1,2,2,1}	0.0041
	GSC_NSGA-II-1	$0.9093{\pm}0.0049$	$0.1854{\pm}0.005$	$0.0896{\pm}0.0071$	{16,5,0,1,1,1}	0.0064
	GSC_TGL-NSGA-II-1	$0.8841{\pm}0.0055$	$0.1191{\pm}0.004$	$0.0129{\pm}0.0015$	{16,3,1,2,1,0}	0.0126
	GSC_TGL-NSGA-II-2	$0.8797{\pm}0.0058$	$0.0982{\pm}0.003$	$0.0134{\pm}0.0016$	{32,3,0,1,1,0}	0.0141

TABLE IX: Tchebycheff-ranked model configurations for both audio tasks (accuracy, size, and FPR reported as mean

\pm

standard deviation over five seeds). Hyperparameters are listed as {filters, kernel size, use BN, residual blocks, FC layers, use dropout}. Lower score is better.

Neural Architecture Search

Neural Architecture Search automates neural network design for target tasks, resource budgets, and hardware constraints.

See all 130 papers

Oct 1, 2026

LESS: Lightweight Evolutionary Supernet Search in Minutes

Low-cost NAS must both explore high-performing architectures and identify them reliably, yet reducing evaluation cost often weakens the fidelity of candidate comparisons. Training-free methods reduce evaluation cost by replacing learned task feedback with proxy signals measured at initialization. We introduce LESS (Lightweight Evolutionary Supernet Search), a data-driven method that combines a brief fair hard-path warm-up with discrete search under a single CMA-ES distribution. Each proposal is evaluated as its decoded hard genotype after six candidate-conditioned supernet updates. On NAS-Bench-201, LESS achieves

93.189\pm0.467\%

CIFAR-10 test accuracy in 409.1 seconds, coming within 0.04 percentage points of FairNAS using approximately

1/24

of its source-reported search time. Matched controls show that calibration improves selected validation accuracy by

0.577

percentage points while changing best-visited accuracy by only

0.054

points, indicating that its primary effect is to reduce selection regret. The frozen configuration transfers without tuning to CIFAR-100 and ImageNet16-120 with

69.615\pm1.139\%

and

43.720\pm1.697\%

accuracy. Applied without tuning to the larger DARTS space, LESS achieves

96.95\pm0.14\%

on CIFAR-10 and

82.43\pm0.80\%

on CIFAR-100, with each search completing in approximately 43.5 minutes on a single GPU. Together, these results show that short, balanced, data-dependent updates enable competitive neural architecture search across datasets and search spaces within minutes.

Aviral Gandhi, Jinglue Xu, Jialong Li +1

The University of Tokyo · Sakana AI · Waseda University

Evolutionary Optimization Covariance Matrix Adaptation Evolution Strategy Neural Architecture Search Neuroevolution

Ask PDF

Aug 10, 2026

ENAS: An Efficient Hardware-Aware Neural Architecture Search Framework for TinyML on Resource-Constrained Microcontrollers

We present ENAS, a hardware-aware Neural Architecture Search (NAS) framework that combines a static feasibility check, a cell-based search space supporting standard, depthwise-separable, and bottleneck blocks with optional skip connections, and a three-stage hybrid search strategy (random

\rightarrow

top-

K

\rightarrow

mutation) with persistent cross-run caching. Unlike many existing NAS frameworks that rely on GPU acceleration, ENAS is designed to operate efficiently without requiring GPUs, making it suitable for resource-constrained development environments. We evaluate ENAS on two TinyML benchmarks, Visual Wake Words and Melanoma Cancer, across eight microcontrollers with memory footprints ranging from 20,KB to 1,MB SRAM and nine input image resolutions. Our experimental results show that ENAS achieves mean search-time speedups of

2.41{\times}

and

1.70{\times}

on the Visual Wake Words and Melanoma Cancer datasets, respectively, while maintaining competitive test accuracy compared with the recent NanoNAS framework. A measured resource analysis further shows that ENAS-selected models use substantially lower peak activation RAM, the binding constraint for microcontroller deployment at matched accuracy. Additionally, ENAS achieves

79.4\%

test accuracy on an STM32H743-based microcontroller, outperforming the greedy CPU-only baseline by

2.6

percentage points. We release the ENAS framework as open-source at: https://github.com/EdgeIntelligenceLab/ENAS

Mohd Moin Khan, Naman Srivastava, Pandarasamy Arjunan

Indian Institute of Science (IISc), Bengaluru 560012, India

Hardware-Aware NAS Neural Architecture Search Tiny ML

Ask PDF

Sep 24, 2026

EvoTreeNAD: Genealogy-Guided Evolution for LLM-Driven Neural Architecture Discovery

AI-driven scientific discovery accelerates research by autonomously developing solutions and designs. Large language model (LLM) agents support this process through iterative generation and evaluation. Yet these iterations alone do not ensure cumulative progress or establish which directions to pursue next. Costly evaluation further constrains the scope of exploration. Neural architecture discovery brings these challenges together, coupling open-ended design with resource-intensive experimentation. We introduce EvoTreeNAD, a genealogy-guided evolutionary algorithm that constructs trainable architectures without a supplied seed or a hand-specified search space. Starting from an empty root, it grows a persistent genealogy in which each new node represents a complete architecture. Top-percentile values computed from each node and its descendants guide lineage selection. Using the selected design history, an Idea Agent proposes a variant and a Code Agent implements it. Each evaluated variant becomes a child node, expanding the genealogy while providing evidence for subsequent lineage selection. Our theoretical analysis establishes the existence of stationary variation regimes as the genealogy grows. Under specified variation assumptions, sustained top-percentile family values quantify the probability of generating high-reward architectures in these regimes. EvoTreeNAD discovers architectures that outperform the compared NAS and NAD baselines, achieving CIFAR-10/100 test errors of

2.05{\pm}0.06\%

and

15.09{\pm}0.22\%

. On all six MedMNIST-v2 tasks, the discovered architectures surpass the strongest listed baselines. A controlled CIFAR-10 study further shows that EvoTreeNAD outperforms direct generation, best-of-

N

greedy continuation, and full-family-mean routing.

Lishan Yu, Derek Jiu, Qizhen Lan +1

Department of Health Data Science and AI McWilliams School of Biomedical Informatics University of Texas Health Science Center at Houston

Large Language Model-Guided NAS Evolutionary Optimization Neural Architecture Search

Ask PDF

Abstract

Neural Architecture Search

Explore similar work

LESS: Lightweight Evolutionary Supernet Search in Minutes

ENAS: An Efficient Hardware-Aware Neural Architecture Search Framework for TinyML on Resource-Constrained Microcontrollers

EvoTreeNAD: Genealogy-Guided Evolution for LLM-Driven Neural Architecture Discovery