Rank-Reliable Teacher-Guided Fitness Approximation for Expensive Evolutionary Optimization: A TinyML Architecture Search Study

Soumen Garai; Suman Samui

cs.AISep 24, 2026

Rank-Reliable Teacher-Guided Fitness Approximation for Expensive Evolutionary Optimization: A TinyML Architecture Search Study

Authors: Soumen Garai, Suman Samui

Organizations: Department of Electronics and Communication Engineering, National Institute of Technology Durgapur, West Bengal 713209, India

Abstract

Expensive evolutionary search does not always need an exact fitness estimate for every candidate. It often needs a reliable answer to a simpler question: which candidate is better? We address this need through Teacher-Guided Learning NSGA-II (TGL-NSGA-II), a low-fidelity framework for constrained Tiny Machine Learning (TinyML) neural architecture search. A pretrained teacher organizes samples into strata defined jointly by difficulty and class. Each candidate then undergoes KD-Lite, a short and capped knowledge-distillation procedure on a compact training set, before being scored on a separate stratified evaluation set. This teacher-guided score is fused with a Gaussian-process surrogate to select candidates for full evaluation. For a fixed candidate population, we analyse evaluation variance, score concentration, pairwise rank inversion, expected Kendall- $τ$ , first-front identification, and hypervolume perturbation. We also derive a variance-aware fusion weight and a capacity-adaptive distillation rule. On keyword spotting and bird-call classification, the measured Kendall- $τ$ values are 0.74 and 0.62, exceeding the corresponding predicted lower bounds of 0.60 and 0.46. Joint stratification reduces proxy-score variance by 41% relative to random evaluation. Selective teacher mismatch, in contrast, increases differential bias and reduces Kendall- $τ$ to 0.41. Under a constrained evaluation budget, TGL-NSGA-II achieves the largest mean hypervolume and smallest generational distance on keyword spotting, records the lowest mean false-positive rate on BirdCLEF, and runs 2.2x faster than full NSGA-II. These guarantees apply to population-level low-fidelity evaluation and do not establish convergence of the complete evolutionary trajectory.

Figures & tables

Literature group	Rank-aware by design	A priori error bound	Linked to HV perturbation
Surrogate-assisted evolutionary NAS (ENAS) [ 8 , 9 , 10 , 18 ]	Surrogate only	No	No
Low-fidelity and zero-cost proxies [ 5 , 7 , 6 , 26 ]	Measured post hoc	No	No
Curriculum learning with KD [ 36 , 37 , 33 ]	Not applicable (training)	Not applicable	No
SAEAs for ECOPs [ 1 , 15 , 16 , 17 ]	Surrogate only	No	No
Hardware-aware NAS [ 4 , 47 , 48 ]	Assumed input	No	No
This work	Proxy itself	Yes (Theorem 2 )	Yes (Proposition 1 )

TABLE I: Positioning of the proposed method relative to representative literature groups. HV denotes hypervolume.

Panel of Figure 2 — Fig. 1: Overview of TGL-NSGA-II. The teacher is used once to score sample difficulty and to form the disjoint proxy-training and proxy-evaluation sets. In each generation, every offspring receives a low-cost KD-Lite score and a GP-surrogate score. Their fused score, together with GP uncertainty, determines which offspring are promoted to expensive full training. Only fully evaluated promoted offspring update the archive and enter environmental selection with the parent population.

Analysis	Evidence
Ranking fidelity	Predicted and measured Kendall- $\tau$ on the same fixed architecture set.
Component contribution	Random, difficulty-only, no-curriculum, and no-teacher ablations with variance and Kendall- $\tau$ .
Teacher robustness	Strong, uniformly weak, and selectively weak teachers with common bias, differential bias, and Kendall- $\tau$ .
Constrained search	Hypervolume, generational distance, spread, feasible rate, FPR, and set coverage under an equal full-evaluation budget.
Efficiency and selection	Wall-clock cost and Tchebycheff-ranked deployable architectures.

TABLE II: Sequence of experimental analyses and the corresponding evidence.

Fig. 2: Residual convolutional backbone used for spectrogram-based audio classification. Bold blue tokens ( filters , kernel_size , batch-normalization flags, residual-block count, dense-layer units , and dropout) are the discrete design variables optimized by the search; a global average pooling layer precedes the softmax classifier to keep the parameter count within the TinyML budget.

Parameter	NSGA- II	MOBO	SA- NSGA-II	TGL
Population $P$	40	—	40	40
Generations $G$	50	50	50	50
Crossover / mutation	0.9 / 0.1	—	0.9 / 0.1	0.9 / 0.1
SBX / mutation index	15 / 20	—	15 / 20	15 / 20
Surrogate	—	GP	GP	GP
Promotion fraction $k$	—	—	20%	20%

TABLE III: Main search configuration used for all reported runs. A dash (—) indicates that a parameter is not applicable to a method.

Dataset	$\hat{\rho}_{\mathrm{str}}$	$\hat{\sigma}^{2}$	$\hat{\bar{\xi}}$	$\tau_{\mathrm{pred}}$	$\tau_{\mathrm{meas}}$
GSC	0.42	0.0964	0.018	0.60	0.74
BirdCLEF	0.34	0.1642	0.028	0.46	0.62

TABLE IV: Predicted and measured ranking fidelity on the fixed architecture set. Values are aggregate estimates from repeated proxy evaluations.

Fig. 3: Generation-wise surrogate fidelity (average of normalized Spearman correlation and Top-5 overlap, three-generation rolling mean). The legacy label SA-KD-NSGA-II denotes the teacher-guided variant folded into TGL-NSGA-II.

Variant / Teacher	$\operatorname{Var}(S)/\operatorname{Var}(S_{\mathrm{rand}})$	$\|\hat{\bar{B}}\|$	$\hat{\bar{\xi}}$	Kendall- $\tau$
Full TGL (strong teacher)	0.59	0.042	0.021	0.68
Random evaluation	1.00	N/A	N/A	0.56
Difficulty strata only	0.71	N/A	N/A	0.63
Without curriculum	0.60	N/A	N/A	0.64
Without teacher guidance	0.83	N/A	N/A	0.49
Uniformly weak teacher	N/A	0.086	0.025	0.62

TABLE V: Component and teacher-quality ablations on the fixed architecture set. Variance is normalized by random evaluation; lower bias terms and higher Kendall-

\tau

are better. N/A indicates that a quantity is not applicable to that control.

Panel of Figure 9 — Fig. 4: Representative single-run audio search diagnostics. (a) HV rises steadily for TGL-NSGA-II; (b) TGL reaches a fully feasible population early and stays near one.

Panel of Figure 10 — Fig. 5: BirdCLEF operational analysis. (a) TGL-NSGA-II attains the lowest mean FPR (lower is better). (b) Three-objective profile with model size and FPR inverted so larger radial values are better on all axes.

Method	HV $\uparrow$	GD $\downarrow$	Spread $\downarrow$
NSGA-II [ 3 ]	$0.0049{\pm}0.0006$	$0.0032{\pm}0.0004$	$1.028{\pm}0.031$
MOBO [ 47 ]	$0.0809{\pm}0.0071$	$0.1210{\pm}0.0153$	$1.218{\pm}0.048$
Zero-Cost NSGA-II [ 5 ]	$0.0714{\pm}0.0090$	$6.290{\pm}0.812$	$1.921{\pm}0.087$
SA-NSGA-II [ 10 ]	$0.0878{\pm}0.0043$	$0.0459{\pm}0.0061$	$\mathbf{0.961{\pm}0.024}$
TGL-NSGA-II	$\mathbf{0.0935{\pm}0.0038}$	$\mathbf{0.0019{\pm}0.0003}$	$1.049{\pm}0.029$

TABLE VI: Pareto-front quality on the KWS runs (mean

\pm

standard deviation over five seeds; higher HV is better, lower GD and spread are better). Best mean per column in bold.

	TGL	NSGA-II	MOBO	SA-NSGA-II	Zero-Cost
TGL-NSGA-II	N/A	0.000	0.000	0.154	0.000
NSGA-II	0.071	N/A	0.071	0.000	0.000
MOBO	0.500	0.000	N/A	0.250	0.000
SA-NSGA-II	0.800	0.000	0.000	N/A	0.000
Zero-Cost	1.000	0.000	0.000	0.667	N/A

TABLE VII: Set-coverage

C

-matrix on the KWS runs (fronts pooled over five seeds). Entry

(i,j)

is the fraction of row-

i

solutions dominated by at least one column-

j

solution; diagonal entries are not applicable.

Method	Cost (s) $\downarrow$
NSGA-II [ 3 ]	$1389.58{\pm}42.7$
MOBO [ 47 ]	$147.10{\pm}9.8$
SA-NSGA-II [ 10 ]	$224.39{\pm}12.3$
TGL-NSGA-II	$621.71{\pm}18.5$

TABLE VIII: Measured computational cost on the audio search (seconds; mean

\pm

standard deviation over five seeds).

Dataset	Model	Accuracy	Size (MB)	FPR	Hyperparameters	Score
GSC	GSC_MOBO-1	$0.9121{\pm}0.0043$	$0.2743{\pm}0.006$	$0.0106{\pm}0.0012$	{32,3,0,1,1,1}	0.0006
	GSC_MOBO-2	$0.9024{\pm}0.0051$	$0.2743{\pm}0.006$	$0.0101{\pm}0.0011$	{32,2,0,1,1,1}	0.0034
	GSC_SA-NSGA-II-1	$0.9059{\pm}0.0047$	$0.3236{\pm}0.008$	$0.0105{\pm}0.0013$	{16,5,1,2,2,1}	0.0041
	GSC_NSGA-II-1	$0.9093{\pm}0.0049$	$0.1854{\pm}0.005$	$0.0896{\pm}0.0071$	{16,5,0,1,1,1}	0.0064
	GSC_TGL-NSGA-II-1	$0.8841{\pm}0.0055$	$0.1191{\pm}0.004$	$0.0129{\pm}0.0015$	{16,3,1,2,1,0}	0.0126
	GSC_TGL-NSGA-II-2	$0.8797{\pm}0.0058$	$0.0982{\pm}0.003$	$0.0134{\pm}0.0016$	{32,3,0,1,1,0}	0.0141

TABLE IX: Tchebycheff-ranked model configurations for both audio tasks (accuracy, size, and FPR reported as mean

\pm

standard deviation over five seeds). Hyperparameters are listed as {filters, kernel size, use BN, residual blocks, FC layers, use dropout}. Lower score is better.

Neural Architecture Search

Neural Architecture Search automates neural network design for target tasks, resource budgets, and hardware constraints.

See all 132 papers

LESS: Lightweight Evolutionary Supernet Search in Minutes
Oct 1, 2026Aviral Gandhi, Jinglue Xu, Jialong Li +1Evolutionary Optimization Covariance Matrix Adaptation Evolution Strategy
The University of Tokyo · Sakana AI · Waseda University
ENAS: An Efficient Hardware-Aware Neural Architecture Search Framework for TinyML on Resource-Constrained Microcontrollers
Aug 10, 2026Mohd Moin Khan, Naman Srivastava, Pandarasamy ArjunanHardware-Aware NAS Neural Architecture Search
Indian Institute of Science (IISc), Bengaluru 560012, India
EvoTreeNAD: Genealogy-Guided Evolution for LLM-Driven Neural Architecture Discovery
Sep 24, 2026Lishan Yu, Derek Jiu, Qizhen Lan +1Large Language Model-Guided NAS Evolutionary Optimization
Department of Health Data Science and AI McWilliams School of Biomedical Informatics University of Texas Health Science Center at Houston

Oct 1, 2026J/K move · Enter open · S save

LESS: Lightweight Evolutionary Supernet Search in Minutes

Aviral Gandhi, Jinglue Xu, Jialong Li, Hitoshi Iba

The University of Tokyo · Sakana AI · Waseda University

Evolutionary Optimization Covariance Matrix Adaptation Evolution Strategy Neural Architecture Search Neuroevolution

Open paper AskPDF arXiv

Low-cost NAS must both explore high-performing architectures and identify them reliably, yet reducing evaluation cost often weakens the fidelity of candidate comparisons. Training-free methods reduce evaluation cost by replacing learned task feedback with proxy signals measured at initialization. We introduce LESS (Lightweight Evolutionary Supernet Search), a data-driven method that combines a brief fair hard-path warm-up with discrete search under a single CMA-ES distribution. Each proposal is evaluated as its decoded hard genotype after six candidate-conditioned supernet updates. On NAS-Bench-201, LESS achieves

93.189\pm0.467\%

CIFAR-10 test accuracy in 409.1 seconds, coming within 0.04 percentage points of FairNAS using approximately

1/24

of its source-reported search time. Matched controls show that calibration improves selected validation accuracy by

0.577

percentage points while changing best-visited accuracy by only

0.054

points, indicating that its primary effect is to reduce selection regret. The frozen configuration transfers without tuning to CIFAR-100 and ImageNet16-120 with

69.615\pm1.139\%

and

43.720\pm1.697\%

accuracy. Applied without tuning to the larger DARTS space, LESS achieves

96.95\pm0.14\%

on CIFAR-10 and

82.43\pm0.80\%

on CIFAR-100, with each search completing in approximately 43.5 minutes on a single GPU. Together, these results show that short, balanced, data-dependent updates enable competitive neural architecture search across datasets and search spaces within minutes.

Abstract

Neural Architecture Search

Explore similar work

LESS: Lightweight Evolutionary Supernet Search in Minutes

ENAS: An Efficient Hardware-Aware Neural Architecture Search Framework for TinyML on Resource-Constrained Microcontrollers

EvoTreeNAD: Genealogy-Guided Evolution for LLM-Driven Neural Architecture Discovery