CAGE-NAS: Certified Functional Descent for Efficient Model Growth

Santiago Florido Gomez; Stéphane Rivaud

cs.LGOct 1, 2026

CAGE-NAS: Certified Functional Descent for Efficient Model Growth

Authors: Santiago Florido Gomez, Stéphane Rivaud

Organizations: Institut Polytechnique de Paris Paris, France · Inria Saclay, TAU Team Paris, France

Abstract

The progressive growth of neural networks requires deciding when the current representation remains sufficient for optimization and when it should be expanded. CAGE-NAS formulates this decision in function space through an admissibility criterion on approximations of the functional gradient. As long as a representation enables a certified Functional Gradient Descent step, the architecture remains fixed; when the criterion fails, a function-preserving expansion is applied and the resulting representation is evaluated again. As the main instance, we study the family induced by the tangent space, using a regularized projection of the functional gradient. In a controlled setting with exact certification, CAGE-NAS produces architectures positioned above the 99.8th performance percentile by held-out RMSE among all admissible alternatives within the same parameter budget, without enumerating them during the growth trajectory.

Figures & tables

Figure 1: CAGE-NAS discovers compact models in one train-and-grow trajectory. The percentile is measured by held-out RMSE against all screened models satisfying the strict

600

-parameter budget.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Seed	Architecture	Grow test RMSE
0	$[16,19,9]$	0.0642
1	$[9,18,18]$	0.0680
2	$[13,16,15]$	0.0585
3	$[11,17,17]$	0.0544

Appendix

Table 1: Final architectures and test RMSE values of the CAGE-NAS train-and-grow runs.

Panel of Figure 2 — Figure 2: Training and architectural-growth behavior of CAGE-NAS on the controlled synthetic $N=1024$ benchmark for seed $0$ over $70$ epochs.

Panel of Figure 3 — Figure 3: Test behavior of CAGE-NAS on the controlled synthetic $N=1024$ benchmark for seed $0$ over $70$ epochs.

Seed	Growth RMSE	Grid RMSE
0	0.0642	0.0538
1	0.0680	0.0674
2	0.0585	0.0589
3	0.0544	0.0641

Appendix

Table 2: Comparison between the final CAGE-NAS growth result and the candidate selected by minimum validation MSE from the saved strict-budget grid results for each seed. Lower test RMSE is better.

Figure 4: Distribution of the test score $1-\mathrm{RMSE}$ for the screened fixed MLPs satisfying $p(A)\leq 600$ . Red stars indicate the corresponding CAGE-NAS train-and-grow runs; higher is better.

Seed	Architectures evaluated	Growth percentile
0	17,339	99.95
1	17,339	99.88
2	17,339	100.00
3	17,339	100.00

Appendix

Table 3: Position of the CAGE-NAS growth result within the distribution of architectures evaluated by grid search.

Seed	Architecture	LR	WD	Scheduler	Val RMSE	Fixed RMSE	Growth RMSE
0	16-19-9	0.05	0.0	cosineannealing	0.0719	0.0705	0.0642
1	9-18-18	0.04	0.001	cosineannealing	0.0660	0.0626	0.0680
2	13-16-15	0.08	0.0	cosineannealing	0.0719	0.0749	0.0585
3	11-17-17	0.08	0.001	cosineannealing	0.0687	0.0721	0.0544

Appendix

Table 4: Comparison between the final CAGE-NAS train-and-grow result and retraining the discovered fixed architecture using a hyperparameter grid search.

(X,Y)\leftarrow\textsc{BuildCertificationSet}(\mathcal{D}_{\mathrm{tr}})

Appendix

Algorithm 1 Epoch-level certified adaptive training

(A_{c},\theta_{c},\varepsilon_{c},s_{c})\leftarrow\textsc{GrowUntilCertified}(A,\theta,X,Y)

Appendix

Algorithm 2 One certified function-space outer step

Explore similar work

CoRA-NAS: Coarse Ranking and Anchor-Residual Refinement for Neural Architecture Search
Sep 10, 2026Yifan Yang, Zhaoyan Wang, Zheng Gao +2Neural Architecture Search Capability-Speed Trade-Offs
University of New South Wales · Korea Advanced Institute of Science and Technology
On the Stability of Growth in Structural Plasticity
May 14, 2026Lute Lillo, Nick CheneyPlasticity Growth
University of Vermont
From Regression to Inference: Meta-Learning Predictors for Neural Architecture Search
May 10, 2026Liping Deng, MingQing XiaoNeural Architecture Search Meta-Learning
Department of Mathematics University of California, Riverside Riverside, CA 92507 USA · School of Mathematical and Statistical Sciences Southern Illinois University Carbondale Carbondale, IL 62901 USA

Sep 10, 2026, cs.LGJ/K move · Enter open · S save

CoRA-NAS: Coarse Ranking and Anchor-Residual Refinement for Neural Architecture Search

Yifan Yang, Zhaoyan Wang, Zheng Gao, Xiaoyu Li+1

University of New South Wales · Korea Advanced Institute of Science and Technology

Neural Architecture Search Capability-Speed Trade-Offs Cross-Architecture Generalization Search Algorithms

Open paper AskPDF arXiv

Zero-cost proxies rank architectures cheaply, but their reliability varies across search spaces. We introduce CoRA-NAS (COarse Ranking + Anchor-residual), a two-stage framework combining a static ranking prior with low-cost learning-curve refinement. CoRA-Rank aggregates capacity and structure-at-initialization proxies through an equal-weight log-rank consensus and a target-free consensus gate. CoRA-Refine samples anchors across this prior, extrapolates their early validation curves, and propagates a learned residual correction with an ExtraTrees model. The refinement uses approximately 1% of the cost of fully training the candidate set. Fully trained architecture-accuracy labels are not used to fit the ranker. One configuration is used across spaces, with space-specific architecture encodings. Across NAS-Bench-201, NAS-Bench-101, TransNAS-Bench-101, and NATS-SSS, CoRA-Refine achieves mean Spearman correlations of 0.946, 0.715, 0.786, and 0.894, respectively. Its worst-space correlation of 0.715 is the highest among the compared methods. On NAS-Bench-201/CIFAR-100, its selected architecture reaches 73.32% accuracy, near the reported ground-truth best of 73.37%. On the pure size space, refinement recovers the static prior's shortfall relative to parameter count, while remaining tied with the strongest capacity proxies within noise. The resulting framework combines cross-space ranking robustness with low-cost architecture selection.

Abstract

Explore similar work

CoRA-NAS: Coarse Ranking and Anchor-Residual Refinement for Neural Architecture Search

On the Stability of Growth in Structural Plasticity

From Regression to Inference: Meta-Learning Predictors for Neural Architecture Search