Large Language Model-Guided Evolutionary Discovery of Native Neural Architectures for Spiking Sequence Modeling

Ruoyu Zhao; Jiaqi Wu; Chenyu Zhu; Zhichao Lu

cs.NESep 30, 2026

Large Language Model-Guided Evolutionary Discovery of Native Neural Architectures for Spiking Sequence Modeling

Authors: Ruoyu Zhao, Jiaqi Wu, Chenyu Zhu, Zhichao Lu

Organizations: Department of Computer Science, City University of Hong Kong

Abstract

Spiking neural networks (SNNs) offer low-energy sequence modeling through sparse, event-driven computation. However, interactions among spike encoding, neuronal dynamics, and information propagation complicate architecture design. Existing SNN sequence models often adapt artificial neural network (ANN) architectures designed for real-valued activations, potentially underusing spike-based communication and temporal state updates, motivating automated discovery of native SNN architectures. Most evolutionary neural architecture search (ENAS) methods operate within predefined configuration spaces, limiting discovery to mechanisms expressible within those spaces. We introduce OpenArchEvo, which uses large language models (LLMs) to evolve executable architecture code in an open program space under spiking-projection constraints. In this space, code differences need not reflect architectural novelty, while direct performance evaluation requires costly training. We construct a three-view representation spanning code, design rationale, and a behavioral fingerprint to support novelty estimation and performance prediction. The search treats predicted performance and estimated novelty as two objectives, using surrogate predictions to select candidates for expensive training evaluations. With an estimated candidate-training cost of 132 V100 GPU-days, the search uncovers multiple native SNN architectures, exemplified by three designs featuring mechanisms such as spike-activity-dependent control of state updates and residual pathways. The discovered NeuroGate surpasses the ANN DeltaNet on WikiText-103, and the discovered architectures reduce estimated architecture-level arithmetic energy by up to 50.6x (LoopMem) relative to a common dense Transformer (ANN) baseline. All code and all discovered architectures will be made publicly available soon.

Figures & tables

Figure 1: Motivation and result preview for native SNN architecture discovery. (a) A qualitative map of LLM-based algorithm design by candidate-characterization difficulty and evaluation cost. (b) Partial rank agreement is shown by separately trained ANN and SNN implementations of the 97 ASI-Arch paired architectures on WikiText-2 (Kendall $\tau_{b}=0.2997$ ; Appendix A.1 ). (c) NeuroGate achieves 26.4 WikiText-103 perplexity and a 31.7 $\times$ estimated architecture-level arithmetic-energy reduction relative to a common dense Transformer (ANN) baseline (Table 1 ), under spiking-projection constraints (Appendix A.2 ). Red arrows mark the added pathway through which spike activity modulates the recurrent update and output gating.

Figure 2: Overview of OpenArchEvo. The inner loop generates feasible SNN architecture programs and uses surrogate predictions to guide evolution. The outer loop selects a small training batch using predicted performance and estimated novelty, then adds measured outcomes to the evaluated archive. Algorithms 1 and 2 in Appendix C give the procedures.

Figure 3: Three-view representation of an SNN architecture program. Code and design rationale describe its implementation and intended choices; a behavioral fingerprint combines architecture statistics with initialization-time response probes. Pairwise comparisons across the views estimate novelty. The fingerprint alone supplies architecture features for surrogate performance prediction and near-duplicate screening.

ANN baselines
Model	WT103	LRA accuracy (%) $\uparrow$							Energy reduction $\uparrow$
Model	PPL $\downarrow$	ListOps	Text	Retrieval	Image	Pathfinder	Path-X	Avg.	Energy reduction $\uparrow$
DeltaNet ( Yang et al., 2024b )	27.5	62.2	85.1	91.3	89.8	94.2	94.9	86.2	–
S4D-Lin † ( Gu et al., 2022 )	–	60.5	87.0	91.0	87.9	94.0	92.8 ∗	85.5	–
SNN baselines
S6-based SNN †	–	55.7	77.6	88.5	80.1	83.4	–	–	–

Table 1: Sequence-modeling performance and estimated energy reduction. WT103: WikiText-103 language modeling ( Merity et al., 2017 ) ; LRA: Long Range Arena ( Tay et al., 2021 ) . Energy reduction is the estimated arithmetic energy of a dense Transformer of the same size divided by that of each SNN under a common architecture setting (Section 4.2 and Appendix F ). Bold marks the best result in each column; “–” denotes unavailable results.

Figure 4: Measured fitness of trained candidates during the main search, in evaluation order. E denotes the initial expert models (grey) and 1–10 the search iterations. The red trace shows the best-so-far fitness, including the experts, and stars mark its improvements. The dashed line marks the baseline fitness of

0.500

, attained by SpikingDeltaNet, the strongest expert.

Figure 5: HomeoResSSM and LoopMem. HomeoResSSM uses spike activity to modulate two residual branches. LoopMem feeds a normalized summary of its recurrent state back to the forget gate and adds activity penalties to the training loss. Red paths highlight activity and feedback connections; paths to the loss act during training. NeuroGate is shown in Fig. 1 (c).

Variant	WT2 PPL $\downarrow$	$\Delta$
① Spike-activity-dependent computation
HomeoResSSM
Discovered	57.5	–
Control driven by continuous inputs	59.2	$+1.7$
Control removed	64.4	$+6.9$
② Spiking-specific design choices

Table 2: Ablations of evolved elements on WT2. Block ① tests whether a control relies on spike activity; block ② tests design choices that depart from ANN practice. Non-spiking variants remove the spiking neurons and all spike-dependent components.

\Delta

is relative to the closest row above marked “–”.

Figure 6: Inner-loop search comparison in six-iteration WT2 searches: (a) best fitness and (b) hypervolume of the nondominated set in fitness and novelty after each stage. The EoH and FunSearch adaptations replace our inner-loop search and share the outer evaluation and surrogate-selection loop. Curves show means over three runs; shaded regions indicate standard deviation.

	Adapted ASI-Arch	OpenArchEvo
Best fitness $\uparrow$	0.653	0.872
Top-10 mean fitness $\uparrow$	0.570	0.714
WT103 PPL $\downarrow$	29.6	26.4
ListOps accuracy $\uparrow$	55.3	61.8
Text accuracy $\uparrow$	82.3	82.4
Retrieval accuracy $\uparrow$	85.3	90.1

Table 3: Outcomes and organization of the completed program-search runs. Downstream results compare SpikingCondFuse and NeuroGate, respectively.

Figure 7: Screening raw proposals in the main search. Compilation failures and causal-check failures account for 18% and 3%; fingerprint-based near-duplicate screening removes 30%, leaving 49% accepted. Rounded percentages share the raw-proposal denominator.

Selection method	Top-1 PPL $\downarrow$	Top-5 PPL $\downarrow$	Novelty $\uparrow$	Variance $\downarrow$
Evolution + surrogate	56.2	57.4	0.534	0.0281
Single island + surrogate	57.3	57.9	0.513	0.0272
Sampling + surrogate	57.3	57.8	0.520	0.0314
Sampling only	57.5	58.5	0.511	0.0414

Table 4: Selection variants in six-iteration WT2 searches (three runs). Statistics are computed over the final-iteration trained candidates pooled across runs: Top-1 and Top-5 give the lowest and the mean of the five lowest PPL, Novelty the mean novelty, and Variance the variance of WT2 fitness.

Figure 8: Offline surrogate studies on WT2 (top, 223 architectures) and ListOps (bottom, 54). Kendall

\tau

compares regressors (a), progressive surrogate-input additions (b), and labeled-context sizes (c). The first two studies share repeated cross-validation splits. In (b), gray shades mark the measurement group added at each cumulative step, and the task color marks the complete fingerprint. The +SWSP step adds both SWSP and FireRate; +Par. adds the parameter count and +Comp the other module statistics, while +Dim, +Res., and +Graph add the dimension-flow, resource-allocation, and subgraph statistics (Table 6 ). Learning curves use fixed holdouts and 20 training-pool subsamples per size, with

\pm 1\sigma

bands. Appendix D.3 gives the protocols.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method and source location	Encoded architectural choices	Relation to the discovered computations
AutoSNN ( Na et al., 2022 ) , Sec. 4.1	Five block choices in a fixed backbone: skip, spiking convolution, and spiking residual blocks, with specified kernel sizes.	The block menu needs a spike-statistic controller and its target connection to select the activity-dependent update or residual modulation.
SNASNet ( Kim et al., 2022 ) , Cell Search Strategy	Four-node cells with zero, skip, convolution, and pooling operations on forward and cross-time backward edges.	Temporal feedback is already permitted. A spike-statistic reduction and learned multiplicative control are additional operations beyond this edge menu.
MSE-NAS ( Pan et al., 2025 ) , Sec. III, Fig. 1	A multiscale genotype selects layer operations, excitatory/inhibitory types, motifs, and global connections under a specified decoder.	These choices alter operations and connectivity; the decoder would need to expose activity-conditioned control equations and their attachment points.
EQ-SpikeLM ( Zhang et al., 2026 ) , Sec. IV-B.1, Eqs. (14)–(16)	Per-layer preserved channel ratios for query–key, value, and feed-forward projections in a pretrained spiking language model.	Channel pruning changes widths within the existing computation; it does not introduce a new spike-derived control dependency.
ENAS (efficient NAS), recurrent space ( Pham et al., 2018 ) , Secs. 2.1, 3.1	Predecessor and activation choices in a recurrent cell, with a prescribed highway-gating construction.	Recurrence and multiplicative gating are present. Spike-statistic inputs and revisions to the controller’s equation or target require extending the template.
DARTS, recurrent space ( Liu et al., 2019 ) , Sec. 3.1.2	Operation choices over linear transforms and activations, identity, and zero, within a recurrent-cell template with highway bypasses.	Selecting operations does not itself expose the spike-reduction and control-target relation; these must be added to the operations or template.

Appendix

Table 5: Scope of documented search encodings. The final column identifies missing choices for directly representing the examined SNN computations. Source locations specify the versions and sections inspected; extending an encoding can change its scope.

Group	Measurement	Type	Description
Module statistics	Params	C	Total trainable parameters
	SpkLinear	C	Spiking linear module count
	Conv1d	C	1-D convolution count
	GateRatio	C	Gating layer fraction
Structural statistics	Depth	F	Shape-changing layer count
	Expand	F	Dim-increase steps

Appendix

Table 6: Complete 21-dimensional behavioral fingerprint used in Section 3.3 . The first two groups form architecture statistics; the third comprises initialization-time probes. Type indicates extraction cost: C = code analysis (static), F = forward-based measurement, F+B = forward + backward computations.

Model	Family	Venue	Ref.	Model	Family	Venue	Ref.
RetNet	Foundational	arXiv 2023	( Sun et al., 2023 )	S4	SSM	ICLR 2022	( Gu et al., 2022 )
LightNet	Foundational	TMLR 2026	( Qin et al., 2026 )	Mamba (S6)	SSM	COLM 2024	( Gu & Dao, 2024 )
GLA	Modern RNN	ICML 2024	( Yang et al., 2024a )	Mamba2 (SSD)	SSM	ICML 2024	( Dao & Gu, 2024 )
HGRN	Modern RNN	NeurIPS 2023	( Qin et al., 2023 )	DeltaNet	Delta Rule	NeurIPS 2024	( Yang et al., 2024b )
HGRN2	Modern RNN	COLM 2024	( Qin et al., 2024 )	DeltaFormer	Delta Rule	arXiv 2025	( Zhong et al., 2025 )
RWKV-6	Modern RNN	COLM 2024	( Peng et al., 2024a )	Gated DeltaNet	Delta Rule	ICLR 2025	( Yang et al., 2025 )

Appendix

Table 7: The 20 architectures in the initial expert archive

\mathcal{D}_{0}

, each adapted to the spiking-projection constraint.

Setting	Value
LLM / maximum generation tokens	Gemini-2.5-Flash / 32,768
Outer iterations / accepted candidates per iteration	10 / 1,080
Batch limit $K$ / evaluation budget $B$	16 / 160
Islands / retained capacity	10 / approximately 20 per island
Population trimming threshold	Twice the retained capacity
Inter-island parent sampling probability	0.5

Appendix

Table 8: Main-search configuration.

Figure 9: Comparison of candidate LLMs as program generators. Each axis is normalized across models, with 1.0 for the best and 0.2 for the worst.

Figure 10: Evolution of the nondominated front in composite fitness and novelty during the main search. Each panel adds the architectures trained in one iteration (colored) to those of earlier iterations (faded); stars mark the nondominated architectures. Step 0 contains the initial expert models. Labels A–C mark the three selected architectures in the iteration where each first appears and on the final front.

Figure 11: Evolution of the nondominated front in ListOps accuracy and WT2 perplexity during the main search, displayed as in Fig. 10 ; the horizontal axis shows

\log_{10}

WT2 perplexity, decreasing to the right.

Setting	WT2	ListOps
Block layers / width	6 / 256	2 / 128
Attention heads	8	4
FFN expansion	4	2
Dropout	0.2	0.1
Batch size	16	32
Epochs	30	25

Appendix

Table 9: Proxy-training configurations used to compute search fitness.

Figure 12: Effect of the warm-start stage in six-iteration WT2 searches: (a) best fitness and (b) hypervolume of the nondominated set in fitness and novelty after each stage, computed as in Fig. 6 . Curves show means over three runs; shaded regions indicate standard deviation.

Architecture	$\rho$ (%)	SOPs (G)	MACs (G)	Energy (mJ)	Reduction
Dense Transformer	–	–	9.809	45.12	1.0 $\times$
Dyn-SSM	9.3	0.800	0.140	1.36	33.1 $\times$
SpikingDeltaNet	9.8	0.883	0.225	1.83	24.7 $\times$
SpikingMamba2	12.2	1.247	0.420	3.05	14.8 $\times$
SpikingCondFuse	8.3	0.782	0.367	2.39	18.9 $\times$
NeuroGate	4.6	0.424	0.226	1.42	31.7 $\times$

Appendix

Table 10: Arithmetic energy under the common architecture setting.

\rho

is the measured, operation-weighted input firing rate; SOP and MAC counts are in billions per sequence; reductions are computed from unrounded counts.

Explore similar work

Apr 10, 2026cs.LG

Structured Progressive Knowledge Activation for LLM-Driven Neural Architecture Search

This paper focuses on a key challenge in Neural Architecture Search (NAS): integrating established architectural knowledge while exploring new designs under expensive evaluations. Large language models (LLMs) are a promising assistant for NAS because they can translate rich architectural and coding priors into executable code edits. However, in practice, seemingly local revisions often propagate into non-local behavioral and performance shifts because a single edit can inadvertently couple multiple interacting functional factors, a phenomenon we refer to as functional entanglement. To make LLM knowledge usable under such entanglement, we propose Structured Progressive Knowledge Activation (SPARK), which activates relevant priors by explicitly selecting the functional factor to modify and conditioning the edit on that factor. This factor-conditioned editing reduces entangled side effects and yields more targeted, reliable architecture modifications. On CLRS-DFS, SPARK achieves a 28.1x sample-efficient architecture evolution speedup and yields a 22.9% relative improvement in OOD accuracy. Our code is available at https://github.com/AIM-ResearchLab/SPARK.

Zhen Liu, Yuhan Liu, Jinjun Wang +3

State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University. · Zhongguancun Academy, Beijing, China. · MiLM Plus, Xiaomi Inc. +1

Neural Architecture Search Entanglement

Ask PDF

Jun 10, 2026cs.NE

SpikeDecoder: Realizing the GPT Architecture with Spiking Neural Networks

The Transformer architecture is widely regarded as the most powerful tool for natural language processing, but due to a high number of complex operations, it inherently faces the issue of high energy consumption. To address this issue, we consider Spiking Neural Networks (SNNs), which are an energy-efficient alternative to conventional Artificial Neural Networks (ANNs) due to their naturally event-driven approach to processing information. However, this inherently makes them difficult to train. Often, many SNN-based models circumvent this issue by converting pre-trained ANNs. More recently, attempts have been made to design directly trainable SNN-based adaptations of the Transformer model structure. Although the results showed great promise, the application field was computer vision. Moreover, the proposed model incorporates only encoder blocks. In this paper, we propose SpikeDecoder, a fully SNN-based implementation of the Transformer decoder block, for applications in natural language processing. In a series of experiments, we analyze the impact of exchanging different blocks of the ANN model with spike-based alternatives to identify trade-offs and significant sources of performance loss. We further investigate the role of residual connections and the selection of SNN-compatible normalization techniques. Besides the work on the model architecture, we formulate and compare different embedding methods to project text data into spikes. Finally, we demonstrate that our proposed SNN-based decoder block reduces the theoretical energy consumption by 87% to 93% compared to the ANN baseline.

Claas Beger, Florian Walter, Alois Knoll

Chair of Robotics, Artificial Intelligence and Real-time Systems Technical University of Munich, Germany

Spiking Neural Networks Transformer Encoder Transformer Architectures Generative Pretrained Transformers Neural Network

Ask PDF

Sep 24, 2026cs.NE

EvoTreeNAD: Genealogy-Guided Evolution for LLM-Driven Neural Architecture Discovery

AI-driven scientific discovery accelerates research by autonomously developing solutions and designs. Large language model (LLM) agents support this process through iterative generation and evaluation. Yet these iterations alone do not ensure cumulative progress or establish which directions to pursue next. Costly evaluation further constrains the scope of exploration. Neural architecture discovery brings these challenges together, coupling open-ended design with resource-intensive experimentation. We introduce EvoTreeNAD, a genealogy-guided evolutionary algorithm that constructs trainable architectures without a supplied seed or a hand-specified search space. Starting from an empty root, it grows a persistent genealogy in which each new node represents a complete architecture. Top-percentile values computed from each node and its descendants guide lineage selection. Using the selected design history, an Idea Agent proposes a variant and a Code Agent implements it. Each evaluated variant becomes a child node, expanding the genealogy while providing evidence for subsequent lineage selection. Our theoretical analysis establishes the existence of stationary variation regimes as the genealogy grows. Under specified variation assumptions, sustained top-percentile family values quantify the probability of generating high-reward architectures in these regimes. EvoTreeNAD discovers architectures that outperform the compared NAS and NAD baselines, achieving CIFAR-10/100 test errors of

2.05{\pm}0.06\%

and

15.09{\pm}0.22\%

. On all six MedMNIST-v2 tasks, the discovered architectures surpass the strongest listed baselines. A controlled CIFAR-10 study further shows that EvoTreeNAD outperforms direct generation, best-of-

N

greedy continuation, and full-family-mean routing.

Lishan Yu, Derek Jiu, Qizhen Lan +1

Department of Health Data Science and AI McWilliams School of Biomedical Informatics University of Texas Health Science Center at Houston

Neural Architecture Search Evolutionary Search Evolutionary Optimization Methods Tree Search

Ask PDF