Per-Node Activation Function Evolution in Indirectly Encoded Substrates: Solvability, Limits, and Emergent Diversity

Romain Claret; Michael O'Neill; Paul Cotofrei; Kilian Stoffel

cs.NESep 10, 2026

Per-Node Activation Function Evolution in Indirectly Encoded Substrates: Solvability, Limits, and Emergent Diversity

Authors: Romain Claret, Michael O'Neill, Paul Cotofrei, Kilian Stoffel

Organizations: University of Neuchâtel, Switzerland · University College Dublin, Ireland

Abstract

Biological neurons achieve computational diversity through specialized types: tonic, bursting, adapting, and fast-spiking cells coexist within the same circuit. Artificial neural networks, by contrast, apply a single activation function uniformly to all nodes, which limits what they can represent. We show that this uniformity creates hard limits for evolutionary search: across sparse evolved substrates, monotonic functions fail to solve parity beyond its smallest instance, XOR, while a single oscillatory unit suffices at all tested scales. The gap is one of search and sparsity, not representation: monotonic networks can represent parity with a modest number of hidden units, and gradient descent recovers that solution. We evolve, to our knowledge for the first time in indirect encoding, per-node activation function assignments from an 18-function palette across more than 4,500 experimental runs spanning Boolean logic, regression, and spatial classification. Testing each of the 18 functions individually on Parity-4 reveals a three-tier solvability structure: oscillatory functions achieve 100%, intermediate functions 6.7-80%, and all 9 monotonic functions 0%. This divide is not universal. Recurrence collapses it, and gradient descent inverts it entirely, showing that the barrier is specific to evolutionary search in sparse substrates. What activation functions a network can use, beyond its topology and weights, determines what evolutionary search can solve. Indirect encoding discovers heterogeneous per-node activation assignments unlikely to be chosen by hand.

Figures & tables

Figure 1: System pipeline. The activation palette, through the per-node selection mechanism, assigns each node its own activation function; the CPPN genome (NEAT) is decoded into a substrate carrying these per-node activations, the substrate is evaluated for fitness, and NEAT selection closes the evolutionary loop. Three assignment methods determine how CPPN output maps to function selection.

Idx	Name	Formula	Category	Solve%	Avg Gen
Multi-crossing (100% solve, 2.9–69.2 gen)
4	sin	$\sin(x)$	Oscillatory	100	2.9
13	osc_adapt	$\sin(x)(1{-}0.2\|x\|)$	Oscillatory	100	3.7
11	burst	$\tanh(x){+}0.5\sin(3x)$	Oscillatory	100	10.7
16	band_pass	$e^{-\|x-1\|}{-}e^{-\|x+1\|}$	Adaptive	100	69.2
Intermediate (6.7–80% solve, 55–246 gen)

Table 1: 18 activation functions and their individual Parity-4 results ( Per-Function Ablation ;

N{=}30

seeds; 300 gen max). The solvability boundary follows oscillatory structure: functions with local extrema solve parity; 9 strictly monotonic functions never solve.

Figure 2: Oscillatory structure predicts solvability. Local extrema count in

[-5,5]

vs Parity-4 solve rate (

N{=}30

) for all 18 activation functions. Functions with more local extrema achieve higher solve rates. The two outliers at 2 extrema reflect extrema amplitude: band_pass has deep sign-reversing extrema (100%), while integrate has shallow perturbations of a monotonic trend (0%).

Problem	Samples	Sin%	Sin Gen	Tanh%	Tanh Fit.
Parity-2	4	100	1.0	50.0	0.930
Parity-3	8	100	2.2	3.3	0.890
Parity-4	16	100	2.9	0	0.863
Parity-5	32	100	2.9	0	0.826
Parity-6	64	100	3.4	0	0.825
Parity-7	128	100	3.7	0	0.821

Table 2: Parity scaling: sin vs tanh (

N{=}30

seeds each; population 150; 100 gen max). Sin achieves 100% across all sizes; tanh degrades monotonically. Percentages are solve rates (replications reaching the threshold).

Pop Size	Sin%	Sin Gen	Tanh Fit.
50	100	5.0	0.808
150	100	3.0	0.863
500	100	2.0	0.894
1,000	100	1.0	0.900
2,000	100	1.0	0.910

Table 3: Population reshapes the dynamics but not the oscillatory divide (Parity-4,

N{=}30

per cell, 100 gen max). Sin solves 100% at every size; tanh 0% at every size. Percentages are solve rates.

Method	Configuration	Accuracy	Std
CPPN-output	cppn_output_4	76.9%	0.29
	cppn_output_6	76.9%	0.29
	cppn_with_sin	76.1%	0.42
Weight-interp.	sign-based	75.2%	0.14
	magnitude-based	75.2%	0.14
	variance-based	75.2%	0.14

Table 4: Assignment method comparison on Two Spirals (

N{=}30

; depth 3). CPPN-output significantly outperforms the other methods; weight-interpretation gives no gain over the uniform baseline. cppn_output_

k

= CPPN-output over a

k

-function palette (4: monotonic only; 6: adds sin, gauss).

Palette Size	Functions	Avg Gen	Std
1 (sin only)	sin	1.7	0.7
3	sin, tanh, relu	3.4	1.2
4	+sigmoid	3.1	1.1
5	+identity	3.2	1.2
6	+gauss	2.6	1.1
7	+lelu	3.3	1.0

Table 5: Palette size vs convergence speed (

N{=}30

seeds per size; Parity-4; all 100% success).

Function	FF%	FF Gen	Rec%	Rec Gen
sin	100	2.9	100	2.0
osc_adapt	100	3.7	100	3.0
burst	100	10.7	100	5.0
band_pass	100	69.2	100	10.0
receptive	80	100.1	100	3.0
resonator	66.7	55.9	100	2.0

Table 6: Feedforward vs recurrent topology on Parity-4 (both

N{=}30

, 300 gen max, pop 150; solve threshold 0.95). Recurrence partially collapses the divide: oscillatory functions remain at 100%; bounded monotonic partially solve; unbounded/linear remain near 0%. % = solve rate.

Connection Type	Solve%	Mean Gen	Avg Fitness
feedforward only	0	—	0.914
h-to-h only	20.0	79.5	0.911
h-to-h + backward	13.3	191.2	0.917
h-to-h + lateral	6.7	117.5	0.881
h-to-h + self-loops	6.7	58.5	0.885
all recurrent	6.7	98.5	0.893

Table 7: Connection type decomposition (tanh on Parity-4,

N{=}30

, 300 gen max, pop 150; solve threshold 0.95). Each row adds one connection type to hidden-to-hidden (h-to-h) as base.

Function	FF%	Full Rec%	Hidden-only%
osc_adapt	100	100	100
sin	100	80	50
resonator	100	70	57
gauss	43	3.3	3.3
burst	0	0	0
sigmoid	3.3	0	6.7

Table 8: Recurrence on concentric circles (

N{=}30

each; solve threshold 0.90). Opposite to parity, recurrence degrades most functions; only osc_adapt stays at 100%. FF% = feedforward solve rate; Full Rec% = full recurrence; Hidden-only% = recurrence on hidden-to-hidden connections only.

	Parity-4		Sine Regr.		Step Func.		Circles
Function	%	Gen	%	Gen	%	Gen	%	Gen
sin	100	2.9	100	1	97	55	100	23
osc_adapt	100	3.7	100	1	100	49	100	2
burst	100	10.7	100	6	100	24	0	—
resonator	67	56	100	1	100	50	100	8
gauss	33	139	100	2	13	152	43	155

Table 9: Cross-problem comparison for representative functions. The three-tier solvability structure replicates on problems requiring non-monotonic decision surfaces (Parity-4, Circles) and vanishes on smooth problems (Sine, Step). All cells

N{=}30

. % = solve rate; Gen = median generations to solve.

Problem	Act.	$h$	Solve%	Avg Epoch
Parity-4	sin	16	100	257
Parity-4	tanh	16	100	182
Parity-5	sin	16	100	477
Parity-5	sin	32	100	543
Parity-5	tanh	16	100	245
Parity-6	sin	16	66.7	2,653

Table 10: Gradient-trained MLP baseline (Adam,

lr{=}0.01

, 5,000 epochs max,

N{=}30

). The divide progressively reverses: tanh maintains 100% through Parity-8; sin degrades to 0%. Increasing

h

does not help sin; it makes it worse on Parity-6 (66.7%

\rightarrow

46.7%).

Neural Network Activation Functions

Neural network activation functions shape how models represent nonlinear patterns, learn, remain stable, and map efficiently to hardware.

See all 78 papers

Bio-Inspired Palette Evolution in Indirectly Encoded Substrates: Timescale Compatibility Shapes Activation Function Discovery
Sep 15, 2026Romain Claret, Michael O'Neill, Paul Cotofrei +1Evolutionary Optimization Neural Network Activation Functions
University of Neuchâtel, Neuchâtel, Switzerland · University College Dublin, Ireland
Multi-Behavioral Evolved Substrates Through Neuromodulation and Activation Selection
Sep 10, 2026Romain Claret, Michael O'Neill, Paul Cotofrei +1Neuroevolution Artificial Life
University of Neuchˆatel, Switzerland · University College Dublin, Ireland
Monotone but Exciting: On Evolving Monotone Boolean Functions with High Nonlinearity
Apr 19, 2026Claude Carlet, Marko Čupić, Marko Ðurasevic +3Evolutionary Optimization
University of Paris 8, 2 rue de la liberté, 93526 Saint-Denis Cedex, France · University of Bergen, Bergen, Norway · Faculty of Electrical Engineering and Computing, University of Zagreb, Unska 3, Zagreb, Croatia +2

Sep 15, 2026J/K move · Enter open · S save

Bio-Inspired Palette Evolution in Indirectly Encoded Substrates: Timescale Compatibility Shapes Activation Function Discovery

Romain Claret, Michael O'Neill, Paul Cotofrei, Kilian Stoffel

University of Neuchâtel, Neuchâtel, Switzerland · University College Dublin, Ireland

Evolutionary Optimization Neural Network Activation Functions Neuroevolution

Open paper AskPDF arXiv

Indirectly encoded neural networks can assign different activation functions to individual nodes, but the right functions are rarely known in advance. When the available set contains only standard monotonic functions, problems like parity become unsolvable, yet an all-inclusive palette underperforms a curated one. How should evolution discover which functions to use? We address this as a meta-learning problem, designing 13 strategies (11 inspired by biological adaptation mechanisms, plus baseline and oracle controls) that modify the set of available activation functions during evolution. Each strategy translates a biological principle into an evolutionary operator: for example, circadian-inspired oscillatory gating cycles functions in and out of the palette on a fixed schedule, while immune-inspired Clonal Selection permanently protects functions that consistently correlate with fitness. We evaluate all strategies across more than 3,000 runs on parity and non-parity problems, first evolving the activation palette alone, then co-evolving a per-node aggregation palette on harder problems; an independent replication with new seeds confirms a stable high-reliability tier, with Circadian holding its top rank. Bio-inspired strategies match the solve rate of a tuned baseline but converge up to twice as fast, with Circadian halving total compute. Strategy rankings reverse across problem types, with no strategy dominating all domains. Strategy success is largely shaped by timescale compatibility: strategies whose characteristic timescale matches the evolutionary evaluation window consistently outperform those that operate too slowly. The practical guideline: match the mechanism's timescale to the evaluation budget. Rescaling the slowest strategy bypasses the oscillatory barrier entirely: all nine solutions solve parity with non-oscillatory activations paired with min or max aggregation.

Abstract

Neural Network Activation Functions

Explore similar work

Bio-Inspired Palette Evolution in Indirectly Encoded Substrates: Timescale Compatibility Shapes Activation Function Discovery

Multi-Behavioral Evolved Substrates Through Neuromodulation and Activation Selection

Monotone but Exciting: On Evolving Monotone Boolean Functions with High Nonlinearity