Spiking Neural Networks for Continuous Control: Neuromorphic Reinforcement Learning in Conventional Computing

Jessica Hunter; Md Maruf Hossain Shuvo; Krishna Roy

cs.LGAug 24, 2026

Spiking Neural Networks for Continuous Control: Neuromorphic Reinforcement Learning in Conventional Computing

Authors: Jessica Hunter, Md Maruf Hossain Shuvo, Krishna Roy

Organizations: Department of Computer Science New Mexico Institute of Mining and Technology Socorro, NM 87801, USA · Department of Electrical and Computer Engineering The University of Texas at El Paso El Paso, TX 79968, USA · Department of Electrical Engineering New Mexico Institute of Mining and Technology Socorro, NM 87801, USA

Abstract

Reinforcement learning (RL) algorithms have made strides over the past decade applying them to a wide range of problems and control tasks. However, the deployment of RL on neuromorphic hardware for continuous control tasks remains under-validated. Namely it is unclear whether replacing a conventional actor network with a spiking neural network (SNN) affects the performance of an agent before any hardware-specific benefits manifest. We provide a systematic validation of a minimal, neuromorphically viable spiking actor variant of Soft Actor-Critic (SAC) on conventional hardware, establishing a baseline for future neuromorphic RL research. In this paper, we propose the Spiking Actor Network Soft Actor Critic (SANSAC) to address the use of RL frameworks in continuous environments, designed as a framework that can be implemented on neuromorphic hardware. We compare a traditional Soft Actor Critic (SAC) network to SANSAC in a traditional computer. We demonstrate the near equivalent performance of SANSAC and SAC, while addressing the impact of hidden dimensions. Our results demonstrate the viability of SNN based algorithms in complex continuous environments, as well as competitive performance to traditional neural networks in traditional computers, providing a basis to continue exploring the use of SNNs in continuous RL frameworks.

Figures & tables

Figure 1: SANSAC/SAC Algorithm Diagram. Dashed border indicates spiking actor in SANSAC. All other components are identical between both algorithms. Critic outputs a state-action value

Q(s,a)

. Actor outputs a policy

\pi(a|s)

	Hidden Dimension 1	Hidden Dimension 2
(1)	$256$	$256$
(2)	$200$	$200$
(3)	$128$	$128$
(4)	$64$	$64$

Table 1: Hidden Dimensions

Panel of Figure 2 — Figure 2: Gymnasium Bipedal Walker: Task is considered solved when 300 points are accumulated in 1600 steps

Dimensions	Mean Reward $\pm$ SD		Success Rate		Statistical Tests
	SANSAC	SAC	SANSAC	SAC	Mann-Whitney P	Cohen’s d
( 1 )	$273.9\pm 130.8$	$224.5\pm 162.9$	70%	80%	0.7913	0.080
( 1 )	$197.3\pm 178.2$	$259.8\pm 146.2$	60%	80%	0.6232	-0.140
( 1 )	$210.9\pm 185.9$	$158.7\pm 213.5$	70%	60%	0.6776	0.120
( 1 )	$26.8\pm 149.4$	$41.8\pm 138.5$	10%	10%	0.5205	-0.180

Table 2: Performance comparison between SAC and SANSAC. Success Rate indicates percentage of agents achieving reward

\geq 300

in tests. Statistical tests show no significant differences between algorithms across all dimensions (Mann-Whitney p

>0.05

). Cohen’s d effect sizes:

|d|<0.2

indicates negligible difference,

0.2\leq|d|<0.5

small,

0.5\leq|d|<0.8

medium. All configurations show negligible to small effect sizes confirming the practical equivalence.

Figure 3: Box plots demonstrating the average reward of agents in testing. Configurations are labeled to separate SANSAC from SAC in each dimension test.

Panel of Figure 4 — Figure 4: Reward progression for SANSAC and SAC across different hidden dimensions. Curves show the mean episodic reward across 10 random seeds. Shaded regions represent the $\pm 1$ standard deviation across seeds. Training was terminated after 1200 episodes or no reward improvement in the last 100 consecutive episodes. Truncated curves indicate early termination either through early convergence or early stopping, not a difference in evaluation protocol.

Panel of Figure 5 — Figure 5: Actor Loss across steps for agent training averaged across 10 runs of random seeds. Curves show the mean episodic loss across 10 random seeds. Shaded regions indicate $\pm 1$ standard deviation across seeds. The inverse relationship between loss and reward is expected as lower actor loss corresponds to higher expected returns.

	Training Time (minutes)
Hidden Dimension	SANSAC	SAC
( 1 )	44	10
( 1 )	66	26
( 1 )	80	23
( 1 )	54	24
Mean	61	21

Table 3: Average time taken by each configuration to train the agent across 10 random seeds.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Parameter		Value
	learning rate	$3\cdot 10^{-4}$
	discount ( $\gamma$ )	0.975
	replay buffer size	$10^{6}$
	number of samples per minibatch	256
	target smoothing coefficient ( $\tau$ )	0.005
	Reward Scale	5

Appendix

Table A.1: Hyperparameters

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,668 papers

S2Act: Simple Spiking Actor
Mar 16, 2026Ugur Akcal, Seung Hyun Kim, Mikihisa Yuasa +6Spiking Neural Networks Robotic RL
Aerospace Engineering · Computer Science · Coordinated Science Laboratory +4
From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments
Jun 2, 2026Saket Tiwari, Tejas Kotwal, George KonidarisRL Control Reinforcement Learning
Department of Computer Science Brown University · Department of Applied Mathematics Brown University
All On-Board: Fully On-Chip Neuromorphic Q-Learning with Embedded CartPole Simulation
Sep 26, 2026Steven C. Nesbit, Giovanni T. Michel, Gerd J. Kunde +2Neuromorphic Computing Q-Learning
College of Computing and Informatics, Drexel University, Philadelphia, PA, USA · Information Sciences (CAI-3), Los Alamos National Laboratory, Los Alamos, NM, USA · Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL, USA +1

Mar 16, 2026J/K move · Enter open · S save

S2Act: Simple Spiking Actor

Ugur Akcal, Seung Hyun Kim, Mikihisa Yuasa, Hamid Osooli+5

Aerospace Engineering · Computer Science · Coordinated Science Laboratory+4

Spiking Neural Networks Robotic RL Reinforcement Learning Neuromorphic Computing

Open paper AskPDF arXiv

Spiking neural networks (SNNs) and biologically-inspired learning mechanisms are attractive in mobile robotics, where the size and performance of onboard neural network policies are constrained by power and computational budgets. Existing SNN approaches, such as population coding, reward modulation, and hybrid artificial neural network (ANN)-SNN architectures, have shown promising results; however, they face challenges in complex, highly stochastic environments due to SNN sensitivity to hyperparameters and inconsistent gradient signals. To address these challenges, we propose simple spiking actor (S2Act), a computationally lightweight framework that deploys an RL policy using an SNN in three steps: (1) architect an actor-critic model based on an approximated network of rate-based spiking neurons, (2) train the network with gradients using compatible activation functions, and (3) transfer the trained weights into physical parameters of rate-based leaky integrate-and-fire (LIF) neurons for inference and deployment. By globally shaping LIF neuron parameters such that their rate-based responses approximate ReLU activations, S2Act effectively mitigates the vanishing gradient problem, while pre-constraining LIF response curves reduces reliance on complex SNN-specific hyperparameter tuning. We demonstrate our method in two multi-agent stochastic environments (capture-the-flag and parking) that capture the complexity of multi-robot interactions, and deploy our trained policies on physical TurtleBot platforms using Intel's Loihi neuromorphic hardware. Our experimental results show that S2Act outperforms relevant baselines in task performance and real-time inference in nearly all considered scenarios, highlighting its potential for rapid prototyping and efficient real-world deployment of SNN-based RL policies.

Abstract

Reinforcement Learning

Explore similar work

S2Act: Simple Spiking Actor

From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments

All On-Board: Fully On-Chip Neuromorphic Q-Learning with Embedded CartPole Simulation