cs.MAMar 16, 2026

S2Act: Simple Spiking Actor

Authors: Ugur Akcal, Seung Hyun Kim, Mikihisa Yuasa, Hamid Osooli, Jiarui Sun, Ribhav Sahu, Mattia Gazzola, Huy T. Tran, 

Organizations: Aerospace Engineering · Computer Science · Coordinated Science Laboratory · Mechanical Science and Engineering · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Spiking neural networks (SNNs) and biologically-inspired learning mechanisms are attractive in mobile robotics, where the size and performance of onboard neural network policies are constrained by power and computational budgets. Existing SNN approaches, such as population coding, reward modulation, and hybrid artificial neural network (ANN)-SNN architectures, have shown promising results; however, they face challenges in complex, highly stochastic environments due to SNN sensitivity to hyperparameters and inconsistent gradient signals. To address these challenges, we propose simple spiking actor (S2Act), a computationally lightweight framework that deploys an RL policy using an SNN in three steps: (1) architect an actor-critic model based on an approximated network of rate-based spiking neurons, (2) train the network with gradients using compatible activation functions, and (3) transfer the trained weights into physical parameters of rate-based leaky integrate-and-fire (LIF) neurons for inference and deployment. By globally shaping LIF neuron parameters such that their rate-based responses approximate ReLU activations, S2Act effectively mitigates the vanishing gradient problem, while pre-constraining LIF response curves reduces reliance on complex SNN-specific hyperparameter tuning. We demonstrate our method in two multi-agent stochastic environments (capture-the-flag and parking) that capture the complexity of multi-robot interactions, and deploy our trained policies on physical TurtleBot platforms using Intel's Loihi neuromorphic hardware. Our experimental results show that S2Act outperforms relevant baselines in task performance and real-time inference in nearly all considered scenarios, highlighting its potential for rapid prototyping and efficient real-world deployment of SNN-based RL policies.

Figures & tables

Fig. 1: S2Act. An ANN with rate-based LIF activation is used for training, with learned weights then converted to a spiking network for deployment on neuromorphic hardware.
Fig. 2: S2Act training-to-deployment pipeline. (a) We employ an ANN-to-SNN conversion strategy, enabling neuromorphic deployment of computationally lightweight SNN RL agents. In the rate-based LIF network training phase, the simulated environment provides discrete visual observations Ot{O_{t}} . These are passed to an actor-critic model, comprising a policy (actor) network and a value (critic) network, both modeled with soft-ReLLIF units (green circles). We use PPO to train these networks. (b) Once trained, the soft-ReLLIF neurons in the actor network are replaced with spiking LIF neurons (blue circles) for deployment on neuromorphic hardware, such as Intel’s Loihi.
Fig. 3: Rate-based LIF neuron variants. Rate-based LIF units (cyan line) exhibit unbounded gradients at the critical input value determined by their neuronal parameters. Although a smoothing term can be used to address these unbounded gradients, the resulting soft rate-based LIF unit (purple line with star markers) will still be prone to vanishing gradients. A common solution to this problem in networks with bounded activation functions is to employ ReLU activation. Therefore, we adjust the neuronal parameters of the soft rate-based LIF units to approximate the ReLU activation. Consequently, we utilize soft ReLLIF (green line) activations.
Fig. 4: Simulated and real-world environment for evaluations. (a) Visualization of our simulated 2 vs. 2 CtF game. Gray squares are obstacles, triangles are agents, and circles are flags. The region highlighted by solid red lines is the border region for a red agent whose policy is the patrol policy. The game has stochastic combat: when two agents are adjacent, whether one is eliminated depends on the territory, the number of nearby enemies, and the number of nearby allies. (b) Real-world CtF arena measures 12’ ×\times 12’ and replicates the 10 ×\times 10 grid-world used during policy training. Blue, red, and gray tiles denote the blue team’s territory, the red team’s territory, and obstacles, respectively. (c) A Qualisys motion capture system with ten Miqus M5 cameras and Qualisys Track Manager software is used to track robot positions in real time. (d) Visualization of the parking environment. The green rectangle is the ego vehicle, and the blue square is the target parking spot.
Fig. 5: Evaluation results. Average training curves of S2Act and other baselines in two CtF scenarios (a, b) and the parking environment (c). Solid lines represent the average mean episodic reward over three seeds with shaded areas representing one standard deviation confidence intervals.
Fig. 6: Episode trajectory. (a) Multi-robot demonstration of the on-chip deployment in the CtF task. (b) Each TurtleBot3 Burger robot is fitted with four sparsely placed marker balls to enable robust pose tracking. Communication among the components is handled via a local wireless network running ROS 2 Humble [ 39 ] . The evaluations are conducted in a 2v2 scenario, where the blue agents engage with defensive red team behaviors. (c) A ground station communicates the on-chip S2Act centralized policy’s commands to the blue team, while red team policies run individually on the ground station. (d-e) We overlay the trajectories executed by robots controlled by the S2Act policy network. A policy first moves agents to the nearest territorial boundary from the target flag, after which the agents coordinate to capture the flag. A supplementary video is provided.
EnvironmentMethodHardwareInference statisticsTerminal conditions
EpIMInTMTrTFlag CaptureDefeatedLost FlagTime-up
[J / inf][ms][hours](%)(%)(%)(%)
1v1 CtFSimulated S2ActGPU0.0796.72.478.120.80.40.7
S2ActLoihi0.03327-72.626.40.40.6
PopSANGPU1.042141.68.367.724.20.37.8
RSNNCPU0.46857.812.14.217.30.178.4
TABLE I: Performance measures. Inference statistics and rate of successful termination.
More in this topic

Spiking Neural Networks

Spiking neural networks process information through discrete, time-dependent neural events, aiming for efficient computation and biologically inspired learning.

See all 336 papers

Explore similar work

CardsList
Dec 6, 2024

Enabling Energy-Efficient Simultaneous Multi-Task Reinforcement Learning through Spiking Neural Networks with Active Dendrites for Bio-inspired Generalist Agents

Reinforcement learning (RL) has demonstrated remarkable capabilities in training agents to solve complex tasks autonomously, such as mobile robots, UAVs/UGVs, and game-playing agents). However, scaling RL to master multiple tasks simultaneously (i.e., so-called multi-task RL) remains a significant challenge. Such a multi-task RL capability especially is important for agents to adapt to changes in real-world operational environments. State-of-the-art works show that, training agents with neural networks and shared structures across tasks promises improved generalization in simultaneous multi-task RL. However, they still suffer from task interference and incur high energy consumption due to intensive computation. To address this, we propose MTSpark, a novel methodology that enables energy-efficient simultaneous multi-task RL using spiking neural networks (SNNs) equipped with active dendrites for bio-inspired generalist agents. Specifically, MTSpark enhances a Deep Spiking Q-Network (DSQN) with active dendrites, a dueling structure, and task-specific context signals to dynamically form specialized sub-networks for individual tasks, while exploiting sparse operations for energy-efficient network processing. Experimental results demonstrate that MTSpark achieves higher performance and efficiency compared to state-of-the-art by obtaining high scores across three Atari games (i.e., Pong: -5.4, Breakout: 0.6, and Enduro: 371.2), approaching human-level performance (i.e., Pong: -3, Breakout: 31, Enduro: 368), while incurring similar memory and about 2x lower energy than state-of-the-art. These results show that our MTSpark potentially advances the frontiers toward energy-efficient generalist agents by combining RL and SNNs.

Rachmad Vidya Wicaksana Putra, Avaneesh Devkota, Muhammad Shafique

eBRAIN Lab · New York University (NYU) Abu Dhabi

Reinforcement LearningDeep Q-LearningSpiking Neural NetworksMulti-Task RL
AskPDF
Aug 24, 2026

Spiking Neural Networks for Continuous Control: Neuromorphic Reinforcement Learning in Conventional Computing

Reinforcement learning (RL) algorithms have made strides over the past decade applying them to a wide range of problems and control tasks. However, the deployment of RL on neuromorphic hardware for continuous control tasks remains under-validated. Namely it is unclear whether replacing a conventional actor network with a spiking neural network (SNN) affects the performance of an agent before any hardware-specific benefits manifest. We provide a systematic validation of a minimal, neuromorphically viable spiking actor variant of Soft Actor-Critic (SAC) on conventional hardware, establishing a baseline for future neuromorphic RL research. In this paper, we propose the Spiking Actor Network Soft Actor Critic (SANSAC) to address the use of RL frameworks in continuous environments, designed as a framework that can be implemented on neuromorphic hardware. We compare a traditional Soft Actor Critic (SAC) network to SANSAC in a traditional computer. We demonstrate the near equivalent performance of SANSAC and SAC, while addressing the impact of hidden dimensions. Our results demonstrate the viability of SNN based algorithms in complex continuous environments, as well as competitive performance to traditional neural networks in traditional computers, providing a basis to continue exploring the use of SNNs in continuous RL frameworks.

Jessica Hunter, Md Maruf Hossain Shuvo, Krishna Roy

Department of Computer Science New Mexico Institute of Mining and Technology Socorro, NM 87801, USA · Department of Electrical and Computer Engineering The University of Texas at El Paso El Paso, TX 79968, USA · Department of Electrical Engineering New Mexico Institute of Mining and Technology Socorro, NM 87801, USA

Reinforcement LearningSpiking Neural NetworksNeuromorphic ComputingSoft Actor-CriticActor-Critic Methods
AskPDF
Sep 23, 2026

Spiking Neural Network Predicting Sequence of the External Worlds States in Model-Based Reinforcement Learning

This paper presents a spiking neural network (SNN) designed to predict the sequence of the external world states starting from the current world state. This SNN does not create the world dynamics model - instead it incorporates the SNN trained to predict the next world state and provides all mechanisms necessary to make the chain of predicted world states. These mechanisms are entirely spiking - they are implemented as spiking neuron ensembles. The present article describes this neuronal structure and tests its operation on a classic RL benchmark - ATARI ping-pong.

Mikhail Kiselev

Chuvash State University, Cheboxary, Russia

World ModelsSpiking Neural NetworksModel-Based RL
AskPDF