Common-Mode Errors Limit Low-Timestep Deep Spiking Q-Networks

Zijie Xu; Bingrui Guo; Yiding Sun; Yiting Dong; Zhile Yang; Zhaofei Yu

cs.NEOct 6, 2026

Common-Mode Errors Limit Low-Timestep Deep Spiking Q-Networks

Authors: Zijie Xu, Bingrui Guo, Yiding Sun, Yiting Dong, Zhile Yang, Zhaofei Yu

Organizations: Peking University Beijing, 100871, China

Abstract

Spiking neural networks (SNNs) offer sparse and event-driven computation, making them attractive for energy-constrained reinforcement learning (RL) on edge devices. In value-based RL, deep spiking Q-networks (DSQNs) combine such efficiency with action-value estimation for decision making. However, existing DSQNs often require multiple simulation timesteps for competitive performance, increasing computational and energy costs, whereas reducing the timesteps can cause substantial performance degradation. We investigate this degradation from the perspective of Q-value estimation errors. By decomposing errors across actions into common-mode and differential-mode components, we find that low-timestep DSQNs suffer disproportionately from common-mode errors shared across action values, which are particularly detrimental to temporal-difference learning through bootstrapped targets. Based on this finding, we propose Common-Mode Compensation Deep Spiking Q-Network (CMC-DSQN), which uses an auxiliary ANN to compensate for common-mode errors in the SNN outputs. At inference, greedy action selection can be performed directly from the SNN outputs, allowing the auxiliary ANN to be completely removed and preserving the energy efficiency of SNNs. Extensive experiments on Atari and MiniAtar environments demonstrate substantial performance improvements under low-timestep settings. CMC-DSQN outperforms state-of-the-art DSQN baselines by nearly $20\%$ at $T=2$ and further surpasses the ANN baseline at $T=4$ .

Figures & tables

Figure 1: Comparison of existing directly trained DSQNs on the MiniAtar environments ( Young and Tian, 2019 ) , including DSQN ( Liu et al., 2022 ) , pbLN ( Sun et al., 2022 ) , DATSQN ( Ghoreishee et al., 2026 ) , CaRe-BN ( Xu et al., 2026c ) , and TP-DSQN ( Chowdhury et al., 2022 ) . Performance is averaged across all environments and normalized by the ANN.

Figure 2: Analysis on deterministic CliffWalking. (a) Environment. (b,c) Greedy trajectories of the ANN and low-timestep SNN, respectively, aggregated over 20 random seeds. Yellow segments denote trajectories, and red crosses mark states leading to infinite loops. (d,e) State-wise common-mode error ratios of the ANN and SNN with respect to the optimal action-value function $Q^{*}$ .

Figure 3: Learning dynamics and Q-value estimation errors on CliffWalking environment. (a) Fraction of successful seeds at each evaluation point. (b) Cumulative fraction of seeds that have solved the task at least once. Curves in (a) and (b) are smoothed for visual clarity. (c) Common- and differential-mode RMSEs with respect to the exact $Q^{\pi}$ of each learned deterministic greedy policy.

Figure 4: Learning dynamics and Q-value estimation errors on CliffWalking. (a) Fraction of successful seeds at each evaluation point. (b) Cumulative fraction of seeds that have solved the task at least once. Curves in (a) and (b) are smoothed for visual clarity. (c) Common- and differential-mode RMSEs with respect to the exact optimal Q-function

Q^{*}

. Unlike Fig. 3 (c), which uses the exact

Q^{\pi}

of each learned policy as the reference, this panel evaluates all methods against

Q^{*}

Timesteps	Method	BeamRider	Breakout	Enduro	Pong	Q*bert	Seaquest	SpaceInvaders	APR
–	Random	354.64	0.97	0.00	-20.26	162.08	60.20	127.83	0.00%
–	DQN(ANN)	6842.99	364.51	295.79	19.86	9224.17	1960.44	1492.50	100.00%
$T=2$	DSQN	4279.90	276.11	280.97	17.60	1300.21	2101.49	806.36	70.75%
$T=2$	CMC-DSQN	4136.44	382.90	316.87	18.34	4186.33	2358.57	1183.40	87.06%
$T=3$	DSQN	4705.58	292.41	273.83	17.23	4076.25	2863.83	935.96	83.32%
$T=3$	CMC-DSQN	5284.37	398.73	313.55	19.59	8782.83	1766.93	1401.10	95.57%

Table 1: Average maximum return over three random seeds and average performance ratio (APR) on Atari environments after 30M environment transitions.

Timesteps	methods	Asterix	Breakout	Freeway	Seaquest	SpaceInvaders	APR
—	Random	$0.493\pm 0.902$	$0.366\pm 0.618$	$0.370\pm 0.586$	$0.116\pm 0.427$	$4.057\pm 3.199$	0.00%
—	DQN (ANN)	$32.22\pm 4.94$	$27.76\pm 4.98$	$62.58\pm 0.34$	$49.70\pm 8.15$	$138.34\pm 3.87$	100.00%
$T=2$	DSQN-CVT	$1.89\pm 1.14$	$4.11\pm 2.40$	$19.17\pm 17.72$	$1.36\pm 0.75$	$3.82\pm 3.77$	10.12%
	DCGS	$18.36\pm 3.55$	$11.64\pm 2.21$	$59.22\pm 0.77$	$26.17\pm 2.84$	$60.56\pm 20.89$	57.33%
	CRPI	$21.02\pm 2.36$	$14.92\pm 0.89$	$59.70\pm 0.64$	$28.56\pm 2.44$	$82.02\pm 21.38$	65.72%
	DSQN	$14.50\pm 3.04$	$23.90\pm 1.22$	$56.92\pm 1.06$	$22.84\pm 15.67$	$100.76\pm 11.51$	67.76%

Table 2: Average maximum return over five random seeds and average performance ratio (APR) on MiniAtar environments after 5M environment transitions, where

\pm

denotes one standard deviation.

Figure 5: Average normalized return across MiniAtar environments. (a)–(c) Comparison of different DSQN variants at

T=2,4,6

, respectively. (d) Comparison of the vanilla ANN DQN and its common-mode compensated variant. All curves are smoothed uniformly for visual clarity.

Explore similar work

Latency and accuracy tradeoffs in Spiking Neural Networks
Sep 28, 2026Zhanglu Yan, Zixuan Zhu, Kaiwen Tang +3Spiking Neural Networks Time-To-First-Spike
Department of computer science, National University of Singapore · Shanghai Advanced Research Institute, Chinese Academy of Science · University of Chinese Academy of Sciences, Beijing +1
Scalable Training of Continuous-Time Spiking Neural Networks with Differentiable Spike-Time Discretization
Jul 16, 2026Yusuke Sakemi, Tomoya Takeuchi, Takeo Hosomi +1Spiking Neural Networks Time-To-First-Spike
Research Center for Mathematical Engineering, Chiba Institute of Technology, Narashino, Japan · International Research Center for Neurointelligence (WPI-IRCN), The University of Tokyo, Tokyo, Japan · NEC Corporation, Kawasaki, Japan
QB-LIF: Learnable-Scale Quantized Burst Neurons for Efficient SNNs
Apr 28, 2026Dewei Bai, Hongxiang Peng, Jiajun Mei +4Spiking Neural Networks Time-To-First-Spike

Sep 28, 2026, cs.LGJ/K move · Enter open · S save

Latency and accuracy tradeoffs in Spiking Neural Networks

Zhanglu Yan, Zixuan Zhu, Kaiwen Tang, Yuyang Cai+2

Department of computer science, National University of Singapore · Shanghai Advanced Research Institute, Chinese Academy of Science · University of Chinese Academy of Sciences, Beijing+1

Spiking Neural Networks Time-To-First-Spike Neural Network Trade-Off

Open paper AskPDF arXiv

Spiking neural networks are attractive for low-power speech command recognition, yet their latency has received far less attention than their energy efficiency, and their multi-timestep execution is widely assumed to make them slower than quantized neural networks. This paper challenges the assumption that more local timesteps necessarily imply higher network latency. By overlapping computation across adjacent layers at the timestep level, SNNs may complete execution in less time than comparable bit-serial QNNs. However, this overlap relies on spikes firing on incomplete inputs, and a spike once generated cannot be withdrawn, so its error persists and reduces accuracy. Waiting for more input before firing would seem to improve accuracy at the cost of reduced overlap. Yet we find and prove that this intuition fails at some layers, where even a small increase in waiting can change spike timing and downstream computation, making the network both slower and less accurate. We therefore propose a Pipeline Delay Search method which selects each layer's delay by balancing task-level accuracy gains against added network latency. We then adapt the selected configurations through spike-based quantization-aware training and bounded tuning of firing thresholds and initial membrane potentials. Together, these steps form Falcon, a framework for Fine-grained Analysis of Latency and Controlled firing which systematically analyzes and optimizes SNN latency under a spatial analog compute-in-memory mapping with shared digital engines. We evaluate Falcon on GSCV2 and SSC, achieving competitive accuracies of 96.31 and 83.02 at modeled network-core latencies of 119.64 and 124.00us, respectively. Together, our analysis and results show that SNNs can compute more yet finish faster, and wait longer yet predict worse, highlighting why Falcon matters for both latency and accuracy.

Abstract

Explore similar work

Latency and accuracy tradeoffs in Spiking Neural Networks

Scalable Training of Continuous-Time Spiking Neural Networks with Differentiable Spike-Time Discretization

QB-LIF: Learnable-Scale Quantized Burst Neurons for Efficient SNNs