SpikeCredit: Temporal Credit Carrier for Reinforcement Learning with Sparse Rewards

Yingchao Yu; Pengfei Sun; Wenxuan Pan; Wei Chen; Yitian Hong; Kuangrong Hao; Yaochu Jin

cs.LGSep 28, 2026

SpikeCredit: Temporal Credit Carrier for Reinforcement Learning with Sparse Rewards

Authors: Yingchao Yu, Pengfei Sun, Wenxuan Pan, Wei Chen, Yitian Hong, Kuangrong Hao, Yaochu Jin

Organizations: School of Information and Intelligent Science, Donghua University, Shanghai, China · School of Engineering, Westlake University, Hangzhou, China · Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom · School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China

Abstract

Reinforcement learning (RL) with sparse rewards is challenging because delayed outcomes provide little guidance about which intermediate computations caused success or failure. We argue that reliable credit assignment requires policy dynamics that preserve and expose credit-relevant information over time, a role we formalize as Temporal Credit Carriers (TCCs) and that spiking neural networks (SNNs) naturally fulfill through graded membrane traces and event-driven spikes. Based on this hypothesis, we propose SpikeCredit, an SNN-based framework for RL with sparse rewards that first performs task-adaptive TCC selection and then closes the loop between a fast TCC-reading pathway, where self-motion feedback constraint uses local behavior-grounded cues to constrain transition-level credit recovery, and a slow TCC-writing pathway, where credit-targeted trace alignment feeds recovered credit back into the actor to make future TCC dynamics more credit-readable. Across sparse-reward MuJoCo tasks, SpikeCredit improves Last10 return over sparse SNN baselines by +1169% on Ant, +953% on Hopper, +723% on Swimmer, and +1781% on Walker2d, and exceeds the dense-reward baseline on Swimmer by +113%. Mechanistic analyses further show substantially stronger alignment with dense rewards than the sparse SNN baseline. These results position spiking dynamics as credit-preserving substrates for sparse-reward RL.

Figures & tables

Figure 1: Motivation and core idea of SpikeCredit. Inspired by biological fast-slow learning, SpikeCredit treats spiking actor dynamics as task-adaptive temporal credit carriers (TCCs): the fast pathway reads credit from current dynamics, while the slow pathway writes recovered targets to reshape future dynamics, forming a closed read-write loop.

Figure 2: Overview of SpikeCredit. Task statistics select membrane- or spike-based TCCs. The fast pathway uses Self-Motion Feedback Constraint (SMF) to read credit from the selected dynamics and generate reshaped rewards and replay targets, while the slow pathway uses Credit-Targeted Trace Alignment (CTT) to write these targets back into the actor, progressively improving future TCC readability.

\hat{r}_{t}=R\hat{S}_{t}^{\mathrm{TCC},+},\qquad y_{t}=\log(T\hat{S}_{t}^{\mathrm{TCC},+}).

Algorithm 1 Closed-Loop Optimization in SpikeCredit

Figure 3: Learning curves of SpikeCredit, sparse-reward SNN and ANN baselines on four MuJoCo tasks. Lines and shaded regions denote the mean and standard deviation over five seeds, respectively.

Figure 4: Overall performance on four MuJoCo tasks. (a) Last10 returns against the sparse SNN baseline. (b) Learning curves against the dense SNN upper bound. (c) Comparisons with three alternative credit-assignment methods. Mean

\pm

standard deviation over five seeds.

Figure 5: Task-adaptive TCC selection. Left: task event scores determine the carrier. Right: Last10 returns verify that the selected candidate performs best on each task. Results are averaged over five seeds.

Figure 6: Alignment between credit signals and dense rewards on Ant-v4. Left: transition-wise correlations. Right: temporal profiles over an episode. Dense rewards are used only as a diagnostic reference.

Figure 7: Task-dependent SMF proxy attribution. Top: feature importance over $s_{t}$ , $\Delta s_{t}$ , and $\|a_{t}\|_{2}^{2}$ . Bottom: attribution aggregated by physical feature groups. Semantic groupings of observation fields used for self-motion proxy attribution are provided in the Appendix.

Methods	Last10 Return	Peak Return	Peak Step
SpikeCredit	$1669.9\pm 402.8$	$1777.5\pm 466.8$	995K
w/o $s_{t}$	$544.4\pm 389.7$	$738.1\pm 91.5$	20K
w/o $\Delta s_{t}$	$1106.3\pm 470.5$	$1212.2\pm 490.5$	985K
w/o $\\|a_{t}\\|_{2}^{2}$	$845.8\pm 672.7$	$939.5\pm 654.9$	980K
w/o SMF	$192.7\pm 78.6$	$738.1\pm 91.5$	20K
SpikeCredit	$1669.9\pm 402.8$	$1777.5\pm 466.8$	995K

Table 1: Ablation of SMF inputs and CTT on Ant-v4. Results are averaged over five seeds.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Rendered views of the four MuJoCo continuous-control environments used in our experiments: (a) Ant-v4, (b) Hopper-v4, (c) Swimmer-v4, and (d) Walker2d-v4.

Parameter	Value
$\alpha$	$0.5$
$V_{\mathrm{th}}$	$0.5$
$w$	$0.5$
$\theta_{v}$	$-0.172$
$\theta_{u}$	$0.529$
$\theta_{r}$	$0.021$

Appendix

Table 2: Dynamic neuron parameters used in all SNN experiments.

Component	Hyperparameter	Value
TD3 critic	Number of critics	$2$
TD3 critic	Input dimension	$d_{s}+d_{a}$
TD3 critic	Hidden layers	$256,256$
TD3 critic	Activation	ReLU
TD3 critic	Output dimension	$1$
SNN actor	Input / output dimension	$d_{s}\rightarrow d_{a}$

Appendix

Table 3: Actor-critic architecture hyperparameters.

Group	Hyperparameter	Value
Training	Random seed	$0,1,2,3,4$
	Batch size	$256$
	Replay buffer size	$10^{6}$
	Warm-up random-action steps	$25{,}000$
	Maximum training steps	$1{,}000{,}000$
	Evaluation interval	$5{,}000$ steps

Appendix

Table 4: RL and SpikeCredit hyperparameters used in the main experiments.

Abbreviation	Category	Meaning
Pose	Body / joint pose	Body height, orientation, joint angles, and foot angles
Fwd Vel.	Forward velocity	Translational velocity along the forward direction
Lat./Vert. Vel.	Lateral / vertical velocity	Sideways or vertical translational velocity
Body Ang. Vel.	Body angular velocity	Angular velocity of torso or body orientation
Joint Vel.	Joint velocity	Hip, thigh, leg, rotor, or non-foot joint velocity
Foot Vel.	Foot / ankle velocity	Foot or ankle joint velocity

Appendix

Table 5: Semantic proxy-feature categories and abbreviations used in the attribution analysis.

Task	Observation indices	Observation fields	Category
Ant-v4	0–12	torso_z; torso_quat_w/x/y/z; hip_1–4_angle; ankle_1–4_angle	Pose
	13	torso_x_velocity	Fwd Vel.
	14–15	torso_y_velocity; torso_z_velocity	Lat./Vert. Vel.
	16–18	torso_roll_velocity; torso_pitch_velocity; torso_yaw_velocity	Body Ang. Vel.
	19, 21, 23, 25	hip_1–4_velocity	Joint Vel.
	20, 22, 24, 26	ankle_1–4_velocity	Foot Vel.

Appendix

Table 6: Task-specific mapping from Gymnasium MuJoCo-v4 observation dimensions to semantic proxy-feature categories.

Figure 9: Hyperparameter sensitivity of SpikeCredit on Ant-v4. We vary

\lambda_{\mathrm{align}}

\lambda_{\mathrm{sparse}}

, and

\lambda_{\mathrm{CTT}}

while keeping other settings fixed, and report Last10 evaluation returns over five seeds. Dashed lines denote Sparse SNN and Mean Redistribution baselines. SpikeCredit consistently outperforms both baselines across all tested values, indicating that its gains are not tied to a narrow hyperparameter choice.

Explore similar work

Learning Process Rewards via Success Visitation Matching for Efficient RL
Jun 22, 2026Raymond Tsao, Andrew Wagenmaker, Sergey LevineOffline Reinforcement Learning Robotic Manipulation Policies
UC Berkeley
Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learning
Apr 22, 2026Aravind Venugopal, Jiayu Chen, Xudong Wu +3Potential-Based Reward Shaping Credit Assignment
Carnegie Mellon University · The University of Hong Kong · INFIFORCE Intelligent Technology +1
ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy
Jul 3, 2026Zijun Xie, Yuyang You, Yongzhi Li +6Credit Assignment
Peking University · Baidu Inc. · Kuaishou Inc.

Jun 22, 2026, cs.LGJ/K move · Enter open · S save

Learning Process Rewards via Success Visitation Matching for Efficient RL

Raymond Tsao, Andrew Wagenmaker, Sergey Levine

UC Berkeley

Offline Reinforcement Learning Robotic Manipulation Policies

Open paper AskPDF arXiv

In many modern applications of reinforcement learning (RL), the natural reward for a task of interest is inherently sparse: a reward of 0 is given everywhere except when the task is completed, when a reward of +1 is given. Training a policy to maximize such a sparse reward requires solving a challenging credit assignment problem, leading to slow or ineffective RL improvement. We propose a simple approach to transform a sparse outcome reward into a dense process reward. Our approach relies on training a discriminator to distinguish between previous successful and unsuccessful episodes, and using this discriminator to incentivize the RL-learned policy to match the state-action visitations of successful episodes, while avoiding those of unsuccessful episodes. By incentivizing the policy to match the visitations over all states, not just those that correspond to task success, this reward provides dense feedback on whether progress is being made towards task completion, and, we show, provably achieves this without changing the optimal policy. Focusing on finetuning of robotic control policies, we demonstrate that our approach leads to significantly faster RL finetuning performance on both simulated and real-world manipulation tasks, as compared to simply maximizing the sparse outcome reward.

Abstract

Explore similar work

Learning Process Rewards via Success Visitation Matching for Efficient RL

Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learning

ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy