CDRL: A Reinforcement Learning Framework Inspired by Cerebellar Circuits and Dendritic Computational Strategies

Sibo Zhang; Rui Jing; Liangfu Lv; Jian Zhang; Yunliang Zang

cs.LGFeb 17, 2026

CDRL: A Reinforcement Learning Framework Inspired by Cerebellar Circuits and Dendritic Computational Strategies

Authors: Sibo Zhang, Rui Jing, Liangfu Lv, Jian Zhang, Yunliang Zang

Organizations: Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, China.

Abstract

Reinforcement learning (RL) has achieved notable performance in high-dimensional sequential decision-making tasks, yet remains limited by low sample efficiency, sensitivity to noise, and weak generalization under partial observability. Most existing approaches address these issues primarily through optimization strategies, while the role of architectural priors in shaping representation learning and decision dynamics is less explored. Inspired by structural principles of the cerebellum, we propose a biologically grounded RL architecture that incorporate large expansion, sparse connectivity, sparse activation, and dendritic-level modulation. Experiments on noisy, high-dimensional RL benchmarks show that both the cerebellar architecture and dendritic modulation consistently improve sample efficiency, robustness, and generalization compared to conventional designs. Sensitivity analysis of architectural parameters suggests that cerebellum-inspired structures can offer optimized performance for RL with constrained model parameters. Overall, our work underscores the value of cerebellar structural priors as effective inductive biases for RL.

Figures & tables

Figure 1: Architecture of the RL framework inspired by cerebellar circuits and dendritic computational strategies.

Figure 2: Self-built Pong game environment.

Param.	Ball Spd. (H)	Ball Spd. (V)	Paddle Len.	Paddle Spd.
Train	12	8	80	5
Test 1	15	10	80	5
Test 2	18	12	80	5
Test 3	12	8	60	5
Test 4	12	8	20	5
Test 5	12	8	80	2

Table 1: Environment parameters used for evaluating model generalization.

Figure 3: Sample efficiency of different models during training, measured by performance versus number of training steps.

Figure 4: Heat map of the difference in robustness between the CDRL model and the baseline model.

Model	Baseline	CDRL-dendrite	CDRL
win rate (%)	91.20 $\pm$ 3.54	97.20 $\pm$ 1.47	97.60 $\pm$ 1.85
reward	8.38 $\pm$ 0.85	9.71 $\pm$ 0.50	10.07 $\pm$ 1.97

Table 2: Performance under sticky action noise (

p=0.25

Figure 5: Generalization performance of different models under unseen environment configurations.

Figure 6: Sensitivity results of the expansion ratio. (a) Observation noise. (b) Action noise.

Figure 7: Sensitivity results of the sparse connectivity. (a) Observation noise. (b) Action noise.

Figure 8: Sensitivity results of the sparse activation. (a) Observation noise. (b) Action noise.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value	Parameter	Value	Parameter	Value
Ball X speed	12	Paddle width	10	Ball Y speed	8
Paddle height	80	Paddle speed	5	FPS	1920
Max score	21	Action noise	0.0	Stack size	4

Appendix

Table 3: Custom Pong environment parameters.

Parameter	Value	Parameter	Value
Model type	Baseline / CDRL	GPU ID	0 / 1
Input channels	4	Batch size	64
Num episodes	1500	Gamma	0.99
Epsilon start	1.0	Epsilon end	0.01
Epsilon decay	200000	Learning rate	5e-07
Memory size	100000	Dendritic gate	Enabled / Disabled

Appendix

Table 4: Training parameters.

Obs / Act	0.00	0.05	0.10	0.15	0.20	0.30
0.00	0.92 $\pm$ 0.04 0.99 $\pm$ 0.01	0.93 $\pm$ 0.04 0.98 $\pm$ 0.01	0.90 $\pm$ 0.04 0.97 $\pm$ 0.02	0.87 $\pm$ 0.07 0.95 $\pm$ 0.03	0.84 $\pm$ 0.06 0.93 $\pm$ 0.02	0.73 $\pm$ 0.13 0.86 $\pm$ 0.04
1.00	0.88 $\pm$ 0.11 0.98 $\pm$ 0.01	0.83 $\pm$ 0.16 0.96 $\pm$ 0.01	0.84 $\pm$ 0.11 0.96 $\pm$ 0.03	0.80 $\pm$ 0.14 0.94 $\pm$ 0.03	0.79 $\pm$ 0.15 0.89 $\pm$ 0.06	0.62 $\pm$ 0.09 0.79 $\pm$ 0.08
2.00	0.58 $\pm$ 0.31 0.84 $\pm$ 0.22	0.54 $\pm$ 0.29 0.81 $\pm$ 0.25	0.54 $\pm$ 0.31 0.79 $\pm$ 0.25	0.51 $\pm$ 0.29 0.76 $\pm$ 0.25	0.52 $\pm$ 0.23 0.72 $\pm$ 0.22	0.42 $\pm$ 0.21 0.62 $\pm$ 0.20
3.00	0.50 $\pm$ 0.29 0.73 $\pm$ 0.26	0.48 $\pm$ 0.30 0.70 $\pm$ 0.27	0.45 $\pm$ 0.23 0.72 $\pm$ 0.24	0.46 $\pm$ 0.25 0.66 $\pm$ 0.25	0.40 $\pm$ 0.21 0.61 $\pm$ 0.23	0.40 $\pm$ 0.18 0.57 $\pm$ 0.21
5.00	0.30 $\pm$ 0.09 0.56 $\pm$ 0.25	0.33 $\pm$ 0.05 0.54 $\pm$ 0.23	0.30 $\pm$ 0.07 0.54 $\pm$ 0.22	0.28 $\pm$ 0.07 0.49 $\pm$ 0.21	0.32 $\pm$ 0.04 0.46 $\pm$ 0.23	0.29 $\pm$ 0.07 0.44 $\pm$ 0.20
10.00	0.27 $\pm$ 0.07 0.37 $\pm$ 0.22	0.31 $\pm$ 0.06 0.38 $\pm$ 0.18	0.29 $\pm$ 0.06 0.35 $\pm$ 0.19	0.26 $\pm$ 0.06 0.34 $\pm$ 0.17	0.28 $\pm$ 0.04 0.31 $\pm$ 0.15	0.27 $\pm$ 0.01 0.28 $\pm$ 0.14

Appendix

Table 5: Win rate comparison of the CDRL model and the baseline model under different observation and action noise.

Obs / Act	0.00	0.05	0.10	0.15	0.20	0.30
0.00	0.92 $\pm$ 0.04 1 $\pm$ 0.01	0.93 $\pm$ 0.04 0.97 $\pm$ 0.01	0.90 $\pm$ 0.04 0.96 $\pm$ 0.03	0.87 $\pm$ 0.07 0.97 $\pm$ 0.01	0.84 $\pm$ 0.06 0.95 $\pm$ 0.03	0.73 $\pm$ 0.13 0.92 $\pm$ 0.05
1.00	0.88 $\pm$ 0.11 0.90 $\pm$ 0.10	0.83 $\pm$ 0.16 0.92 $\pm$ 0.08	0.84 $\pm$ 0.11 0.90 $\pm$ 0.09	0.80 $\pm$ 0.14 0.91 $\pm$ 0.08	0.79 $\pm$ 0.15 0.88 $\pm$ 0.09	0.62 $\pm$ 0.09 0.85 $\pm$ 0.09
2.00	0.58 $\pm$ 0.31 0.62 $\pm$ 0.24	0.54 $\pm$ 0.29 0.64 $\pm$ 0.23	0.54 $\pm$ 0.31 0.63 $\pm$ 0.27	0.51 $\pm$ 0.29 0.62 $\pm$ 0.27	0.52 $\pm$ 0.23 0.59 $\pm$ 0.24	0.42 $\pm$ 0.21 0.60 $\pm$ 0.21
3.00	0.50 $\pm$ 0.29 0.48 $\pm$ 0.23	0.48 $\pm$ 0.30 0.51 $\pm$ 0.26	0.45 $\pm$ 0.23 0.46 $\pm$ 0.26	0.46 $\pm$ 0.25 0.51 $\pm$ 0.21	0.40 $\pm$ 0.21 0.46 $\pm$ 0.24	0.40 $\pm$ 0.18 0.49 $\pm$ 0.20
5.00	0.30 $\pm$ 0.09 0.34 $\pm$ 0.24	0.33 $\pm$ 0.05 0.31 $\pm$ 0.23	0.30 $\pm$ 0.07 0.33 $\pm$ 0.24	0.28 $\pm$ 0.07 0.30 $\pm$ 0.18	0.32 $\pm$ 0.04 0.31 $\pm$ 0.18	0.29 $\pm$ 0.07 0.32 $\pm$ 0.15
10.00	0.27 $\pm$ 0.07 0.29 $\pm$ 0.05	0.31 $\pm$ 0.06 0.28 $\pm$ 0.04	0.29 $\pm$ 0.06 0.31 $\pm$ 0.07	0.26 $\pm$ 0.06 0.31 $\pm$ 0.08	0.28 $\pm$ 0.04 0.28 $\pm$ 0.06	0.27 $\pm$ 0.01 0.27 $\pm$ 0.06

Appendix

Table 6: Win rate comparison of the CDRL-dendrite model and the baseline model under different observation and action noise.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Structured Representation Learning with Locally Linear Embeddings and Adaptive Feature Fusion
Jun 16, 2026Somjit Nath, Jackson J Cone, Derek Nowrouzezahrai +1Disentangled Representation Learning State Representation Learning
Department of Electrical and Computer Engineering McGill University Mila – Quebec AI Institute · Department of Psychology Department of Biomedical Engineering University of Calgary Canada Research Chair in Computational Behavioural Neuroscience (Tier II) · Department of Electrical and Software Engineering Schulich School of Engineering University of Calgary Mila – Quebec AI Institute Canada CIFAR AI Chair
Dynamical Priors as a Training Objective in Reinforcement Learning
Apr 23, 2026Sukesh SubaharanReinforcement Learning Policy Gradient Methods
Coimbatore Medical College Coimbatore, TN India
A Reference Architecture of Reinforcement Learning Frameworks
Mar 6, 2026Xiaoran Liu, Istvan DavidReinforcement Learning
McMaster University, Hamilton, Canada · McMaster Centre for Software Certification, Hamilton, Canada

Jun 16, 2026J/K move · Enter open · S save

Structured Representation Learning with Locally Linear Embeddings and Adaptive Feature Fusion

Somjit Nath, Jackson J Cone, Derek Nowrouzezahrai, Samira Ebrahimi Kahou

Department of Electrical and Computer Engineering McGill University Mila – Quebec AI Institute · Department of Psychology Department of Biomedical Engineering University of Calgary Canada Research Chair in Computational Behavioural Neuroscience (Tier II) · Department of Electrical and Software Engineering Schulich School of Engineering University of Calgary Mila – Quebec AI Institute Canada CIFAR AI Chair

Disentangled Representation Learning State Representation Learning

Open paper AskPDF arXiv

Neuroscientific research has revealed that the brain encodes complex behaviors by leveraging structured, low-dimensional manifolds and dynamically fusing multiple sources of information through adaptive gating mechanisms. Inspired by these principles, we propose a novel reinforcement learning (RL) framework that encourages the disentanglement of dynamics-specific and reward-specific features, drawing direct parallels to how neural circuits separate and integrate information for efficient decision-making. Our approach leverages locally linear embeddings (LLEs) to capture the intrinsic, locally linear structure inherent in many environments, mirroring the local smoothness observed in neural population activity, while concurrently deriving reward-specific features through the standard RL objective. An attention mechanism, analogous to cortical gating, adaptively fuses these complementary representations on a per-state basis. Experimental results on benchmark tasks demonstrate that our method, grounded in neuroscientific principles, improves learning efficiency and overall performance compared to conventional RL approaches, highlighting the benefits of explicitly modeling local state structures and adaptive feature selection as observed in biological systems.

Abstract

Reinforcement Learning

Explore similar work

Structured Representation Learning with Locally Linear Embeddings and Adaptive Feature Fusion

Dynamical Priors as a Training Objective in Reinforcement Learning

A Reference Architecture of Reinforcement Learning Frameworks