All On-Board: Fully On-Chip Neuromorphic Q-Learning with Embedded CartPole Simulation
Authors: Steven C. Nesbit, Giovanni T. Michel, Gerd J. Kunde, Edward Kim, Andrew T. Sornborger
Organizations: College of Computing and Informatics, Drexel University, Philadelphia, PA, USA · Information Sciences (CAI-3), Los Alamos National Laboratory, Los Alamos, NM, USA · Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL, USA · Nuclear & Particle Physics & Applications (P-3), Los Alamos National Laboratory, Los Alamos, NM, USA
As AI models grow in size and usage, their energy demands increase dramatically, raising sustainability and economic concerns. Neuromorphic hardware, inspired by the energy efficiency of the brain, seeks to address this challenge by offering low-power, fast-processing alternatives to conventional computing. Such hardware is particularly well-suited to control systems deployed in resource-constrained environments, which are best trained via reinforcement learning (RL). This contribution presents the design and implementation of a fully on-chip, closed-loop Loihi 2 RL agent. Our neuromorphic circuit consists of a fully embedded Q-learning algorithm and an on-chip simulation of the CartPole-v0 environment on Loihi 2. Our Q-learning algorithm trained the same number of successful agents as the CPU implementation in only half the execution time and with two orders of magnitude less dynamic power. These findings demonstrate the viability of RL on neuromorphic hardware and highlight its promise for building energy-efficient, real-time, embedded AI systems.
Figures & tables
Fig. 1: Visualizations of CartPole environment. Corresponding 30-second videos are available at https://github.com/nesbitsc/all-on-board-figure-1 . a : OpenAI Gym environment inference using agent with ideal Q-table, b : Loihi 2 environment inference using agent with ideal Q-table, c : OpenAI Gym environment inference using on-chip trained agent, d : Loihi 2 environment inference using same agent as (c), e : OpenAI Gym environment inference using on-chip trained agent, unique from (c) and (d), f : Loihi 2 environment inference using same agent as (e), g : OpenAI Gym environment inference using CPU-trained agent, h : OpenAI Gym environment inference using CPU-trained agent, unique from (g), i : Loihi 2 environment inference using on-chip-trained agent trained for 0, j : 10 000, k : 20 000, l : 30 000, m : 40 000 weight updates, n : Loihi 2 environment training at 0, o : 10 000, p : 20 000, q : 30 000, r : 40 000 weight updates, s : Loihi 2 environment inference using exploration rate decay function beginning at 30% and t : 10% randomness.
Fig. 2: Overview of neuromorphic Q-learning circuit architecture. Angle and angular velocity measurements are sent from the CartPole neuron to the policy network, which sends these values through the Q-table. Q-table output from the policy network goes to the decision process module, which determines whether or not to take a left or right action. The CartPole neuron also sends reward information to the random decision module, which decides whether (exploration) or not (exploitation) to select an action randomly, and if so, makes a random decision and overrides the greedy decision made by the decision process module. The policy network also sends its Q-table output values to the weight update module, which in turn sends Q-value update information to the policy network. Finally, the action selected by the decision process module is sent to the CartPole neuron to calculate the next state of the CartPole environment.
Fig. 3: Functional connectivity of the neuromorphic Q-learning circuit. Layers are only shown when gated ‘on,’ and synapses are only shown when their target is gated on. The names of the neuron layers are given on the left margin so that the row corresponds to layer identity. The columns correspond to the time steps of the algorithm, which correlate to the time steps on Loihi 2. Arrow labels indicate information flow through the synapse. Variables ϵ and α refer to the exploration and learning rates, respectively (see Equation 11 ). arand , random action; y1 , constrained update term (see Sections II-B2 and II-B5 ).
Hyperparameter
Value
Q-value update function
Equation 1
α , ϵ decay function
Equation 11
reward rt+1
±8
discount factor γ
0.99
Q-table shape
2×72
Q-value initialization
−127
TABLE I: Hyperparameters used in neuromorphic Q-learning.
Fig. 4: Agent performance on CartPole-v0 after Q-learning on CPU and Loihi 2. a : percentage of CPU-trained agents scoring 195 or higher, b : percentage of Loihi 2-trained agents scoring 195 or higher, c : CPU-trained top-performing agent, d : Loihi 2-trained top-performing agent, e : CPU-trained top 50% of agents, f : Loihi 2-trained top 50% of agents, g : all CPU-trained agents, h : all Loihi 2-trained agents.
Task
Hardware
Power (W)
Latency
Static
Dynamic
Total
( µ s)
Loihi 2 training
x86 cores
1.56
0.00
1.56
neurocores (34)
0.22
0.02
0.25
whole board
1.78
0.02
1.80
58.42
CPU training
Intel Core i7-10870H
31.23
12.00
43.22
162.54
Loihi 2 inference
x86 cores
1.56
0.00
1.56
TABLE II: Breakdown of performance profiling for Q-learning training and inference on Loihi 2 and the CPU.
Dept. of Electrical and Computer Engineering, University of California, Santa Cruz, CA, USA · Dept. of Bioengineering, University of California, San Diego, La Jolla, CA, USA
Department of Computer Science New Mexico Institute of Mining and Technology Socorro, NM 87801, USA · Department of Electrical and Computer Engineering The University of Texas at El Paso El Paso, TX 79968, USA · Department of Electrical Engineering New Mexico Institute of Mining and Technology Socorro, NM 87801, USA