Unipolar Neuron Networks Trained by Recirculation for Reward-Based Continual Learning
Organizations: Faculty of Automatic Control and Computer Engineering “Gheorghe Asachi” Technical University of Ias,i Bd. Mangeron 27, 700050 Ias,i, Romania
Abstract
Unipolar neurons are the predominant type of nerve cell in the octopus and in most other invertebrates. A single neurite (a fiber extending from the cell body) branches into a tree that carries both input and output synapses. The branches send graded local outputs to nearby cells, and one site on the main neurite sends a global output to distant targets. This paper proposes a network of such neurons and a learning rule for it. Each neuron is modeled as a tree of passive segments, whose activations are computed exactly as the steady state of a linear system. The network learns from reward alone, with a rule called recirculation. After each forward pass, a second pass sends requested changes of the outputs back through the same connections. Each synapse then adjusts its weight using only quantities available at its own location. Because the system of each neuron is symmetric, this update closely follows the gradient of the error in the predicted reward. Two mechanisms limit forgetting when tasks are learned in sequence: a task code inhibits a different subset of branches in each task, while consolidation lowers the learning rate of synapses that changed strongly in the recent past. The model is evaluated on a sequence of MNIST and Iris tasks. The network maintains high accuracy with low forgetting, whereas gradient-based training of the same network forgets earlier tasks almost completely.
Figures & tables
| Inputs | ||||||
|---|---|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 0 | 0.00 | |
| 0.524 | 0.048 | 0.095 | 0.224 | 0 | 0.00 | |
| 0.048 | 0.524 | 0.095 | 0 | 0.224 | 0.00 | |
| 0.545 | 0.545 | 0.182 | 0.245 | 0.245 | 1.00 |
| Inputs | ||||||
|---|---|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 0 | 0.00 | |
| 0.522 | 0.011 | 0.089 | 0.222 | 0 | 0.99 | |
| 0.011 | 0.522 | 0.089 | 0 | 0.222 | 0.99 | |
| 0.207 | 0.207 | 0.069 | 0 | 0 | 0.00 |
| Option | Change from the default model |
|---|---|
| No task code | The network receives no task code, so no branch is gated, and all tasks use the same branches and neurons. |
| Trunk gating | The task code also inhibits the trigger segment of a random subset of neurons, which silences their global output under that task. |
| Readout from all layers | The global outputs of all layers reach the readout, not only those of the last layer (Eq. 10 ). |
| Readout bias | Each action has a bias for the readout, shared by all inputs. |
| Synapses on branches only | No synapse is placed on the trunk segment. |
| Outputs at branch tips | Only the distal segment of each branch has a local output, 4 per neuron instead of 8. |
| Learner | Steps | After | Final | Forgetting | Early |
|---|---|---|---|---|---|
| training | reward | ||||
| Gradient, supervised | 1500 | 0.993 | 0.197 | 0.994 | – |
| Gradient, action values | 1500 | 0.994 | 0.198 | 0.995 | 0.846 |
| Recirculation | 500 | 0.944 | 0.939 | 0.009 | 0.632 |
| Recirculation | 1500 | 0.966 | 0.868 | 0.123 | 0.611 |
| Steps | After | Final | Forgetting | |||
| training | ||||||
| 0.0002 | 20 | 500 | 0.944 | 0.939 | 0.009 | |
| 0.0002 | 20 | 1500 | 0.966 | 0.868 | 0.123 | |
| 0.0002 | 5 | 1500 | 0.966 | 0.866 | 0.126 | |
| 0 | 20 | 1500 | 0.971 | 0.786 | 0.230 | |
| 0.0002 | 20 | 1500 | 0.972 | 0.733 | 0.298 |
| Configuration | Steps | After | Final | Forgetting | |
| training | |||||
| Default | 500 | 0.944 | 0.939 | 0.009 | |
| No task code | 500 | 0.895 | 0.195 | 0.875 | |
| Readout from all layers | 500 | 0.969 | 0.297 | 0.840 | |
| Default | 1500 | 0.966 | 0.868 | 0.123 | |
| Single layer | 1500 | 0.980 | 0.203 | 0.971 |
| Configuration | After | Final | Forgetting | Early |
|---|---|---|---|---|
| training | reward | |||
| Full rule | 0.939 | 0.934 | 0.008 | 0.520 |
| No reward part | 0.939 | 0.933 | 0.009 | 0.521 |
| No branch part | 0.936 | 0.931 | 0.008 | 0.510 |
| Readout only | 0.936 | 0.931 | 0.008 | 0.510 |
| No consolidation | 0.947 | 0.326 | 0.777 | 0.572 |
| After training on | Setosa, | Versicolor, | All three |
|---|---|---|---|
| versicolor | virginica | species | |
| Setosa, versicolor | 1.000 | 0.500 | 0.667 |
| Versicolor, virginica | 0.980 | 0.947 | 0.964 |
| All three species | 0.980 | 0.947 | 0.964 |
Continual RL
Continual reinforcement learning studies how agents adapt to changing tasks while retaining skills learned earlier.
See all 58 papers