K-prop: Deep Online Learning by Backpropagating Temporal Kernels
Organizations: Department of Physiology, University of Bern, Bern, Switzerland
Abstract
Self-recurrence represents an important component for temporal processing in neural networks, both artificial and biological, where it appears naturally in the form of leaky integration. However, due to the complex temporal dependencies of the gradient, an efficient and effective method of online training is still lacking. Here we propose K-prop, a novel online learning method for leaky neural networks. It dramatically reduces the computational and memory cost of real time recurrent learning (RTRL) by propagating kernel gains. K-prop also outperforms existing RTRL approximations and achieves performance comparable to backpropagation through time in sequence learning, classification, and reinforcement learning tasks. We further analyze how its sole approximation affects gradient quality and identify conditions under which K-prop remains accurate and effective.
Figures & tables
| Method | MNIST1d-Lk4 | MNIST1d-Lk5 | GSC |
|---|---|---|---|
| BPTT Werbos (2002) | |||
| OSTL Bohnstingl et al. (2022) | |||
| K-prop (Ours) |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | (ms) | Leaky layers | L1 | L2 | L3 | L4 | L5 | |
|---|---|---|---|---|---|---|---|---|
| N-sum | 0.25 | 3 | [0, 6.28] | 0.25, 3, 6 | 2, 4, 10 | |||
| Lk4 | 1 | 4 | [0, 10] | 3, 6 | 2, 7 | 2.5, 8 | ||
| Lk5 | 1 | 5 | [0, 10] | 3, 6 | 2, 7 | 1, 8 | 2.5, 6.6 | |
| GSC | 2 | 4 | [0, 40] | 3,12, 24 | 4, 16, 36 | 50, 400 | ||
| Cartpole | 5 | 3 | [0, 40] | 5, 10 | 11, 24 | |||
| Lunarlander | 5 | 3 | [0, 20] | 5, 20 | 6, 10 |
| Model | Network width | Learning rate | Update Period | Batch size | |
|---|---|---|---|---|---|
| N-sum | 6-6-6-2 | 0.6 | 0.1 | 50 ms | 1 |
| Lk4 | 36-54-60-60-72-10 | 0.6 | 0.01 | 180 ms | 100 |
| Lk5 | 54-60-60-60-60-72-10 | 0.5 | 0.01 | 180 ms | 100 |
| GSC | 240-360-360-400-400-20 | 0.5 | 0.001 | 1s | 256 |
| Cartpole | 32-40-30-20-2 | 0.6 | 0.01 | 3 s | 1 |
| Lunarlander | 120-120-80-20-4 | 0.6 | 0.0002 | - | 1 |
Recurrent Neural Networks
Recurrent neural networks process sequences by updating a persistent hidden state, enabling temporal modeling, memory, and streaming computation.
See all 329 papers