TY - JOUR
T1 - Online Spatio-Temporal Learning in Deep Neural Networks
AU - Bohnstingl, Thomas
AU - Wozniak, Stanislaw
AU - Pantazi, Angeliki
AU - Eleftheriou, Evangelos
N1 - Publisher Copyright:
Author
PY - 2022
Y1 - 2022
N2 - Biological neural networks are equipped with an inherent capability to continuously adapt through online learning. This aspect remains in stark contrast to learning with error backpropagation through time (BPTT) that involves offline computation of the gradients due to the need to unroll the network through time. Here, we present an alternative online learning algorithm framework for deep recurrent neural networks (RNNs) and spiking neural networks (SNNs), called online spatio-temporal learning (OSTL). It is based on insights from biology and proposes the clear separation of spatial and temporal gradient components. For shallow SNNs, OSTL is gradient equivalent to BPTT enabling for the first time online training of SNNs with BPTT-equivalent gradients. In addition, the proposed formulation unveils a class of SNN architectures trainable online at low time complexity. Moreover, we extend OSTL to a generic form, applicable to a wide range of network architectures, including networks comprising long short-term memory (LSTM) and gated recurrent units (GRUs). We demonstrate the operation of our algorithm framework on various tasks from language modeling to speech recognition and obtain results on par with the BPTT baselines.
AB - Biological neural networks are equipped with an inherent capability to continuously adapt through online learning. This aspect remains in stark contrast to learning with error backpropagation through time (BPTT) that involves offline computation of the gradients due to the need to unroll the network through time. Here, we present an alternative online learning algorithm framework for deep recurrent neural networks (RNNs) and spiking neural networks (SNNs), called online spatio-temporal learning (OSTL). It is based on insights from biology and proposes the clear separation of spatial and temporal gradient components. For shallow SNNs, OSTL is gradient equivalent to BPTT enabling for the first time online training of SNNs with BPTT-equivalent gradients. In addition, the proposed formulation unveils a class of SNN architectures trainable online at low time complexity. Moreover, we extend OSTL to a generic form, applicable to a wide range of network architectures, including networks comprising long short-term memory (LSTM) and gated recurrent units (GRUs). We demonstrate the operation of our algorithm framework on various tasks from language modeling to speech recognition and obtain results on par with the BPTT baselines.
KW - Approximation algorithms
KW - Backpropagation
KW - backpropagation through time (BPTT)
KW - Biological neural networks
KW - Biology
KW - Heuristic algorithms
KW - Neurons
KW - online learning
KW - real-time recurrent learning (RTRL)
KW - spiking neurons.
KW - Training
UR - http://www.scopus.com/inward/record.url?scp=85126511953&partnerID=8YFLogxK
U2 - 10.1109/TNNLS.2022.3153985
DO - 10.1109/TNNLS.2022.3153985
M3 - Article
AN - SCOPUS:85126511953
SN - 2162-237X
JO - IEEE Transactions on Neural Networks and Learning Systems
JF - IEEE Transactions on Neural Networks and Learning Systems
ER -