Neural Wave Machines

in Research

MNIST_Waves_Recon Observed transformation (left), Latent Variable Waves (middle), and Reconstruction (right). We see the Neural Wave Machine learns to encode the observed transformations as traveling waves. In our paper, we show that such coordinated synchronous dynamics ultimately result in improved forecasting ability and efficiency when similarly modeling smooth continutous transformations as input.

Traveling waves have been measured at a diversity of regions and scales in the brain, however a consensus as to their computational purpose has yet to be reached. An intriguing hypothesis is that traveling waves serve to structure neural representations both in space and time, thereby acting as an inductive bias towards natural data. In this work, we investigate this hypothesis by introducing the Neural Wave Machine (NWM) – a locally coupled oscillatory recurrent neural network capable of exhibiting traveling waves in its hidden state. After training on simple dynamic sequences, we show that this model indeed learns static spatial structure such topographic organization, and further uses complex spatiotemporal structure such traveling waves to encode observed transformations. To measure the computational implications of this structure, we use a suite of sequence classification and physical dynamics modeling tasks to show that the NWM is both more parameter efficient, and is able to forecast future trajectories of simple physical dynamical systems more accurately than existing state of the art counterparts. We conclude with a discussion of how this model may allow for novel investigations of the computational hypotheses surrounding traveling waves which were previously challenging or impossible.

T. Anderson Keller and Max Welling

PDF: https://openreview.net/forum?id=5tJSt3kn4s
Accepted at: ICML 2023

Github.com/akandykeller/NeuralWaveMachines

Tweet-print