FIAS Scientific Report 2011 - Frankfurt Institute for Advanced Studies ...

Neurally plausible and statistically optimal approaches to learning
Collaborators: J. Lücke 1 , J. A. Shelton 1 , A. S. Sheikh 1 , J. Bornschein 1 , Z. Dai 1 , C. Savin 2 , P. Berkes 3
1 Frankfurt Institute for Advanced Studies, 2 Computational and Biological Learning Lab, Dept of Engineering, University
of Cambridge, UK, 3 Visual Information Processing and Learning Lab, Life Science, Brandeis University, USA
The brain’s ability to recognize and interpret images, sounds and other sensory data is unmatched by any
artificial system, so far. Much or most of this ability neural brain circuits acquire through a long process of
small improvements while being exposed to sensory stimuli. This process we call ’learning’, and it has been
recognized as the key to build intelligent systems. In our studies we bring together the information theoretic
foundations of learning and learning in neural circuits. Our results help improve the understanding of brain
functions and help building functional approaches for scientific data analysis and computer vision.
Figure A: Hierarchical graphic memory model for pattern classification. B: Assignment of autonomously learned handwritten
digits (bottom row) to classes using few labelled examples. C: Classification results of a statistical generative
model for classification compared to neural circuits. Both can be made almost equal (green and black line).
As part of the Bernstein Focus Neurotechnology Frankfurt and of the DFG project “Non-linear Generative
Models for Representational Recognition and Unsupervised Learning in Vision” we study novel approaches
for efficient and neurally plausible learning. We directly relate to recent experimental findings (Berkes et al.,
Science, 2011) suggesting that neural response variability and spontaneous activity are a consequence of a
probabilistic neural code that processes stimuli by stochastically activating neurons encoding possible stimulus
interpretations. In the paper by Shelton et al. (2011) we made the high efficiency of neural processing and
learning consistent with these experimental finding. Prior work on efficient learning (in collaboration with
the Honda Research Institute, Europe; Lücke, Eggert, 2010) has, furthermore, allowed for the development of
several new learning algorithms.
In the project by Keck et al. (2012) we go a step further and study concrete neural interaction rules that allow
for optimal learning in an information theoretical sense. We could thus show that well-known neural interaction
rules and mechanisms such as feed-forward inhibition and synaptic scaling can result in a statistically optimal
learning of pattern memories (see figure).
All our approaches exploit the hardware for massively parallel computing available at the FIAS and the
CSC Frankfurt (compare Dai et al., 2011).
Related publications in 2011:
Keck, C., Savin, C., and Lücke, J. Input normalization and synaptic scaling – two sides of the same coin? PLoS
Computational Biology, submitted 2011, accepted 2012.
Shelton, J. A., Bornschein, J., Sheikh, A. S., Berkes, P., and Lücke, J. Select and sample – A model of efficient
neural inference and learning. Advances in Neural Information Processing Systems 24, 2618-2626, 2011.
Dai, Z., Shelton, J. A., Bornschein, J., Sheikh, A. S., and Lücke, J. Combining approximate inference methods
for efficient learning on large computer clusters. NIPS Workshop: Big Learning - Algorithms, Systems, and
Tools for Learning at Scale, 2011.
86