Towards a cortically inspired deep learning model: Semi-supervised learning, divisive normalization, and synaptic pruning

Abstract

Deep learning has driven dramatic advances in performance on a wide range of difficult machine perception tasks, and its applications abound. Yet for many tasks it still lags far behind the mammalian brain in term of performance and efficiency in natural tasks. Building a brain-inspired learning system to narrow the gap between artificial and biological neural networks has been a long sought-after goal in both the neuroscience and machine learning communities. To take a step towards a neurally plausible learning system, we build a class of models that use functional elements and computational principles of the cortex for more robust and versatile machine learning. In particular, we incorporate the following three major neural features into the Deep Convolutional Networks (DCNs): semi-supervised learning, divisive normalization, and synaptic pruning. These neural features are derived from a recently developed generative model underlying DCNs - the Deep Rendering Mixture Model (DRMM). Our semi-supervised learning algorithm achieves state-of-the-art performance on the MNIST and SVHN datasets and competitive results on CIFAR10 amongst all methods that do not use data augmentation. Our divisive normalization enables faster and more stable training. Using our synaptic pruning method, we can compress the model significantly with little loss in accuracy.