Deep learning research landscape & roadmap in a nutshell: past, present and future -- Towards deep cortical learning
DDeep learning research landscape & roadmap in a nutshell:past, present and future - Towards deep cortical learning
Aras R. DargazanyAugust 7, 2019
Abstract
The past, present and future of deep learning is presented in this work. Given this landscape& roadmap, we predict that deep cortical learning will be the convergence of deep learning &cortical learning which builds an artificial cortical column ultimately.
Deep learning horizon, landscape and research roadmap in nutshell is presented in this figure 1.The historical development and timeline of deep learning & neural network is separately illustratedFigure 1: Deep learning research landscape & roadmap: past, present, future. The future ishighlighted as deep cortical learning.in figure 2. The Origin of neural nets [WR17] is thoroughly reviewed in terms of the evolutionaryhistory of deep learning models. Vernon Mountcastle discovery of cortical columns in somatosen-sory cortex [Mou97] was a breakthrough in brain science. The big bang was the discovery of Hubel& Wiesel of simple cells and complex cell in visual cortex [HW59] which won the Nobel prize forthis discovery in 1981. This work was heavily founded on Vernon Mountcastle discovery of corticalcolumns in somatosensory cortex [Mou97]. After the discovery of Hubel & Wiesel, Fukushimaproposed a pattern recognition architecture based on the simple cell and complex cell discovery,known as NeoCognitron [FM82]. In this work, a deep neural network was proposed using simplecell layer and complex cell layer repeatedly. In 80s and maybe a bit earlier backpropagation havebeen proposed by multiple people but the first time it was well-explained and applied for learningneural nets was done by Hinton and his colleagues in 1987 [RHW86].1 a r X i v : . [ c s . N E ] J u l igure 2: Neural nets origin, timeline & history made by Favio Vazquez Convolutional nets was invented by LeCun [LBD +
89] which led to deep learning conspiracy whichalso started by the three founding fathers of the field: LeCun, Bengio and Hinton [LBH15]. Themain hype in deep learning happened in 2012 when the state-of-the-art result in Imagenet classi-fication and TIMIT speech recognition task were dramatically reduced using an end-to-end deepconvolutional network [KSH12] and deep belief net [HDY + +
17] and TIMIT using multiple GPUs in an end-to-end fashion meaning directlyfrom raw inputs, all the way the desired outputs. Alexnet [KSH12] used two GPUs for Imagenetclassification which is a very big dataset of images, almost 1.5 million images of size 215x215.Kaiming He et al. [GDG +
17] proposed a highly scalable approach for training on Image using256 GPUs for almost an hour which shows an amazingly powerful approach based stochasticgradient descent for applying big cluster of GPUs on huge datasets. Very many application domainshave been revolutionized using deep learning architectures such as image classifications [KSH12],machine translation [WSC +
16, JSL + + + The main direction and inclination in the deep learning for future is the ability to bridge the gapbetween the cortical architecture and deep learning architectures, specifically convolutional nets.In this quest, Hinton proposed capsule network [SFH17] as an effort to get rid of pooling layersand replace it with capsules which are highly inspired bu cortical mini-columns in cortical columnsand layers and include the location information or pose information of parts.Another important quest in deep learning is understanding the biological root of learning in ourbrain, specifically in our cortex. Backpropagation is not biologically inspired and plausible. Hintonand the other founding fathers of deep learning have been trying to understand how backprop2ight be feasible biologically in brain. Feedback alignment [LCTA16] and spike time-dependentplasticity or STDP-based backprop [BSR +
18] are some of the works which have been done byTimothy Lillicrap, Blake Richards, and Hinton in order to model backprop biologically based onthe pyramidal neuron in the cortex.In the far future, the main goal should be the merge of two very independent quest to buildcortical structure in our brain: The first one is heavily target by the big and active deep learningcommunity; The second one is targeted independently and neuroscientifically by Numenta andGeoff Hawkins [HAD11]. These people argue that the cortical structure and our neocortex is themain source of our intelligence and for building a true intelligent machine, we should be able toreconstruct the cortex and to do so, we should first focus more on the cortex and understand whatcortex is made out of.
By merging deep learning and cortical learning, a very more focused and detailed architectures,named deep cortical learning might be created. We might be able to understand and reconstructthe cortical structure with much more accuracy and have a better idea what the true intelligence isand how artificial general intelligence or AGI might be reproducible. Deep cortical learning mightbe the algorithm behind one cortical column in the neocortex.
References [BSR +
18] Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton,and Timothy Lillicrap. Assessing the scalability of biologically-motivated deep learningalgorithms and architectures. In
Advances in Neural Information Processing Systems ,pages 9368–9378, 2018.[Bur14] Neil Burgess. The 2014 nobel prize in physiology or medicine: a spatial model forcognitive neuroscience.
Neuron , 84(6):1120–1125, 2014.[FM82] Kunihiko Fukushima and Sei Miyake. Neocognitron: A self-organizing neural networkmodel for a mechanism of visual pattern recognition. In
Competition and cooperationin neural nets , pages 267–285. Springer, 1982.[GDG +
17] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, AapoKyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatchsgd: training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 , 2017.[HAD11] Jeff Hawkins, Subutai Ahmad, and D Dubinsky. Hierarchi-cal temporal memory including htm cortical learning algorithms. , 2011.[HDY +
12] Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, NavdeepJaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deepneural networks for acoustic modeling in speech recognition: The shared views of fourresearch groups.
IEEE Signal Processing Magazine , 29(6):82–97, 2012.[HW59] David H Hubel and Torsten N Wiesel. Receptive fields of single neurones in the cat’sstriate cortex.
The Journal of physiology , 148(3):574–591, 1959.[JSL +
16] Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, ZhifengChen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al.Google’s multilingual neural machine translation system: Enabling zero-shot trans-lation. arXiv preprint arXiv:1611.04558 , 2016.[KSH12] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification withdeep convolutional neural networks. In
Advances in neural information processingsystems , pages 1097–1105, 2012. 3LBD +
89] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard,Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zipcode recognition.
Neural computation , 1(4):541–551, 1989.[LBH15] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning.
Nature ,521(7553):436–444, 2015.[LCTA16] Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Ran-dom synaptic feedback weights support error backpropagation for deep learning.
Naturecommunications , 7:13276, 2016.[MKS +
15] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness,Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, GeorgOstrovski, et al. Human-level control through deep reinforcement learning.
Nature ,518(7540):529–533, 2015.[Mou97] Vernon B Mountcastle. The columnar organization of the neocortex.
Brain: a journalof neurology , 120(4):701–722, 1997.[RHW86] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representa-tions by back-propagating errors.
Nature , 323:533–536, 1986.[SFH17] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between cap-sules. In
Advances in Neural Information Processing Systems , pages 3859–3869, 2017.[WR17] Haohan Wang and Bhiksha Raj. On the origin of deep learning. arXiv preprintarXiv:1702.07800 , 2017.[WSC +
16] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, WolfgangMacherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neuralmachine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144arXiv preprint arXiv:1609.08144