Itamar Arel

A world survey of artificial brain projects, Part II: Biologically inspired cognitive architectures

A number of leading cognitive architectures that are inspired by the human brain, at various levels of granularity, are reviewed and compared, with special attention paid to the way their internal structures and dynamics map onto neural processes. Four categories of Biologically Inspired Cognitive Architectures (BICAs) are considered, with multiple examples of each category briefly reviewed, and selected examples discussed in more depth: primarily symbolic architectures (e.g. ACT-R), emergentist architectures (e.g. DeSTIN), developmental robotics architectures (e.g. IM-CLEVER), and our central focus, hybrid architectures (e.g. LIDA, CLARION, 4D/RCS, DUAL, MicroPsi, and OpenCog). Given the state of the art in BICA, it is not yet possible to tell whether emulating the brain on the architectural level is going to be enough to allow rough emulation of brain function; and given the state of the art in neuroscience, it is not yet possible to connect BICAs with large-scale brain simulations in a thoroughgoing way. However, it is nonetheless possible to draw reasonably close function connections between various components of various BICAs and various brain regions and dynamics, and as both BICAs and brain simulations mature, these connections should become richer and may extend further into the domain of internal dynamics as well as overall behavior.

A Fast and Stable Incremental Clustering Algorithm

Clustering is a pivotal building block in many data mining applications and in machine learning in general. Most clustering algorithms in the literature pertain to off-line (or batch) processing, in which the clustering process repeatedly sweeps through a set of data samples in an attempt to capture its underlying structure in a compact and efficient way. However, many recent applications require that the clustering algorithm be online, or incremental, in the that there is no a priori set of samples to process but rather samples are provided one iteration at a time. Accordingly, the clustering algorithm is expected to gradually improve its prototype (or centroid) constructs. Several problems emerge in this context, particularly relating to the stability of the process and its speed of convergence. In this paper, we present a fast and stable incremental clustering algorithm, which is computationally modest and imposes minimal memory requirements. Simulation results clearly demonstrate the advantages of the proposed framework in a variety of practical scenarios.

30.10 A 1TOPS/W analog deep machine-learning engine with floating-gate storage in 0.13μm CMOS

An analog implementation of a deep machine-learning system for efficient feature extraction is presented in this work. It features online unsupervised trainability and non-volatile floating-gate analog storage. It utilizes a massively parallel reconfigurable current-mode analog architecture to realize efficient computation, and leverages algorithm-level feedback to provide robustness to circuit imperfections in analog signal processing. A 3-layer, 7-node analog deep machine-learning engine was fabricated in a 0.13 μm standard CMOS process, occupying 0.36 mm 2 active area. At a processing speed of 8300 input vectors per second, it consumes 11.4 μW from the 3 V supply, achieving 1×10 12 operation per second per Watt of peak energy efficiency. Measurement demonstrates real-time cluster analysis, and feature extraction for pattern recognition with 8-fold dimension reduction with an accuracy comparable to the floating-point software simulation baseline.

Mapping the Landscape of Human-Level Artificial General Intelligence

We present the broad outlines of a roadmap toward human-level artificial general intelligence (henceforth, AGI). We begin by discussing AGI in general, adopting a pragmatic goal for its attainment and a necessary foundation of characteristics and requirements. An initial capability landscape will be presented, drawing on major themes from developmental psychology and illuminated by mathematical, physiological and information processing perspectives. The challenge of identifying appropriate tasks and environments for measuring AGI will be addressed, and seven scenarios will be presented as milestones suggesting a roadmap across the AGI landscape along with directions for future research and collaboration.

A condensation-based application of Cramer's rule for solving large-scale linear systems

State-of-the-art software packages for solving large-scale linear systems are predominantly founded on Gaussian elimination techniques (e.g. LU-decomposition). This paper presents an efficient framework for solving large-scale linear systems by means of a novel utilization of Cramer@?s rule. While the latter is often perceived to be impractical when considered for large systems, it is shown that the algorithm proposed retains an O(N^3) complexity with pragmatic forward and backward stability properties. Empirical results are provided to substantiate the stated accuracy and computational complexity claims.

Deep Spatiotemporal Feature Learning with Application to Image Classification

Deep machine learning is an emerging framework for dealing with complex high-dimensionality data in a hierarchical fashion which draws some inspiration from biological sources. Despite the notable progress made in the field, there remains a need for an architecture that can represent temporal information with the same ease that spatial information is discovered. In this work, we present new results using a recently introduced deep learning architecture called Deep Spatio-Temporal Inference Network (DeSTIN). DeSTIN is a discriminative deep learning architecture that combines concepts from unsupervised learning for dynamic pattern representation together with Bayesian inference. In DeSTIN the spatiotemporal dependencies that exist within the observations are modeled inherently in an unguided manner. Each node models the inputs by means of clustering and simple dynamics modeling while it constructs a belief state over the distribution of sequences using Bayesian inference. We demonstrate that information from the different layers of this hierarchical system can be extracted and utilized for the purpose of pattern classification. Earlier simulation results indicated that the framework is highly promising, consequently in this work we expand DeSTIN to a popular problem, the MNIST data set of handwritten digits. The system as a preprocessor to a neural network achieves a recognition accuracy of 97.98% on this data set. We further show related experimental results pertaining to automatic cluster adaptation and termination.

Unsupervised neuron selection for mitigating catastrophic forgetting in neural networks

Catastrophic forgetting is a well studied problem in artificial neural networks in which past representations are rapidly lost as new representations are constructed. We hypothesize that such forgetting occurs due to overlap in the hidden layers, as well as the global nature in which neurons encode information. We introduce a novel technique to mitigate forgetting which effectively minimizes activation overlapping by using online clustering to effectively select neurons in the feedforward and back-propagation phases. We demonstrate the memory retention properties of the proposed scheme using the MNIST digit recognition data set.

On the Impact of Approximate Computation in an Analog DeSTIN Architecture

Deep machine learning (DML) holds the potential to revolutionize machine learning by automating rich feature extraction, which has become the primary bottleneck of human engineering in pattern recognition systems. However, the heavy computational burden renders DML systems implemented on conventional digital processors impractical for large-scale problems. The highly parallel computations required to implement large-scale deep learning systems are well suited to custom hardware. Analog computation has demonstrated power efficiency advantages of multiple orders of magnitude relative to digital systems while performing nonideal computations. In this paper, we investigate typical error sources introduced by analog computational elements and their impact on system-level performance in DeSTIN-a compositional deep learning architecture. These inaccuracies are evaluated on a pattern classification benchmark, clearly demonstrating the robustness of the underlying algorithm to the errors introduced by analog computational elements. A clear understanding of the impacts of nonideal computations is necessary to fully exploit the efficiency of analog circuits.

Hierarchical spatiotemporal feature extraction using recurrent online clustering

Deep machine learning offers a comprehensive framework for extracting meaningful features from complex observations in an unsupervised manner. The majority of deep learning architectures described in the literature primarily focus on extracting spatial features. However, in real-world settings, capturing temporal dependencies in observations is critical for accurate inference. This paper introduces an enhancement to DeSTIN - a compositional deep learning architecture in which each layer consists of multiple instantiations of a common node - that learns to represent spatiotemporal patterns in data based on a novel recurrent clustering algorithm. Contrary to mainstream deep architectures, such as deep belief networks where layer-by-layer training is assumed, each of the nodes in the proposed architecture is trained independently and in parallel. Moreover, top-down and bottom-up information flows facilitate rich feature formation. A semi-supervised setting is demonstrated achieving state-of-the-art results on the MNIST classification benchmarks. A GPU implementation is discussed further accentuating the scalability properties of the proposed framework.

Reinforcement learning based visual attention with application to face detection

Visual attention is the cognitive process of directing our gaze on one aspect of the visual field while ignoring others. The mainstream approach to modeling focal visual attention involves identifying saliencies in the image and applying a search process to the salient regions. However, such inference schemes commonly fail to accurately capture perceptual attractors, require massive computational effort and, generally speaking, are not biologically plausible. This paper introduces a novel approach to the problem of visual search by framing it as an adaptive learning process. In particular, we devise an approximate optimal control framework, based on reinforcement learning, for actively searching a visual field. We apply the method to the problem of face detection and demonstrate that the technique is both accurate and scalable. Moreover, the foundations proposed here pave the way for extending the approach to other large-scale visual perception problems.