Amirhossein Tavanaei

Acquisition of visual features through probabilistic spike-timing-dependent plasticity

This paper explores modifications to a feedforward five-layer spiking convolutional network (SCN) of the ventral visual stream [Masquelier, T., Thorpe, S., Unsupervised learning of visual features through spike timing dependent plasticity. PLoS Computational Biology, 3(2), 247-257]. The original model showed that a spike-timing-dependent plasticity (STDP) learning algorithm embedded in an appropriately selected SCN could perform unsupervised feature discovery. The discovered features where interpretable and could effectively be used to perform rapid binary decisions in a classifier. In order to study the robustness of the previous results, the present research examines the effects of modifying some of the components of the original model. For improved biological realism, we replace the original non-leaky integrate-and-fire neurons with Izhikevich-like neurons. We also replace the original STDP rule with a novel rule that has a probabilistic interpretation. The probabilistic STDP slightly but significantly improves the performance for both types of model neurons. Use of the Izhikevich-like neuron was not found to improve performance although performance was still comparable to the IF neuron. This shows that the model is robust enough to handle more biologically realistic neurons. We also conclude that the underlying reasons for stable performance in the model are preserved despite the overt changes to the explicit components of the model.

A spiking network that learns to extract spike signatures from speech signals

Spiking neural networks (SNNs) with adaptive synapses reflect core properties of biological neural networks. Speech recognition, as an application involving audio coding and dynamic learning, provides a good test problem to study SNN functionality. We present a simple, novel, and efficient nonrecurrent SNN that learns to convert a speech signal into a spike train signature. The signature is distinguishable from signatures for other speech signals representing different words, thereby enabling digit recognition and discrimination in devices that use only spiking neurons. The method uses a small, nonrecurrent SNN consisting of Izhikevich neurons equipped with spike timing dependent plasticity (STDP) and biologically realistic synapses. This approach introduces an efficient and fast network without error-feedback training, although it does require supervised training. The new simulation results produce discriminative spike train patterns for spoken digits in which highly correlated spike trains belong to the same category and low correlated patterns belong to different categories. The proposed SNN is evaluated using a spoken digit recognition task where a subset of the Aurora speech dataset is used. The experimental results show that the network performs well in terms of accuracy rate and complexity.

Training a Hidden Markov Model with a Bayesian Spiking Neural Network

It is of some interest to understand how statistically based mechanisms for signal processing might be integrated with biologically motivated mechanisms such as neural networks. This paper explores a novel hybrid approach for classifying segments of sequential data, such as individual spoken works. The approach combines a hidden Markov model (HMM) with a spiking neural network (SNN). The HMM, consisting of states and transitions, forms a fixed backbone with nonadaptive transition probabilities. The SNN, however, implements a biologically based Bayesian computation that derives from the spike timing-dependent plasticity (STDP) learning rule. The emission (observation) probabilities of the HMM are represented in the SNN and trained with the STDP rule. A separate SNN, each with the same architecture, is associated with each of the states of the HMM. Because of the STDP training, each SNN implements an expectation maximization algorithm to learn the emission probabilities for one HMM state. The model was studied on synthesized spike-train data and also on spoken word data. Preliminary results suggest its performance compares favorably with other biologically motivated approaches. Because of the model’s uniqueness and initial promise, it warrants further study. It provides some new ideas on how the brain might implement the equivalent of an HMM in a neural circuit.

Multi-layer unsupervised learning in a spiking convolutional neural network

Spiking neural networks (SNNs) have advantages over traditional, non-spiking networks with respect to biorealism, potential for low-power hardware implementations, and theoretical computing power. However, in practice, spiking networks with multi-layer learning have proven difficult to train. This paper explores a novel, bio-inspired spiking convolutional neural network (CNN) that is trained in a greedy, layer-wise fashion. The spiking CNN consists of a convolutional/pooling layer followed by a feature discovery layer, both of which undergo bio-inspired learning. Kernels for the convolutional layer are trained using a sparse, spiking auto-encoder representing primary visual features. The feature discovery layer uses a probabilistic spike-timing-dependent plasticity (STDP) learning rule. This layer represents complex visual features using WTA-thresholded, leaky, integrate-and-fire (LIF) neurons. The new model is evaluated on the MNIST digit dataset using clean and noisy images. Intermediate results show that the convolutional layer is stack-admissible, enabling it to support a multi-layer learning architecture. The recognition performance for clean images is above 98%. This performance is accounted for by the independent and informative visual features extracted in a hierarchy of convolutional and feature discovery layers. The performance loss for recognizing the noisy images is in the range 0.1% to 8.5%. This level of performance loss indicates that the network is robust to additive noise.

Towards recognition of protein function based on its structure using deep convolutional networks

This paper proposes a novel method for protein function recognition using deep learning. Recently, deep convolutional neural networks (DCNNs) demonstrated high performances in many areas of pattern recognition. Protein function is often associated with its tertiary structure denoting the active domain of a protein. This investigation develops a novel DCNN for protein functionality recognition based on its tertiary structure. Two rounds of experiments are performed. The initial experiment on tertiary protein structure alignment shows promising performances (94% accuracy rate) such that it shows the model robustness against rotations, local translations, and scales of the 3D structure. With these results, the main experiments contain five different datasets obtained by similarity measures between pairs of gene ontology terms. The experimental results for protein function recognition on selected datasets show 87.6% and 80.7% maximum and average accuracy rates respectively. The initial success of the DCNN in tertiary protein structure recognition supports further investigations with respect to tertiary protein retrieval and pattern mining on large scale problems.

Studying the interaction of a hidden Markov model with a Bayesian spiking neural network

This paper explores a novel hybrid approach for classifying sequential data such as isolated spoken words. The approach combines a hidden Markov model (HMM) with a spiking neural network (SNN). The HMM, consisting of states and transitions, forms a fixed backbone with nonadaptive transition probabilities. The SNN, however, implements a Bayesian computation by using an appropriately selected spike timing dependency (STDP) learning rule. A separate SNN, each with the same architecture, is associated with each of the p states of the HMM. Because of the STDP tuning, each SNN implements an expectation maximization (EM) algorithm to learn the particular observation probabilities for one particular HMM state. When applied to an isolated spoken word (as a popular sequential data) recognition problem, the hybrid model performs well and efficiently with a desirable accuracy rate. Because of the models uniqueness and initial success, it warrants further study. Future work intends to broaden its capabilities and improve the biological realism.

Bio-inspired Multi-layer Spiking Neural Network Extracts Discriminative Features from Speech Signals

Spiking neural networks (SNNs) enable power-efficient implementations due to their sparse, spike-based coding scheme. This paper develops a bio-inspired SNN that uses unsupervised learning to extract discriminative features from speech signals, which can subsequently be used in a classifier. The architecture consists of a spiking convolutional/pooling layer followed by a fully connected spiking layer for feature discovery. The convolutional layer of leaky, integrate-and-fire (LIF) neurons represents primary acoustic features. The fully connected layer is equipped with a probabilistic spike-timing-dependent plasticity learning rule. This layer represents the discriminative features through probabilistic, LIF neurons. To assess the discriminative power of the learned features, they are used in a hidden Markov model (HMM) for spoken digit recognition. The experimental results show performance above 96% that compares favorably with popular statistical feature extraction methods. Our results provide a novel demonstration of unsupervised feature acquisition in an SNN.

Representation learning using event-based STDP

Although representation learning methods developed within the framework of traditional neural networks are relatively mature, developing a spiking representation model remains a challenging problem. This paper proposes an event-based method to train a feedforward spiking neural network (SNN) layer for extracting visual features. The method introduces a novel spike-timing-dependent plasticity (STDP) learning rule and a threshold adjustment rule both derived from a vector quantization-like objective function subject to a sparsity constraint. The STDP rule is obtained by the gradient of a vector quantization criterion that is converted to spike-based, spatio-temporally local update rules in a spiking network of leaky, integrate-and-fire (LIF) neurons. Independence and sparsity of the model are achieved by the threshold adjustment rule and by a softmax function implementing inhibition in the representation layer consisting of WTA-thresholded spiking neurons. Together, these mechanisms implement a form of spike-based, competitive learning. Two sets of experiments are performed on the MNIST and natural image datasets. The results demonstrate a sparse spiking visual representation model with low reconstruction loss comparable with state-of-the-art visual coding approaches, yet our rule is local in both time and space, thus biologically plausible and hardware friendly.

A novel data-driven model for real-time influenza forecasting

We provide data-driven machine learning methods that are capable of making real-time influenza forecasts that integrate the impacts of climatic factors and geographical proximity to achieve better forecasting performance. The key contributions of our approach are both applying deep learning methods and incorporation of environmental and spatio-temporal factors to improve the performance of the influenza forecasting models. We evaluate the method on Influenza Like Illness (ILI) counts and climatic data, both publicly available data sets. Our proposed method outperforms existing known influenza forecasting methods in terms of their Mean Absolute Percentage Error and Root Mean Square Error. The key advantages of the proposed data-driven methods are as following: (1) The deep-learning model was able to effectively capture the temporal dynamics of flu spread in different geographical regions, (2) The extensions to the deep-learning model capture the influence of external variables that include the geographical proximity and climatic variables such as humidity, temperature, precipitation and sun exposure in future stages, (3) The model consistently performs well for both the city scale and the regional scale on the Google Flu Trends (GFT) and Center for Disease Control (CDC) flu counts. The results offer a promising direction in terms of both data-driven forecasting methods and capturing the influence of spatio-temporal and environmental factors for influenza forecasting methods.