Azarakhsh Jalalvand

Acoustic Modeling With Hierarchical Reservoirs

Accurate acoustic modeling is an essential requirement of a state-of-the-art continuous speech recognizer. The Acoustic Model (AM) describes the relation between the observed speech signal and the non-observable sequence of phonetic units uttered by the speaker. Nowadays, most recognizers use Hidden Markov Models (HMMs) in combination with Gaussian Mixture Models (GMMs) to model the acoustics, but neural-based architectures are on the rise again. In this work, the recently introduced Reservoir Computing (RC) paradigm is used for acoustic modeling. A reservoir is a fixed - and thus non-trained - Recurrent Neural Network (RNN) that is combined with a trained linear model. This approach combines the ability of an RNN to model the recent past of the input sequence with a simple and reliable training procedure. It is shown here that simple reservoir-based AMs achieve reasonable phone recognition and that deep hierarchical and bi-directional reservoir architectures lead to a very competitive Phone Error Rate (PER) of 23.1% on the well-known TIMIT task.

Robust continuous digit recognition using Reservoir Computing

HighlightsStudy of robustness of Reservoir Computing (RC) based continuous digit recognizers.Discovery of new relations between RC control parameters, input and output dynamics.Use of these relations to find heuristics to reduce the reservoir development time.Creation of an RC-based recognizer that is more noise robust than the AFE-GMM-HMM. It is acknowledged that Hidden Markov Models (HMMs) with Gaussian Mixture Models (GMMs) as the observation density functions achieve excellent digit recognition performance at high signal to noise ratios (SNRs). Moreover, many years of research have led to good techniques to reduce the impact of noise, distortion and mismatch between training and test conditions on the recognition accuracy. Nevertheless, we still await systems that are truly robust against these confounding factors. The present paper extends recent work on acoustic modeling based on Reservoir Computing (RC), a concept that has its roots in Machine Learning. By introducing a novel analysis of reservoirs as non-linear dynamical systems, new insights are gained and translated into a new reservoir design recipe that is extremely simple and highly comprehensible in terms of the dynamics of the acoustic features and the modeled acoustic units. By tuning the reservoir to these dynamics, one can create RC-based systems that not only compete well with conventional systems in clean conditions, but also degrade more gracefully in noisy conditions. Control experiments show that noise-robustness follows from the random fixation of the reservoir neurons whereas, tuning the reservoir dynamics increases the accuracy without compromising the noise-robustness.

Real-Time Reservoir Computing Network-Based Systems for Detection Tasks on Visual Contents

Among the various types of artificial neural networks used for event detection in visual contents, those with the ability of processing temporal information, such as recurrent neural networks, have been proved to be more effective. However, training of such networks is often difficult and time consuming. In this work, we show how Reservoir Computing Networks (RCNs) can be used for detecting purposes on raw images. The applicability of RCNs is illustrated using two example challenges, namely isolated digit handwriting recognition on the MNIST dataset as well as detection of the status of a door using self-developed moving pictures from a surveillance camera. Achieving an error rate of 0.92 percent on MNIST, we show that RCN can be a serious competitor to the state-of-the-art. Moreover, we show how RCNs with their simple and yet robust training procedure can be practically used for real surveillance tasks using very low resolution camera sensors.

Towards using Reservoir Computing Networks for noise-robust image recognition

Reservoir Computing Network (RCN) is a special type of the single layer recurrent neural networks, in which the input and the recurrent connections are randomly generated and only the output weights are trained. Besides the ability to process temporal information, the key points of RCN are easy training and robustness against noise. Recently, we introduced a simple strategy to tune the parameters of RCN resulted in an effective and noise-robust RCN-based model for speech recognition. The aim of this work is to extend that study to the field of image processing. In particular, we investigate the potential of RCNs in achieving a competitive performance on the well-known MNIST dataset by following the aforementioned parameter optimizing strategy. Moreover, we achieve good noise robust recognition by utilizing such a network to denoise images and supplying them to a recognizer that is solely trained on clean images. The conducted experiments demonstrate that the proposed RCN-based handwritten digit recognizer achieves an error rate of 0.81 percent on the clean test data of the MNIST benchmark and that the proposed RCN-based denoiser can effectively reduce the error rate on the various types of noise.

Adaptive multicast routing method for 3D mesh-based Networks-on-Chip

The amalgamation of 3D VLSI technology and Networks-on-Chip (NoCs) offers a promising architectural platform for the future Multi-Processor Systems-on-Chip (MPSoCs). Since multicast communication is frequently exploited in such systems, it is highly desirable to design NoC-based routing methods that support multicast. In this paper, a highly adaptive and deadlock-free multicast routing method is proposed for 3D mesh-based NoCs without using virtual channels. Unlike conventional turn models which prohibit certain turns to avoid deadlock, the proposed scheme restricts the locations where certain turns can take place to provide a more even degree of routing adaptiveness. Moreover, different sets of rules are employed for different layers of the NoC to decrease the likelihood of congestion in the network. The simulation results demonstrate the efficiency of the proposed method due to a balanced traffic distribution across the network.

Feature enhancement with a Reservoir-based Denoising Auto Encoder

Recently, automatic speech recognition has advanced significantly by the introduction of deep neural networks for acoustic modeling. However, there is no clear evidence yet that this does not come at the price of less generalization to conditions that were not present during training. On the other hand, acoustic modeling with Reservoir Computing (RC) did not offer improved clean speech recognition but it leads to good robustness against noise and channel distortions. In this paper, the aim is to establish whether adding feature denoising in the front-end can further improve the robustness of an RC-based recognizer, and if so, whether one can devise an RC-based Denoising Auto Encoder that outperforms a traditional denoiser like the ETSI Advanced Front-End. In order to answer these questions, experiments are conducted on the Aurora-2 benchmark.

Noise robust continuous digit recognition with reservoir-based acoustic models

Notwithstanding the many years of research, more work is needed to create automatic speech recognition (ASR) systems with a close-to-human robustness against confounding factors such as ambient noise, channel distortion, etc. Whilst most work thus far focused on the improvement of ASR systems embedding Gaussian Mixture Models (GMM)s to compute the acoustic likelihoods in the states of a Hidden Markov Model (HMM), the present work focuses on the noise robustness of systems employing Reservoir Computing (RC) as an alternative acoustic modeling technique. Previous work already demonstrated good noise robustness for continuous digit recognition (CDR). The present paper investigates whether further progress can be achieved by driving reservoirs with noise-robust inputs that have been shown to raise the robustness of GMM-based systems, by introducing bi-directional reservoirs and by combining reservoirs with GMMs in a single system. Experiments on Aurora-2 demonstrate that it is indeed possible to raise the noise robustness without significantly increasing the system complexity.

On the application of reservoir computing networks for noisy image recognition

Abstract Reservoir Computing Networks (RCNs) are a special type of single layer recurrent neural networks, in which the input and the recurrent connections are randomly generated and only the output weights are trained. Besides the ability to process temporal information, the key points of RCN are easy training and robustness against noise. Recently, we introduced a simple strategy to tune the parameters of RCNs. Evaluation in the domain of noise robust speech recognition proved that this method was effective. The aim of this work is to extend that study to the field of image processing, by showing that the proposed parameter tuning procedure is equally valid in the field of image processing and conforming that RCNs are apt at temporal modeling and are robust with respect to noise. In particular, we investigate the potential of RCNs in achieving competitive performance on the well-known MNIST dataset by following the aforementioned parameter optimizing strategy. Moreover, we achieve good noise robust recognition by utilizing such a network to denoise images and supplying them to a recognizer that is solely trained on clean images. The experiments demonstrate that the proposed RCN-based handwritten digit recognizer achieves an error rate of 0.81 percent on the clean test data of the MNIST benchmark and that the proposed RCN-based denoiser can effectively reduce the error rate on the various types of noise.

Object localization in handheld thermal images for fireground understanding

Despite the broad application of the handheld thermal imaging cameras in firefighting, its usage is mostly limited to subjective interpretation by the person carrying the device. As remedies to overcome this limitation, object localization and classification mechanisms could assist the fireground understanding and help with the automated localization, characterization and spatio-temporal (spreading) analysis of the fire. An automated understanding of thermal images can enrich the conventional knowledge-based firefighting techniques by providing the information from the data and sensing-driven approaches. In this work, transfer learning is applied on multi-labeling convolutional neural network architectures for object localization and recognition in monocular visual, infrared and multispectral dynamic images. Furthermore, the possibility of analyzing fire scene images is studied and their current limitations are discussed. Finally, the understanding of the room configuration (i.e., objects location) for indoor localization in reduced visibility environments and the linking with Building Information Models (BIM) are investigated.

An Automated End-To-End Pipeline for Fine-Grained Video Annotation using Deep Neural Networks

The searchability of video content is often limited to the descriptions authors and/or annotators care to provide. The level of description can range from absolutely nothing to fine-grained annotations at the level of frames. Based on these annotations, certain parts of the video content are more searchable than others. Within the context of the STEAMER project, we developed an innovative end-to-end system that attempts to tackle the problem of unsupervised retrieval of news video content, leveraging multiple information streams and deep neural networks. In particular, we extracted keyphrases and named entities from transcripts, subsequently refining these keyphrases and named entities based on their visual appearance in the news video content. Moreover, to allow for fine-grained frame-level annotations, we temporally located high-confidence keyphrases in the news video content. To that end, we had to tackle challenges such as the automatic construction of training sets and the automatic assessment of keyphrase imageability. In this paper, we discuss the main components of our end-to-end system, capable of transforming textual and visual information into fine-grained video annotations.