John B. Butcher

Reservoir computing and extreme learning machines for non-linear time-series data analysis

Random projection architectures such as Echo state networks (ESNs) and Extreme Learning Machines (ELMs) use a network containing a randomly connected hidden layer and train only the output weights, overcoming the problems associated with the complex and computationally demanding training algorithms traditionally used to train neural networks, particularly recurrent neural networks. In this study an ESN is shown to contain an antagonistic trade-off between the amount of non-linear mapping and short-term memory it can exhibit when applied to time-series data which are highly non-linear. To overcome this trade-off a new architecture, Reservoir with Random Static Projections (R(2)SP) is investigated, that is shown to offer a significant improvement in performance. A similar approach using an ELM whose input is presented through a time delay (TD-ELM) is shown to further enhance performance where it significantly outperformed the ESN and R(2)SP as well other architectures when applied to a novel task which allows the short-term memory and non-linearity to be varied. The hard-limiting memory of the TD-ELM appears to be best suited for the data investigated in this study, although ESN-based approaches may offer improved performance when processing data which require a longer fading memory.

Defect Detection in Reinforced Concrete Using Random Neural Architectures

This article discusses how detecting defects within reinforced concrete is vital to the safety and durability of infrastructure. A non-invasive technique, ElectroMagnetic Anomaly Detection (EMAD) is used in this article to provide information into the electromagnetic properties of reinforcing steel for which data analysis is currently performed visually. The first use of two neural network approaches to automate the analysis of this data is investigated in this article. These approaches are called Echo State Networks (ESNs) and Extreme Learning Machines (ELMs) where fast and efficient training procedures allow networks to be trained and evaluated in less time than traditional neural network approaches. Data collected from real-world concrete structures are analyzed in this article using these two approaches as well as using a simple threshold measure and a standard recurrent neural network. Two ESN architectures provided the best performance for a mesh-reinforced concrete structure, while the ELM approach offers a large improvement in the performance of a single tendon-reinforced structure.

Artificial neural network analysis of hydrocarbon profiles for the ageing of Lucilia sericata for post mortem interval estimation

In analytical chemistry large datasets are collected using a variety of instruments for multiple tasks, where manual analysis can be time-consuming. Ideally, it is desirable to automate this process while obtaining an acceptable level of accuracy, two aims that artificial neural networks (ANNs) can fulfil. ANNs possess the ability to classify novel data based on their knowledge of the domain to which they have been exposed. ANNs can also analyse non-linear data, tolerate noise within data and are capable of reducing time taken to classify large amounts of novel data once trained, making them well-suited to the field of analytical chemistry where large datasets are present (such as that collected from gas chromatography-mass spectrometry (GC-MS)). In this study, the use of ANNs for the autonomous analysis of GC-MS profiles of Lucilia sericata larvae is investigated, where ANNs are required to estimate the age of the larvae to aid in the estimation of the post mortem interval (PMI). Two ANN analysis approaches are presented, where the ANN correctly classified the data with accuracy scores of 80.8% and 87.7% and Cohens Kappa coefficients of 0.78 and 0.86. Inspection of these results shows the ANN to confuse two consecutive days which are of the same life stage and as a result are very similar in their chemical profile, which can be expected. The grouping of these two days into one class further improved results where accuracy scores 89% and 97.5% were obtained for the two analysis approaches.

Astrocyte and Neuronal Plasticity in the Somatosensory System

Changing the whisker complement on a rodents snout can lead to two forms of experience-dependent plasticity (EDP) in the neurons of the barrel cortex, where whiskers are somatotopically represented. One form, termed coding plasticity, concerns changes in synaptic transmission and connectivity between neurons. This is thought to underlie learning and memory processes and so adaptation to a changing environment. The second, called homeostatic plasticity, serves to maintain a restricted dynamic range of neuronal activity thus preventing its saturation or total downregulation. Current explanatory models of cortical EDP are almost exclusively neurocentric. However, in recent years, increasing evidence has emerged on the role of astrocytes in brain function, including plasticity. Indeed, astrocytes appear as necessary partners of neurons at the core of the mechanisms of coding and homeostatic plasticity recorded in neurons. In addition to neuronal plasticity, several different forms of astrocytic plasticity have recently been discovered. They extend from changes in receptor expression and dynamic changes in morphology to alteration in gliotransmitter release. It is however unclear how astrocytic plasticity contributes to the neuronal EDP. Here, we review the known and possible roles for astrocytes in the barrel cortex, including its plasticity.

Pruning reservoirs with Random Static Projections

Reservoir Computing is a relatively new field of Recurrent Neural Networks in which only the output weights are re-calculated by the training process, removing the problems associated with traditional gradient descent algorithms. As the reservoir is recurrent, it can possess short term memory, but there is a trade-off between the amount of memory a reservoir can have and its nonlinear mapping capabilities. A new, custom architecture was recently proposed to overcome this by combining a reservoir with an extreme learning machine to deliver improved results. This paper extends this architecture further by introducing a ranking and pruning algorithm which removes neurons according to their significance. This provides further insight into the type of reservoir characteristics needed for a given task, and is supported by further reservoir measures of non-linearity and memory. These techniques are demonstrated on artificial and real world data.

Structural health monitoring of a footbridge using Echo State Networks and NARMAX

Abstract Echo State Networks (ESNs) and a Nonlinear Auto-Regressive Moving Average model with eXogenous inputs (NARMAX) have been applied to multi-sensor time-series data arising from a test footbridge which has been subjected to multiple potentially damaging interventions. The aim of the work was to automatically classify known potentially damaging events, while also allowing engineers to observe and localise any long term damage trends. The techniques reported here used data from ten temperature sensors as inputs and were tasked with predicting the output signal from eight tilt sensors embedded at various points over the bridge. Initially, interventions were identified by both ESNs and NARMAX. In addition, training ESNs using data up to the first event, and determining the ESNs’ subsequent predictions, allowed inferences to be made not only about when and where the interventions occurred, but also the level of damage caused, without requiring any prior data pre-processing or extrapolation. Finally, ESNs were successfully used as classifiers to characterise various different types of intervention that had taken place.

Advances in Biologically Inspired Reservoir Computing

The interplay between randomness and optimization has always been a major theme in the design of neural networks [3]. In the last 15 years, the success of reservoir computing (RC) showed that, in many scenarios, the algebraic structure of the recurrent component is far more important than the precise fine-tuning of its weights. As long as the recurrent part of the network possesses a form of fading memory of the input, the dynamics of the neurons are enough to efficiently process many spatio-temporal signals, provided that their activations are sufficiently heterogeneous. Even if today it is feasible to fully optimize deep recurrent networks, their implementation still requires a vast degree of experience and practice, not to mention vast computational resources, limiting their applicability in simpler architectures (e.g., embedded systems) or in areas where time is of key importance (e.g., online systems). Not surprisingly, then, RC remains a powerful tool for quickly solving

Artificial Neural Network Analysis of Volatile Organic Compounds for the Detection of Lung Cancer

Lung cancer is a widespread disease and it is well understood that systematic, non-invasive and early detection of this progressive and life-threatening disorder is of vital importance for patient outcomes. In this work we present a convergence of familiar and less familiar artificial neural network techniques to help address this task. Our preliminary results demonstrate that improved, automated, early diagnosis of lung cancer based on the classification of volatile organic compounds detected in the exhaled gases of patients seems possible. Under strictly controlled conditions, using Selected Ion Flow Tube Mass Spectrometry (SIFT-MS), the naturally occurring concentrations of a range of volatile organic compounds in the exhaled gases of 20 lung cancer patients and 20 healthy individuals provided the dataset that has been analysed. We investigated the performance of several artificial neural network architectures, each with complementary pattern recognition properties, from the domains of supervised, unsupervised and recurrent neural networks. The neural networks were trained on a subset of the data, with their performance evaluated using unseen test data and classification accuracies ranging from 56% to 74% were obtained. In addition, there is promise that the topological ordering properties of the unsupervised networks’ clusters will be able to provide further diagnostic insights, for example into patients who may have been heavy smokers but so far have not presented with any lung cancer. With the collection of data from a larger number of subjects across a long time period there is promise that an automated assistive tool in the diagnosis of lung cancer via breath analysis could soon be possible.

Adult fly age estimations using cuticular hydrocarbons and Artificial Neural Networks in forensically important Calliphoridae species

Blowflies (Diptera: Calliphoridae) are forensically important as they are known to be one of the first to colonise human remains. The larval stage is typically used to assist a forensic entomologists with adult flies rarely used as they are difficult to age because they remain morphologically similar once they have gone through the initial transformation upon hatching. However, being able to age them is of interest and importance within the field. This study examined the cuticular hydrocarbons (CHC) of Diptera: Calliphoridae species Lucilia sericata, Calliphora vicina and Calliphora vomitoria. The CHCs were extracted from the cuticles of adult flies and analysed using Gas Chromatography-Mass Spectrometry (GC-MS). The chemical profiles were examined for the two Calliphora species at intervals of day 1, 5, 10, 20 and 30 and up to day 10 for L. sericata. The results show significant chemical changes occurring between the immature and mature adult flies over the extraction period examined in this study. With the aid of a Principal Component Analysis (PCA) and Artificial Neural Networks (ANN), samples were seen to cluster, allowing for the age to be established within the aforementioned time frames. The use of ANNs allowed for the automatic classification of novel samples with very good performance. This was a proof of concept study, which developed a method allowing to age post-emergence adults by using their chemical profiles.

Locally Excited State – Charge Transfer State Coupled Dyes as Optically Responsive Neuron Firing Probes

A selection of NIR-optically responsive neuron probes was produced comprising of a donor julolidyl group connected to a BODIPY core and several different styryl and vinylpyridinyl derived acceptor moieties. The strength of the donor-acceptor interaction was systematically modulated by altering the electron withdrawing nature of the aryl unit. The fluorescence quantum yield was observed to decrease as the electron withdrawing effect of the aryl subunit increased in line with changes of the Hammett parameter. The effectiveness of these fluorophores as optically responsive dyes for neuronal imaging was assessed by measuring the toxicity and signal-to-noise ratio (SNR) of each dye. A great improvement of SNR was obtained when compared to the first-generation BODIPY-based voltage sensitive dyes with concomitant toxicity decrease. The mechanism for the optical response is disparate from conventional cyanine-based dyes, opening up a new way to produce effective voltage sensitive dyes that respond well into the NIR region.