Harry R. Erwin

Towards multimodal neural robot learning

Abstract Learning by multimodal observation of vision and language offers a potentially powerful paradigm for robot learning. Recent experiments have shown that ‘mirror’ neurons are activated when an action is being performed, perceived, or verbally referred to. Different input modalities are processed by distributed cortical neuron ensembles for leg, arm and head actions. In this overview paper we consider this evidence from mirror neurons by integrating motor, vision and language representations in a learning robot.

2009 Special Issue: Robotic sound-source localisation architecture using cross-correlation and recurrent neural networks

In this paper we present a sound-source model for localising and tracking an acoustic source of interest along the azimuth plane in acoustically cluttered environments, for a mobile service robot. The model we present is a hybrid architecture using cross-correlation and recurrent neural networks to develop a robotic model accurate and robust enough to perform within an acoustically cluttered environment. This model has been developed with considerations of both processing power and physical robot size, allowing for this model to be deployed on to a wide variety of robotic systems where power consumption and size is a limitation. The development of the system we present has its inspiration taken from the central auditory system (CAS) of the mammalian brain. In this paper we describe experimental results of the proposed model including the experimental methodology for testing sound-source localisation systems. The results of the system are shown in both restricted test environments and in real-world conditions. This paper shows how a hybrid architecture using band pass filtering, cross-correlation and recurrent neural networks can be used to develop a robust, accurate and fast sound-source localisation model for a mobile robot.

Auditory robotic tracking of sound sources using hybrid cross-correlation and recurrent networks

This paper describes an auditory robotic system capable of computing the angle of incidence of a sound source on the horizontal plane (azimuth). The system, with the use of an Elman type recurrent neural network (RNN), is able to dynamically track this sound source as it changes azimuthally within the environment. The RNN is used to enable fast tracking responses to the overall system over a set time, as opposed to waiting for the next sound position before moving. The system is first tested in a simulated environment and then these results are compared with testing on the robotic system. The results show that the development of a hybrid system incorporating cross-correlation and recurrent neural networks is an effective mechanism for the control of a robot that tracks sound sources azimuthally.

A biologically inspired spiking neural network model of the auditory midbrain for sound source localisation

This paper proposes a spiking neural network (SNN) of the mammalian subcortical auditory pathway to achieve binaural sound source localisation. The network is inspired by neurophysiological studies on the organisation of binaural processing in the medial superior olive (MSO), lateral superior olive (LSO) and the inferior colliculus (IC) to achieve a sharp azimuthal localisation of a sound source over a wide frequency range. Three groups of artificial neurons are constructed to represent the neurons in the MSO, LSO and IC that are sensitive to interaural time difference (ITD), interaural level difference (ILD) and azimuth angle (@q), respectively. The neurons in each group are tonotopically arranged to take into account the frequency organisation of the auditory pathway. To reflect the biological organisation, only ITD information extracted by the MSO is used for localisation of low frequency (<1kHz) sounds; for sound frequencies between 1 and 4kHz the model also uses ILD information extracted by the LSO. This information is combined in the IC model where we assume that the strengths of the inputs from the MSO and LSO are proportional to the conditional probability of P(@q|ITD) or P(@q|ILD) calculated based on the Bayes theorem. The experimental results show that the addition of ILD information significantly increases sound localisation performance at frequencies above 1kHz. Our model can be used to test different paradigms for sound localisation in the mammalian brain, and demonstrates a potential practical application of sound localisation for robots.

Multiple Sound Source Localisation in Reverberant Environments Inspired by the Auditory Midbrain

This paper proposes a spiking neural network (SNN) of the mammalian auditory midbrain to achieve binaural multiple sound source localisation. The network is inspired by neurophysiological studies on the organisation of binaural processing in the medial superior olive (MSO), lateral superior olive (LSO) and the inferior colliculus (IC) to achieve a sharp azimuthal localisation of sound sources over a wide frequency range in a reverberant environment. Three groups of artificial neurons are constructed to represent the neurons in the MSO, LSO and IC that are sensitive to interaural time difference (ITD), interaural level difference (ILD) and azimuth angle respectively. The ITD and ILD cues are combined in the IC to estimate the azimuth direction of a sound source. To deal with echo, we propose an inter-inhibited onset network in the IC, which can extract the azimuth information from the direct path sound and avoid the effects of reverberation. Experiments show that the proposed onset cell network can localise two sound sources efficiently taking into account the room reverberation.

A Biologically Inspired Spiking Neural Network for Sound Localisation by the Inferior Colliculus

We introduce a biologically inspired azimuthal sound localisation system, which simulates the functional organisation of the human auditory midbrain up to the inferior colliculus (IC). Supported by recent neurophysiological studies on the role of the IC and superior olivary complex (SOC) in sound processing, our system models two ascending pathways of the auditory midbrain: the ITD (Interaural Time Difference) pathway and ILD (Interaural Level Difference) pathway. In our approach to modelling the ITD pathway, we take into account Yins finding that only a single delay line exists in the ITD processing from cochlea to SOC for the ipsilateral ear while multiple delay lines exists for the contralateral ear. The ILD pathway is modelled without varied delay lines because of neurophysiological evidence that indicates the delays along that pathway are minimal and constant. Level-locking auditory neurons are introduced for the ILD pathway network to encode sound amplitude into spike sequence, that are similar to the phase-locking auditory neurons which encode time information to the ITD pathway. A leaky integrate-and-fire spiking neural model is adapted to simulate the neurons in the SOC that process ITD and ILD. Experimental results show that our model performs sound localisation that approaches biological performance. Our approach brings not only new insight into the brain mechanism of the auditory system, but also demonstrates a practical application of sound localisation for mobile robots.

Mobile robot broadband sound localisation using a biologically inspired spiking neural network

A biologically inspired azimuthal broadband sound localisation system is introduced to simulates the functional organisation of the human auditory midbrain up to the inferior colliculus (IC). Supported by recent neurophysiological studies on the role of the IC and superior olivary complex (SOC) in sound processing, our system models two ascending pathways of the auditory midbrain: the ITD (Interaural Time Difference) pathway and ILD (Interaural Level Difference) pathway. In our approach to modelling the ITD pathway, we take account of Yinpsilas finding that only a single delay line exists in the ITD processing from cochlea to SOC for the ipsilateral ear while multiple delay lines exists for the contralateral ear. The ILD pathway is modelled without varied delay lines because of neurophysiological evidence that indicates the delays along that pathway are minimal and constant. First, two-dimensional (2D) tonotopical ITD and ILD spike maps over frequency and ITD/ILD are calculated by a spiking neural network which follows the biological delay structure. Then these maps are weighted considering the advance of ITD in low frequency and ILD in middle and high frequency. Finally, ITD and ILD maps are merged together to find out the best estimation of the sound source. Experimental results involving noise and voice show that our model performs sound localisation that approaches biological performance. Our approach brings not only new insight into the brain mechanism of the auditory system, but also demonstrates a practical application of sound localisation for mobile robots.

Bioinspired Auditory Sound Localisation for Improving the Signal to Noise Ratio of Socially Interactive Robots

In this paper we describe a bioinspired hybrid architecture for acoustic sound source localisation and tracking to increase the signal to noise ratio (SNR) between speaker and background sources for a socially interactive robots speech recogniser system. The model presented incorporates the use of interaural time difference for azimuth estimation and recurrent neural networks for trajectory prediction. The results are then presented showing the difference in the SNR of a localised and non-localised speaker source, in addition to presenting the recognition rates between a localised and non-localised speaker source. From the results presented in this paper it can be seen that by orientating towards the sound source of interest the recognition rates of that source can be increased

A neural network classifier for notch filter classification of sound-source elevation in a mobile robot

An important aspect of all robotic systems is sensing and there are many sensing modalities used including vision, tactile, olfactory and acoustics to name a few. This paper presents a robotic system for sensing in acoustics, specifically in elevation localization. The model presented is a two-stage model incorporating spectral analysis using artificial pinna and an artificial neural network for classification and elevation estimation. The spectral classifier uses notch filters to analyze changes in attenuation of certain frequencies with elevation. This paper shows how using the spectral output of a signal generated by an artificial pinna can be classified by a feed-forward backpropagation neural network to estimate the elevation of a sound-source.

Threat modeling using formal methods: A new approach to develop secure web applications

Software security problems exist since the early days of computer systems. Operating system level approaches, network level approaches and machine level approaches are not sufficient for the security of software systems. Software security has gained attention in the recent years as an internal security issue of software systems as compared to external protective measures. Threat modeling is a technique being used to model threats into software systems. By applying threat modeling at the early stages of software development life cycle, all possible threats to software systems can be identified and mitigated and hence in this way, a more secure software application can be developed. Various threat modeling approaches such as CLASP, SDL, STRIDE, DREAD, TAM and Touch Points are being used by many organizations for threat modeling into software systems. But all of the approaches being used for threat modeling are based on informal and semi formal techniques. Formal methods are based on mathematics and provide state of the art techniques for secure software development. Formal methods had been used successfully in many critical systems such as CICS, Paris Railway System and British Air Traffic Control System etc. In the proposed approach VDM++, will be used for specification of core components: STRIDE, DREAD and Security Mechanisms. VDM++Tools will be used for type checking and proof obligations.