Francisco Ortega-Zamorano

FPGA Implementation of the C-Mantec Neural Network Constructive Algorithm

Competitive majority network trained by error correction (C-Mantec), a recently proposed constructive neural network algorithm that generates very compact architectures with good generalization capabilities, is implemented in a field programmable gate array (FPGA). A clear difference with most of the existing neural network implementations (most of them based on the use of the backpropagation algorithm) is that the C-Mantec automatically generates an adequate neural architecture while the training of the data is performed. All the steps involved in the implementation, including the on-chip learning phase, are fully described and a deep analysis of the results is carried on using the two sets of benchmark problems. The results show a clear increase in the computation speed in comparison to the standard personal computer (PC)-based implementation, demonstrating the usefulness of the intrinsic parallelism of FPGAs in the neurocomputational tasks and the suitability of the hardware version of the C-Mantec algorithm for its application to real-world problems.

Layer multiplexing FPGA implementation for deep back-propagation learning

Training of large scale neural networks, like those used nowadays in Deep Learning schemes, requires long computational times or the use of high performance computation solutions like those based on cluster computation, GPU boards, etc. As a possible alternative, in this work the Back-Propagation learning algorithm is implemented in an FPGA board using a multiplexing layer scheme, in which a single layer of neurons is physically implemented in parallel but can be reused any number of times in order to simulate multi-layer architectures. An on-chip implementation of the algorithm is carried out using a training/validation scheme in order to avoid overfitting effects. The hardware implementation is tested on several configurations, permitting to simulate architectures comprising up to 127 hidden layers with a maximum number of neurons in each layer of 60 neurons. We confirmed the correct implementation of the algorithm and compared the computational times against C and Matlab code executed in a multicore supercomputer, observing a clear advantage of the proposed FPGA scheme. The layer multiplexing scheme used provides a simple and flexible approach in comparison to standard implementations of the Back-Propagation algorithm representing an important step towards the FPGA implementation of deep neural networks, one of the most novel and successful existing models for prediction problems.

Smart sensor/actuator node reprogramming in changing environments using a neural network model

The techniques currently developed for updating software in sensor nodes located in changing environments require usually the use of reprogramming procedures, which clearly increments the costs in terms of time and energy consumption. This work presents an alternative to the traditional reprogramming approach based on an on-chip learning scheme in order to adapt the node behaviour to the environment conditions. The proposed learning scheme is based on C-Mantec, a novel constructive neural network algorithm especially suitable for microcontroller implementations as it generates very compact size architectures. The Arduino UNO board was selected to implement this learning algorithm as it is a popular, economic and efficient open source single-board microcontroller. C-Mantec has been successfully implemented in a microcontroller board by adapting it in order to overcome the limitations imposed by the limited resources of memory and computing speed of the hardware device. Also, this work brings an in-depth analysis of the solutions adopted to overcome hardware resource limitations in the learning algorithm implementation (e.g., data type), together with an efficiency assessment of this approach when the algorithm is tested on a set of circuit design benchmark functions. Finally, the utility, efficiency and versatility of the system is tested in three different-nature case studies in which the environmental conditions change its behaviour over time.

Efficient Implementation of the Backpropagation Algorithm in FPGAs and Microcontrollers

The well-known backpropagation learning algorithm is implemented in a field-programmable gate array (FPGA) board and a microcontroller, focusing in obtaining efficient implementations in terms of a resource usage and computational speed. The algorithm was implemented in both cases using a training/validation/testing scheme in order to avoid overfitting problems. For the case of the FPGA implementation, a new neuron representation that reduces drastically the resource usage was introduced by combining the input and first hidden layer units in a single module. Further, a time-division multiplexing scheme was implemented for carrying out product computations taking advantage of the built-in digital signal processor cores. In both implementations, the floating-point data type representation normally used in a personal computer (PC) has been changed to a more efficient one based on a fixed-point scheme, reducing system memory variable usage and leading to an increase in computation speed. The results show that the modifications proposed produced a clear increase in computation speed in comparison with the standard PC-based implementation, demonstrating the usefulness of the intrinsic parallelism of FPGAs in neurocomputational tasks and the suitability of both implementations of the algorithm for its application to the real world problems.

High precision FPGA implementation of neural network activation functions

The efficient implementation of artificial neural networks in FPGA boards requires tackling several issues that strongly affect the final result. One of these issues is the computation of the neurons activation function. In this work, a detailed analysis of the FPGA implementations of the Sigmoid and Exponential functions is carried out, in a approach combining a lookup table with a linear interpolation procedure. Further, to optimize board resources utilization, a time division multiplexing of the multiplier attached to the neurons was used. The results are evaluated in terms of the absolute and relative errors obtained and also through measuring a quality factor and the resource utilization, showing a clear improvement in relationship to previously published works.

Smart motion detection sensor based on video processing using self-organizing maps

A low cost smart motion detector is presented.It is based on the Arduino DUE microcontroller.The software architecture employs a fixed point arithmetic paradigm.The self-organizing map neural network is implemented on chip.The performance is substantially higher than that of the traditional detector. Most current approaches to computer vision are based on expensive, high performance hardware to meet the heavy computational requirements of the employed algorithms. These system architectures are severely limited in their practical application due to financial and technical limitations. In this work a different strategy is used, namely the development of an inexpensive and easy to deploy computer vision system for motion detection. This is achieved by three means. First of all, an affordable and flexible hardware platform is employed. Secondly, the motion detection algorithm is specifically tailored to involve a very small computational load. Thirdly, a fixed point programming paradigm is followed in implementing the system so as to further reduce the computational requirements. The proposed system is experimentally compared to the standard motion detector for a wide range of benchmark videos. The reported results indicate that our proposal attains substantially better performance, while it remains affordable and easy to install in practice.

Implementation of the C-mantec neural network constructive algorithm in an arduino uno microcontroller

A recently proposed constructive neural network algorithm, named C-Mantec, is fully implemented in a Arduino board. The C-Mantec algorithm generates very compact size neural architectures with good prediction abilities, and thus the board can be potentially used to learn on-site sensed data without needing to transmit information to a central control unit. An analysis of the more difficult steps of the implementation is detailed, and a test is carried out on a set of benchmark functions normally used in circuit design to show the correct functioning of the implementation.

FPGA Implementation Comparison Between C-Mantec and Back-Propagation Neural Network Algorithms

Recent advances in FPGA technology have permitted the implementation of neurocomputational models, making them an interesting alternative to standard PCs in order to speed up the computations involved taking advantage of the intrinsic FPGA parallelism. In this work, we analyse and compare the FPGA implementation of two neural network learning algorithms: the standard Back-Propagation algorithm and C-Mantec, a constructive neural network algorithm that generates compact one hidden layer architectures. One of the main differences between both algorithms is the fact that while Back-Propagation needs a predefined architecture, C-Mantec constructs its network while learning the input patterns. Several aspects of the FPGA implementation of both algorithms are analysed, focusing in features like logic and memory resources needed, transfer function implementation, computation time, etc. Advantages and disadvantages of both methods are discussed in the context of their application to benchmark problems.

Committee C-mantec: a probabilistic constructive neural network

C-Mantec is a recently introduced constructive algorithm that generates compact neural architectures with good generalization abilities. Nevertheless, it produces a discrete output value and this might be a drawback in certain situations. We propose in this work two approaches in order to obtain a continuous output network such as the output can be interpreted as the probability of a given pattern to belong to one of the output classes. The CC-Mantec approach utilizes a committee strategy and the results obtained both with the XOR Boolean function and with a set of benchmark functions shows the suitability of the approach, as an improvement over the standard C-Mantec algorithm is obtained in almost all cases.

Unsupervised learning by cluster quality optimization

Abstract Most clustering algorithms are designed to minimize a distortion measure which quantifies how far the elements of the clusters are from their respective centroids. The assessment of the results is often carried out with the help of cluster quality measures which take into account the compactness and separation of the clusters. However, these measures are not amenable to optimization because they are not differentiable with respect to the centroids even for a given set of clusters. Here we propose a differentiable cluster quality measure, and an associated clustering algorithm to optimize it. It turns out that the standard k-means algorithm is a special case of our method. Experimental results are reported with both synthetic and real datasets, which demonstrate the performance of our approach with respect to several standard quantitative measures.