Nikola Kasabov

DENFIS: dynamic evolving neural-fuzzy inference system and its application for time-series prediction

This paper introduces a new type of fuzzy inference systems, denoted as dynamic evolving neural-fuzzy inference system (DENFIS), for adaptive online and offline learning, and their application for dynamic time series prediction. DENFIS evolve through incremental, hybrid (supervised/unsupervised), learning, and accommodate new input data, including new features, new classes, etc., through local element tuning. New fuzzy rules are created and updated during the operation of the system. At each time moment, the output of DENFIS is calculated through a fuzzy inference system based on m-most activated fuzzy rules which are dynamically chosen from a fuzzy rule set. Two approaches are proposed: (1) dynamic creation of a first-order Takagi-Sugeno-type fuzzy rule set for a DENFIS online model; and (2) creation of a first-order Takagi-Sugeno-type fuzzy rule set, or an expanded high-order one, for a DENFIS offline model. A set of fuzzy rules can be inserted into DENFIS before or during its learning process. Fuzzy rules can also be extracted during or after the learning process. An evolving clustering method (ECM), which is employed in both online and offline DENFIS models, is also introduced. It is demonstrated that DENFIS can effectively learn complex temporal sequences in an adaptive way and outperform some well-known, existing models.

Evolving fuzzy neural networks for supervised/unsupervised online knowledge-based learning

This paper introduces evolving fuzzy neural networks (EFuNNs) as a means for the implementation of the evolving connectionist systems (ECOS) paradigm that is aimed at building online, adaptive intelligent systems that have both their structure and functionality evolving in time. EFuNNs evolve their structure and parameter values through incremental, hybrid supervised/unsupervised, online learning. They can accommodate new input data, including new features, new classes, etc., through local element tuning. New connections and new neurons are created during the operation of the system. EFuNNs can learn spatial-temporal sequences in an adaptive way through one pass learning and automatically adapt their parameter values as they operate. Fuzzy or crisp rules can be inserted and extracted at any time of the EFuNN operation. The characteristics of EFuNNs are illustrated on several case study data sets for time series prediction and spoken word classification. Their performance is compared with traditional connectionist methods and systems. The applicability of EFuNNs as general purpose online learning machines, what concerns systems that learn from large databases, life-long learning systems, and online adaptive systems in different areas of engineering are discussed.

HyFIS: adaptive neuro-fuzzy inference systems and their application to nonlinear dynamical systems

This paper proposes an adaptive neuro-fuzzy system, HyFIS (Hybrid neural Fuzzy Inference System), for building and optimising fuzzy models. The proposed model introduces the learning power of neural networks to fuzzy logic systems and provides linguistic meaning to the connectionist architectures. Heuristic fuzzy logic rules and input-output fuzzy membership functions can be optimally tuned from training examples by a hybrid learning scheme comprised of two phases: rule generation phase from data; and rule tuning phase using error backpropagation learning scheme for a neural fuzzy system. To illustrate the performance and applicability of the proposed neuro-fuzzy hybrid model, extensive simulation studies of nonlinear complex dynamic systems are carried out. The proposed method can be applied to an on-line incremental adaptive learning for the prediction and control of nonlinear dynamical systems. Two benchmark case studies are used to demonstrate that the proposed HyFIS system is a superior neuro-fuzzy modelling technique.

Incremental linear discriminant analysis for classification of data streams

This paper presents a constructive method for deriving an updated discriminant eigenspace for classification when bursts of data that contains new classes is being added to an initial discriminant eigenspace in the form of random chunks. Basically, we propose an incremental linear discriminant analysis (ILDA) in its two forms: a sequential ILDA and a Chunk ILDA. In experiments, we have tested ILDA using datasets with a small number of classes and small-dimensional features, as well as datasets with a large number of classes and large-dimensional features. We have compared the proposed ILDA against the traditional batch LDA in terms of discriminability, execution time and memory usage with the increasing volume of data addition. The results show that the proposed ILDA can effectively evolve a discriminant eigenspace over a fast and large data stream, and extract features with superior discriminability in classification, when compared with other methods.

Evolving Intelligent Systems: Methodology and Applications

From theory to techniques, the first all-in-one resource for EIS There is a clear demand in advanced process industries, defense, and Internet and communication (VoIP) applications for intelligent yet adaptive/evolving systems. Evolving Intelligent Systems is the first self- contained volume that covers this newly established concept in its entirety, from a systematic methodology to case studies to industrial applications. Featuring chapters written by leading world experts, it addresses the progress, trends, and major achievements in this emerging research field, with a strong emphasis on the balance between novel theoretical results and solutions and practical real-life applications. Explains the following fundamental approaches for developing evolving intelligent systems (EIS): the Hierarchical Prioritized Structure the Participatory Learning Paradigm the Evolving Takagi-Sugeno fuzzy systems (eTS+) the evolving clustering algorithm that stems from the well-known Gustafson-Kessel offline clustering algorithm Emphasizes the importance and increased interest in online processing of data streams Outlines the general strategy of using the fuzzy dynamic clustering as a foundation for evolvable information granulation Presents a methodology for developing robust and interpretable evolving fuzzy rule-based systems Introduces an integrated approach to incremental (real-time) feature extraction and classification Proposes a study on the stability of evolving neuro-fuzzy recurrent networks Details methodologies for evolving clustering and classification Reveals different applications of EIS to address real problems in areas of: evolving inferential sensors in chemical and petrochemical industry learning and recognition in robotics Features downloadable software resources Evolving Intelligent Systems is the one-stop reference guide for both theoretical and practical issues for computer scientists, engineers, researchers, applied mathematicians, machine learning and data mining experts, graduate students, and professionals.

2013 Special Issue: Dynamic evolving spiking neural networks for on-line spatio- and spectro-temporal pattern recognition

On-line learning and recognition of spatio- and spectro-temporal data (SSTD) is a very challenging task and an important one for the future development of autonomous machine learning systems with broad applications. Models based on spiking neural networks (SNN) have already proved their potential in capturing spatial and temporal data. One class of them, the evolving SNN (eSNN), uses a one-pass rank-order learning mechanism and a strategy to evolve a new spiking neuron and new connections to learn new patterns from incoming data. So far these networks have been mainly used for fast image and speech frame-based recognition. Alternative spike-time learning methods, such as Spike-Timing Dependent Plasticity (STDP) and its variant Spike Driven Synaptic Plasticity (SDSP), can also be used to learn spatio-temporal representations, but they usually require many iterations in an unsupervised or semi-supervised mode of learning. This paper introduces a new class of eSNN, dynamic eSNN, that utilise both rank-order learning and dynamic synapses to learn SSTD in a fast, on-line mode. The paper also introduces a new model called deSNN, that utilises rank-order learning and SDSP spike-time learning in unsupervised, supervised, or semi-supervised modes. The SDSP learning is used to evolve dynamically the network changing connection weights that capture spatio-temporal spike data clusters both during training and during recall. The new deSNN model is first illustrated on simple examples and then applied on two case study applications: (1) moving object recognition using address-event representation (AER) with data collected using a silicon retina device; (2) EEG SSTD recognition for brain-computer interfaces. The deSNN models resulted in a superior performance in terms of accuracy and speed when compared with other SNN models that use either rank-order or STDP learning. The reason is that the deSNN makes use of both the information contained in the order of the first input spikes (which information is explicitly present in input data streams and would be crucial to consider in some tasks) and of the information contained in the timing of the following spikes that is learned by the dynamic synapses as a whole spatio-temporal pattern.

Learning fuzzy rules and approximate reasoning in fuzzy neural networks and hybrid systems

The paper considers both knowledge acquisition and knowledge interpretation tasks as tightly connected and continuously interacting processes in a contemporary knowledge engineering system. Fuzzy rules are used here as a framework for knowledge representation. An algorithm REFuNN for fuzzy rules extraction from adaptive fuzzy neural networks (FuNN) is proposed. A case study of Iris classification is chosen to illustrate the algorithm. Interpretation of fuzzy rules is possible by using fuzzy neural networks or by using standard fuzzy inference methods. Both approaches are compared in the paper based on the case example. A hybrid environment FuzzyCOPE which facilitates neural network simulation, fuzzy rules extraction from fuzzy neural networks and fuzzy rules interpretation by using different methods for approximate reasoning is briefly described.

Quantum-Inspired Evolutionary Algorithm: A Multimodel EDA

The quantum-inspired evolutionary algorithm (QEA) applies several quantum computing principles to solve optimization problems. In QEA, a population of probabilistic models of promising solutions is used to guide further exploration of the search space. This paper clearly establishes that QEA is an original algorithm that belongs to the class of estimation of distribution algorithms (EDAs), while the common points and specifics of QEA compared to other EDAs are highlighted. The behavior of a versatile QEA relatively to three classical EDAs is extensively studied and comparatively good results are reported in terms of loss of diversity, scalability, solution quality, and robustness to fitness noise. To better understand QEA, two main advantages of the multimodel approach are analyzed in details. First, it is shown that QEA can dynamically adapt the learning speed leading to a smooth and robust convergence behavior. Second, we demonstrate that QEA manipulates more complex distributions of solutions than with a single model approach leading to more efficient optimization of problems with interacting variables.

SPAN: SPIKE PATTERN ASSOCIATION NEURON FOR LEARNING SPATIO-TEMPORAL SPIKE PATTERNS

Spiking Neural Networks (SNN) were shown to be suitable tools for the processing of spatio-temporal information. However, due to their inherent complexity, the formulation of efficient supervised learning algorithms for SNN is difficult and remains an important problem in the research area. This article presents SPAN - a spiking neuron that is able to learn associations of arbitrary spike trains in a supervised fashion allowing the processing of spatio-temporal information encoded in the precise timing of spikes. The idea of the proposed algorithm is to transform spike trains during the learning phase into analog signals so that common mathematical operations can be performed on them. Using this conversion, it is possible to apply the well-known Widrow-Hoff rule directly to the transformed spike trains in order to adjust the synaptic weights and to achieve a desired input/output spike behavior of the neuron. In the presented experimental analysis, the proposed learning algorithm is evaluated regarding its learning capabilities, its memory capacity, its robustness to noisy stimuli and its classification performance. Differences and similarities of SPAN regarding two related algorithms, ReSuMe and Chronotron, are discussed.

Multiple Gene Expression Classifiers from Different Array Platforms Predict Poor Prognosis of Colorectal Cancer

Purpose: This study aimed to develop gene classifiers to predict colorectal cancer recurrence. We investigated whether gene classifiers derived from two tumor series using different array platforms could be independently validated by application to the alternate series of patients. Experimental Design: Colorectal tumors from New Zealand (n = 149) and Germany (n = 55) patients had a minimum follow-up of 5 years. RNA was profiled using oligonucleotide printed microarrays (New Zealand samples) and Affymetrix arrays (German samples). Classifiers based on clinical data, gene expression data, and a combination of the two were produced and used to predict recurrence. The use of gene expression information was found to improve the predictive ability in both data sets. The New Zealand and German gene classifiers were cross-validated on the German and New Zealand data sets, respectively, to validate their predictive power. Survival analyses were done to evaluate the ability of the classifiers to predict patient survival. Results: The prediction rates for the New Zealand and German gene-based classifiers were 77% and 84%, respectively. Despite significant differences in study design and technologies used, both classifiers retained prognostic power when applied to the alternate series of patients. Survival analyses showed that both classifiers gave a better stratification of patients than the traditional clinical staging. One classifier contained genes associated with cancer progression, whereas the other had a large immune response gene cluster concordant with the role of a host immune response in modulating colorectal cancer outcome. Conclusions: The successful reciprocal validation of gene-based classifiers on different patient cohorts and technology platforms supports the power of microarray technology for individualized outcome prediction of colorectal cancer patients. Furthermore, many of the genes identified have known biological functions congruent with the predicted outcomes.