Eiji Mizutani

Levenberg-Marquardt method for ANFIS learning

Presents the results of applying the Levenberg-Marquardt method (K. Levenberg, 1944, and D.W. Marquardt, 1963), which is a popular nonlinear least-squares method, to the ANFIS (Adaptive Neuro-Fuzzy Inference System) architecture proposed by Jang (IEEE Trans. on Systems, Man and Cybernctics, vol. 23, no. 3, pp 665-685, May 1993). Through empirical studies, we discuss the strengths and weaknesses of using such an efficient nonlinear regression technique for neuro-fuzzy modeling, and explain the tradeoffs between mapping precision and membership function interpretability.

Coactive neural fuzzy modeling

We discuss the neuro-fuzzy modeling and learning mechanisms of CANFIS (coactive neuro-fuzzy inference system) wherein both neural networks and fuzzy systems play active roles together in an effort to reach a specific goal. Their mutual dependence presents unexpected learning capabilities. CANFIS has extended the basic ideas of its predecessor ANFIS (adaptive network-based fuzzy inference system): the ANFIS concept has been extended to any number of input-output pairs. In addition, CANFIS yields advantages from nonlinear fuzzy rules. In light of some model-related limitations, this paper serves to highlight both neuro-fuzzy learning capacities and practical obstacles encountered in performing neuro-fuzzy modeling.

Powell's dogleg trust-region steps with the quasi-Newton augmented Hessian for neural nonlinear least-squares learning

This paper highlights Powells dogleg trust-region algorithms with self-scaling quasi-Newton Hessian augmentation for neural-network (NN) nonlinear least squares problems. The dogleg algorithms approximate a restricted Levenberg-Marquardt step within the trust region of the local quadratic model in a piecewise-linear fashion. Furthermore, the second-derivative term of the Hessian is approximated by quasi-Newton iteration to obtain augmented Gauss-Newton model Hessian, which may be useful for highly nonlinear residuals when starting with a poor initial point (i.e., randomly initialized weight parameters). By small-scale examples, we illustrate how those devices come into play as a promising NN learning algorithm.

Evolving color recipes

This paper highlights an evolutionary computing intelligence for a computerized color recipe prediction that requires function approximation and combinatorial solution of colorants to produce color recipes for a given target color sample. We attack this real challenging problem in the color (paint) industry by using an evolutionary computing system that consists of a problem-specific knowledge and three principal constituents of soft-computing: neural networks, a fuzzy system, and a genetic algorithm. Departing from the recipe results obtained by neural networks (NN) approaches, the evolutionary system attempts to improve them in conjunction with fuzzy classification, a knowledge base and neural fitness functions. All components function synergistically in obtaining precise color recipe outputs through simulation of color paint manufacturing process. Such computational intelligence can be useful, especially when an exact mathematical model of the real-world process under consideration is not available explicitly.

Second-order backpropagation algorithms for a stagewise-partitioned separable Hessian matrix

Recent advances in computer technology allows the implementation of some important methods that were assigned lower priority in the past due to their computational burdens. Second-order backpropagation (BP) is such a method that computes the exact Hessian matrix of a given objective function. We describe two algorithms for feed-forward neural-network (NN) learning with emphasis on how to organize Hessian elements into a so-called stagewise-partitioned block-arrow matrix form: (1) stagewise BP, an extension of the discrete-time optimal-control stagewise Newton of Dreyfus 1966; and (2) nodewise BP, based on direct implementation of the chain rule for differentiation attributable to Bishop 1992. The former, a more systematic and cost-efficient implementation in both memory and operation, progresses in the same layer-by-layer (i.e., stagewise) fashion as the widely-employed first-order BP computes the gradient vector. We also show intriguing separable structures of each block in the partitioned Hessian, disclosing the rank of blocks.

Stagewise Newton, differential dynamic programming, and neighboring optimum control for neural-network learning

The theory of optimal control is applied to multi-stage (i.e., multiple-layered) neural-network (NN) learning for developing efficient second-order algorithms, expressed in NN notation. In particular, we compare differential dynamic programming, neighboring optimum control, and stagewise Newton methods. Understanding their strengths and weaknesses would prove useful in pursuit of an effective intermediate step between the steepest descent and the Newton directions, arising in supervised NN-learning as well as reinforcement learning with function approximators.

Color device characterization of electronic cameras by solving adaptive networks nonlinear least squares problems

This paper describes an application of adaptive networks (AN) techniques to color device characterization, which can achieve color correction and compensation on the images taken by electronic imaging devices. We shall demonstrate ANs nonlinear mapping capability and its generalization capacity in characterizing electronic cameras for enhancing color quality.

On using discretized Cohen-Grossberg node dynamics for model-free actor-critic neural learning in non-Markovian domains

We describe how multi-stage non-Markovian decision problems can be solved using actor-critic reinforcement learning by assuming that a discrete version of Cohen-Grossberg node dynamics describes the node-activation computations of neural network (NN). Our NN is capable of rendering the process Markovian implicitly and automatically in a totally model-free fashion without learning by how much the state apace must be augmented so that the Markov property holds. This serves as an alternative to using Elman or Jordan-type function as a history memory in order to develop sensitivity to non-Markovian dependencies. We shall demonstrate our concept using a small-scale non-Markovian deterministic path problem, in which our actor-critic NN finds an optimal sequence of actions, although it needs much iteration due to the nature of neural model-free learning. This is, in spirit, a neuro-dynamic programming approach.

Coactive neuro-fuzzy modelling for colour recipe prediction

Explores neuro-fuzzy approaches to computerized colour recipe prediction which relates surface spectral reflectance of a target colour to several colorant proportions. The approaches are expressed within the framework of CANFIS (co-active neuro-fuzzy inference system) where both neural networks (NNs) and fuzzy systems (FSs) play active roles together in pursuit of a given task. To find an ideal adaptive model for this problem, the authors have investigated a variety of structures, they feature knowledge-embedded architectures and an adaptive FS, which serves to determine colour selection. They have enormous potential for augmenting prediction capability.

Road lane marker extraction by motion-detector CNNs

This paper describes an application of motion-detector CNN (cellular nonlinear/neural networks) to road lane-marker extraction for automobile position assessment. We assume that an automated vehicle drives down a freeway by tracking lane markers. A vehicle vision system, while taking road snapshots, needs to extract lane marker information from sequential road images. This feature-extraction task requires fast image processing to timely adjust vehicle heading under changing road conditions. Since the CNN has been employed for a variety of image processing, we have tested a motion-detector CNN for the lane-marker extraction. We shall demonstrate the power of the motion-detector CNN and present its current limitations, as well as its promising possibilities. We believe this application example may help pave the way for future autonomous vehicle control.