Kazuhiko Ozeki

Dependency structure analysis as combinatorial optimization

This paper is concerned with a problem of selecting optimal word sequences from a word lattice: a problem which often appears in automatic recognition of spoken language. Based on the notion of dependency grammar, a measure of favorability for dependency structures is defined. Then the problem is formulated as that of combinatorial optimization to find the dependency structures which maximize the favorability. A set of recurrence equations are derived, which indicate that the problem allows application of the principle of dynamic programming. The equations lead to a two-phase algorithm which finds optimal word sequences and their optimal dependency structures simultaneously in polynomial time. The order of time complexity is O(M2N5), where M is the number of candidate words in each time segment and N is the length of the word lattice. The algorithm can be considered as a twofold extension of the conventional dependency analyzer. A parallel and layered computation structure to implement the algorithm is also presented.

Noise Reduction in Time Domain Using Referential Reconstruction

We present a novel approach for single-channel noise reduction of speech signals contaminated by additive noise. In this approach, the system requires speech samples to be uttered in advance by the same speaker as that of the input signal. Speech samples used in this method must have enough phonetic variety to reconstruct the input signal. In the proposed method, which we refer to as referential reconstruction, we have used a small database created from examples of speech, which will be called reference signals. Referential reconstruction uses an example-based approach, in which the objective is to find the candidate speech frame which is the most similar to the clean input frame without noise, although the input frame is contaminated with noise. When candidate frames are found, they become final outputs without any special processing. In order to find the candidate frames, a correlation coefficient is used as a similarity measure. Through automatic speech recognition experiments, the proposed method was shown to be effective, particularly for low-SNR speech signals corrupted with white noise or noise in high-frequency bands. Since the direct implementation of this method requires infeasible computational cost for searching through reference signals, a coarse-to-fine strategy is introduced in this paper.

A polynomial-order algorithm for optimal phrase sequence selection from a phrase lattice and its parallel layered implementation

This paper deals with a problem of selecting an optimal phrase sequence from a phrase lattice, which is often encountered in language processing such as word processing and post-processing for speech recognition. The problem is formulated as one of combinatorial optimization and a polynomial order algorithm is derived. This algorithm finds an optimal phrase sequence and its dependency structure simultaneously, and is therefore particularly suited for an interface between speech recognition and various language processing. What the algorithm does is numerical optimization rather than symbolic operation unlike conventional parsers. A parallel and layered structure to implement the algorithm is also presented. Although the language taken up here is Japanese, the algorithm can be extended to cover a wider family of languages.

Kernel Affine Projection Algorithm

The unknown system to be identified by an adaptive filter is usually assumed to be a linear system. Based on this assumption, we model the unknown system by a linear filter. In reality, however, there are cases where a linear filter is inadequate. In spite of this problem, using a very general nonlinear filter is not a good idea. To construct an adaptation algorithm for a general nonlinear filter is not simple. Moreover, an adaptive model that has too many free parameters is not desirable from a machine learning theoretic point of view, because such a model exhibits poor generalization. In this chapter, we make a review of a work that extends the APA by the kernel trick so that it is applicable to identification of a nonlinear system. We start with the kernel perceptron as a simple example to show how the kernel trick is used to extend the perceptron so that it can learn a nonlinear discriminant function without losing the simplicity of the original linear structure. Then, noting that the kernel trick replaces the inner product with the kernel function, we extend the APA to the kernel APA. It is seen that the kernel APA has a similar structure with the resource-allocating network. In the perceptron, the training data set is finite and fixed. In the APA, on the other hand, the set of training data, i.e., the set of regressors, is infinite. To keep the set of regressors actually used in adaptation to be finite, we sieve the regressors by the novelty criterion that checks if a newly arrived regressor is informative enough for adaptation.

Variable Parameter APAs

The standard APA has three parameters: the step-size parameter, the projection order, and the regularization factor, which are all assumed to be invariable in the preceding chapters. However, by adjusting those parameters adaptively, it is possible to improve the performance of the APA. This chapter gives an overview of such variable parameter APAs. First, two variable step-size APAs are reviewed. The second topic is the proportionate APA, in which each component of the coefficient vector of the adaptive filter is given a step-size that is approximately proportional to its own magnitude. The proportionate technique is effective for echo cancellation, where the impulse response of the unknown system is sparse and long. The third topic is the evolving order APA. The projection order gives a definite influence on the behavior of the APA, i.e., a larger projection order results in faster convergence with a larger misadjustment. In the evolving order APA, the projection order is so adjusted that it takes a large value in the initial convergence phase, and a small value in the steady-state phase. The last topic is the variable regularized APA. Two methods are described, one based on minimization of the a posteriori error, and the other using a variable regularization matrix. In the latter method, the eigenvalues of the matrix to be regularized are given different regularization parameters individually.

Affine Projection Algorithm

The normalized least-mean-squares (NLMS) algorithm has a problem that the convergence slows down for correlated input signals. The reason for this phenomenon is explained by looking at the algorithm from a geometrical point of view. This observation motivates the affine projection algorithm (APA) as a natural generalization of the NLMS algorithm. The APA exploits most recent multiple regressors, while the NLMS algorithm uses only the current, single regressor. In the APA, the current coefficient vector is orthogonally projected onto the affine subspace defined by the regressors for updating the coefficient vector. By increasing the number of regressors, which is called the projection order, the convergence rate of the APA is improved especially for correlated input signals. The role of the step-size is made clear. Investigations from the affine projection point of view give us a deep insight into the properties of the APA. We also see that alternative approaches are possible to derive the update equation for the APA. To stabilize the numerical inversion of a matrix in the update equation, a regularization term is often added. This variant of the APA is called the regularized APA (R-APA), whereas the original APA is called the basic APA (B-APA). This chapter also explains that the B-APA with unity step-size has a decorrelating property, and that there are formal similarities between the recursive least-squares (RLS) algorithm and the R-APA.

Convergence Behavior of APA

The behavior of an adaptive filter depends on the input signal. However, since there are infinitely many variations of signals, it is difficult to draw general, useful conclusions on the behavior of an adaptive filter taking the waveform of each signal into consideration. Therefore, we often resort to statistical approach, where the signals appearing in the update equation are replaced with random variables. Even in this stochastic framework, we need many assumptions on the statistical properties of those random variables to make the analysis tractable. We are concerned with the behavior of the expectations of the error signal and the squared norm of the coefficient error vector. We are also interested in stability condition on the range of the step-size. In the first part of this chapter, the fundamental behavior of the B-APA is discussed. Then, we review two works based on simplifying assumptions on regressors. One of those works assumes that a regressor can take only one of finite number of orientations. Although this assumption is unrealistic, the analysis shows many of important properties of the B-APA. The other work assumes that the input signal is an autoregressive process, and that the step-size equals unity. This work uses the update equation developed for the D-APA. The last work we review in this chapter gives a general treatment for the convergence behavior of the R-APA. Based on the energy conservation relation and the weighted energy conservation relation, it yields useful results without extreme assumptions.

Classical Adaptation Algorithms

In this chapter, two classical adaptation algorithms, the least-mean-squares algorithm (LMS algorithm) and the normalized least-mean-squares algorithm (NLMS algorithm), are reviewed. The LMS algorithm, also known as the Widrow-Hoff algorithm, is motivated by the steepest-descent method to minimize the expected error. The LMS algorithm has a parameter called the step-size that appears in the process of replacing differential with finite difference. By adjusting the step-size, the convergence rate and the convergence error at steady-state are controlled. The NLMS algorithm is an improved version of the LMS algorithm, in which the correction term to update the coefficients of the filter is normalized by the squared norm of the current regressor. In the LMS algorithm, the effective value of the step-size varies depending on the volume of the input signal. In the NLMS algorithm, on the other hand, the step-size has a definite meaning that is independent of the volume of the input signal. The NLMS algorithm can be looked at from a geometrical point of view. In fact, when the step-size equals unity, the updated coefficient vector is the orthogonal projection of the current coefficient vector onto the hyperplane defined by the current regressor. The geometrical interpretation of the NLMS algorithm leads to the affine projection algorithm (APA), the main theme of this book.

Reduction of Computational Complexity

If a naive method is employed to implement the APA, the number of arithmetic operations per sample is approximately given by a polynomial \(\alpha p^{3} + \beta np^{2}\), where \(\alpha \) and \(\beta \) are positive coefficients, n is the filter length, and p is the projection order. This amount of computation is a heavy burden for real-time applications of the APA. The data used to update the coefficient vector in the APA have a time-shift property, i.e., the data used at time \(k-1\) and those at time k are largely overlapped. In this chapter, we review two works on reduction of the computational complexity of the APA by exploiting this time-shift property: the fast affine projection (FAP) algorithm and the block exact fast affine projection (BEFAP) algorithm. The key point of the FAP algorithm is recursive computations of the quantities that appear in the update equation for the coefficient vector. The amount of computation for the FAP algorithm per sample is approximately given by a linear function of n and p. By blocking the input data and lower the frequency of updating the coefficient vector, one can reduce the computational complexity per sample. However, the behavior of this algorithm is different from the original APA. In the BEFAP algorithm, a correction term is introduced so that the output becomes exactly the same as the original APA. In the algorithm, the product of a Hankel matrix and a vector appears, which can be computed efficiently either by the convolution method using the FFT, or by a divide and conquer method. We see that the BEFAP algorithm further reduces the computational complexity compared with the FAP algorithm.

Family of Affine Projection Algorithms

After the birth of the basic affine projection algorithm (B-APA) in the middle of 1980s, several adaptive filtering algorithms that also exploit multiple regressors were put forward independently. They are now recognized as variants of the B-APA, forming a family of affine projection algorithms. In the family, there are such algorithms as the regularized affine projection algorithm (R-APA), the decorrelation affine projection algorithm (D-APA), the partial-rank algorithm (PRA), the normalized least-mean-squares algorithm with orthogonal correction factors (NLMS-OCF), the binormalized data-reusing least-mean-squares algorithm (BNDR-LMS). The R-APA and the D-APA are discussed in the preceding chapter. If we update the coefficient vector every p samples instead of every sample, where p is the projection order, then we obtain the PRA. By applying the Gram–Schmidt orthogonalization to the regressors, the B-APA is transformed into the NLMS-OCF. The BNDR-LMS is just a special case of the B-APA where the projection order equals 2. We can also use sparse regressors in the APAs. These algorithms are formulated by a single update equation with several parameters. By setting those parameters at appropriate values, the update equation for each of the algorithms in the APA family is expressed.