Todd K. Moon

The expectation-maximization algorithm

A common task in signal processing is the estimation of the parameters of a probability distribution function. Perhaps the most frequently encountered estimation problem is the estimation of the mean of a signal in noise. In many parameter estimation problems the situation is more complicated because direct access to the data necessary to estimate the parameters is impossible, or some of the data are missing. Such difficulties arise when an outcome is a result of an accumulation of simpler outcomes, or when outcomes are clumped together, for example, in a binning or histogram operation. There may also be data dropouts or clustering in such a way that the number of underlying data points is unknown (censoring and/or truncation). The EM (expectation-maximization) algorithm is ideally suited to problems of this sort, in that it produces maximum-likelihood (ML) estimates of parameters when there is a many-to-one mapping from an underlying distribution to the distribution governing the observation. The EM algorithm is presented at a level suitable for signal processing practitioners who have had some exposure to estimation theory.

Joint signal detection and parameter estimation in multiuser communications

The problem of simultaneously detecting the information bits and estimating signal amplitudes and phases in a K-user asynchronous direct-sequence spread-spectrum multiple-access communication system is addressed. The joint maximum-likelihood (ML) estimator has a computational complexity that is exponential in the total number of bits transmitted and thus does not represent a practical solution to the problem. An estimator that combines a suboptimum tree-search algorithm with a recursive least-squares estimator of complex signal amplitude is considered. The complexity of this estimator is O(K/sup 2/) computations per decoded bit, and its performance is very close to that of the joint ML receiver. This receiver has the advantage that the transmitted signal powers and phases are extracted from the received signal in an adaptive fashion without using a test sequence. >

Parameter estimation in a multi-user communication system

Many of the previously-developed multiple-access detection algorithms have assumed knowledge of the time delay, the amplitude, and the phase of each user. This paper presents methods of estimating the amplitudes and phases, both when the time delays are known and when the delays are unknown. In neither case are the transmitted bits assumed to be known. The estimators are unbiased and consistent, and each has a computational complexity linear in the number of users. Simulations are provided to demonstrate performance. >

Multiple Constraint Satisfaction by Belief Propagation: An Example Using Sudoku

The popular Sudoku puzzle bears structural resemblance to the problem of decoding linear error correction codes: solution is over a discrete set, and several constraints apply. We express the constraint satisfaction using a Tanner graph. The belief propagation algorithm is applied to this graph. Unlike conventional computer-based solvers, which rely on humanly specified tricks for solution, belief propagation is generally applicable, and requires no human insight to solve a problem. The presence of short cycles in the graph creates biases so that not every puzzle is solved by this method. However, all puzzles are at least partly solved by this method. The Sudoku application thus demonstrates the potential effectiveness of BP algorithms on a general class of constraint satisfaction problems

Sinkhorn Solves Sudoku

The Sudoku puzzle is a discrete constraint satisfaction problem, as is the error correction decoding problem. We propose here an algorithm for solution to the Sinkhorn puzzle based on Sinkhorn balancing. Sinkhorn balancing is an algorithm for projecting a matrix onto the space of doubly stochastic matrices. The Sinkhorn balancing solver is capable of solving all but the most difficult puzzles. A proof of convergence is presented, with some information theoretic connections. A random generalization of the Sudoku puzzle is presented, for which the Sinkhorn-based solver is also very effective.

Entropy Minimization for Solving Sudoku

Solving Sudoku puzzles is formulated as an optimization problem over a set of probabilities. The constraints for a given puzzle translate into a convex polyhedral feasible set for the probabilities. The solution to the puzzle lies at an extremal point of the polyhedron where the probabilities are either zero or one and the entropy is zero. Because the entropy is positive at all other feasible points, an entropy minimization approach is adopted to solve Sudoku. To escape local entropy minima at nonsolution extremal points, a search procedure is proposed in which each iteration involves solving a simple convex optimization problem. This approach is evaluated on thousands of puzzles spanning four levels of difficulty from “easy” to “evil”.

Average acquisition time for SSMA channels

The acquisition time performance of a two-user spread-spectrum multiple-access system is examined. The acquisition is accomplished using conventional (single-user) correlating/energy-detecting acquisition methods. The intent of the analysis is to determine how much performance penalty is incurred because of the presence of the multiple-user interference. The analysis indicates that for good spreading codes the performance degradation is not severe when the users amplitudes are all similar. However, in the presence of significant near/far power differences, the acquisition time increases dramatically. This suggests the need for acquisition methods which explicitly account for the presence of multiple users.<<ETX>>

Parameter estimation for a bilinear time series model

A direct approach to the estimation of the parameters associated with a bilinear time series model is presented. The approach depends critically on the expressions for certain higher-order statistics of the signals that satisfy the bilinear model. These expressions are linear in most of the parameters of the model. The parameters are then estimated from an overdetermined set of equations. Results of an experiment that employs this technique and demonstrates its good properties are included.<<ETX>>

A generalized LDPC decoder for blind turbo equalization

The equations for iteratively decoding low-density parity-check (LDPC) codes are generalized to compute joint probabilities of arbitrary sets of codeword bits and parity checks. The standard iterative LDPC decoder, which computes single variable probabilities, is realized as a special case. Another specialization allows pairwise joint posterior probabilities of pairs of codeword bits to be computed. These pairwise joint probabilities are used in an expectation-maximization (EM) based blind channel estimator that is ignorant of the code constraints. Channel estimates are input to a turbo equalizer that exploits the structure of the LDPC code. Feeding pairwise posterior probabilities back to the channel estimator eliminates the need to average across time for channel estimation. Therefore, this scheme can be used to equalize very long codewords, even when the channel is time varying. This blind turbo equalizer is evaluated through computer simulations and found to perform as well as a channel-informed turbo equalizer but with approximately twice the number of turbo iterations.

Similarity methods in signal processing

A signal may contain information that is preserved by certain transformations of the signal. For example, the information phase-modulated signal is not altered by amplitude scaling of the signal. Many processing techniques have been developed to exploit such similarities. In the past, these algorithms have been developed in isolation without regard to common principles of invariance that tie them together. Similarity methods are presented as a unified method of designing processing algorithms invariant to specified transformations. These methods are based upon groups of continuous transformations known as local Lie groups and lead to a quasilinear partial differential equation. Solution of this partial differential equation specifies the form the signal processing operations must take. This form can then be applied using engineering judgment for algorithmic implementation. The paper presents an extended tutorial on Lie groups and similarity methods and quasilinear differential equations drawn from the mathematical literature. This is followed by several examples of signal processing interest that demonstrate that the similarity techniques may be applicable in certain kinds of signal processing problems.