Robert G. Gallager

Low-density parity-check codes

A low-density parity-check code is a code specified by a parity-check matrix with the following properties: each column contains a small fixed number j \geq 3 of ls and each row contains a small fixed number k > j of ls. The typical minimum distance of these codes increases linearly with block length for a fixed rate and fixed j . When used with maximum likelihood decoding on a sufficiently quiet binary-input symmetric channel, the typical probability of decoding error decreases exponentially with block length for a fixed rate and fixed j . A simple but nonoptimum decoding scheme operating directly from the channel a posteriori probabilities is described. Both the equipment complexity and the data-handling capacity in bits per second of this decoder increase approximately linearly with block length. For j > 3 and a sufficiently low rate, the probability of error using this decoder on a binary symmetric channel is shown to decrease at least exponentially with a root of the block length. Some experimental results show that the actual probability of decoding error is much smaller than this theoretical bound.

Information Theory and Reliable Communication

Communication Systems and Information Theory. A Measure of Information. Coding for Discrete Sources. Discrete Memoryless Channels and Capacity. The Noisy-Channel Coding Theorem. Techniques for Coding and Decoding. Memoryless Channels with Discrete Time. Waveform Channels. Source Coding with a Fidelity Criterion. Index.

A Distributed Algorithm for Minimum-Weight Spanning Trees

Abstract : A distributed algorithm is presented that constructs the minimum weight spanning tree in a connected undirected graph with distinct edge weights. A processor exists at each node of the graph, knowing initially only the weights of the adjacent edges. The processors obey the same algorithm and exchange messages with neighbors until the tree is constructed. The total number of messages required for a graph of N nodes and E edges is at most 5N log of N to the base 2 + 2E and a message contains at most one edge weight plus log of 8N to the base 2 bits. The algorithm can be initiated spontaneously at any node or at any subset of nodes.

A Minimum Delay Routing Algorithm Using Distributed Computation

An algorithm is defined for establishing routing tables in the individual nodes of a data network. The routing table at a node i specifies, for each other node j , what fraction of the traffic destined for node j should leave node i on each of the links emanating from node i . The algorithm is applied independently at each node and successively updates the routing table at that node based on information communicated between adjacent nodes about the marginal delay to each destination. For stationary input traffic statistics, the average delay per message through the network converges, with successive updates of the routing tables, to the minimum average delay over all routing assignments. The algorithm has the additional property that the traffic to each destination is guaranteed to be loop free at each iteration of the algorithm. In addition, a new global convergence theorem for noncontinuous iteration algorithms is developed.

Efficient Modulation for Band-Limited Channels

This paper attempts to present a comprehensive tutorial survey of the development of efficient modulation techniques for bandlimited channels, such as telephone channels. After a history of advances in commercial high-speed modems and a discussion of theoretical limits, it reviews efforts to optimize two-dimensional signal constellations and presents further elaborations of uncoded modulation. Its principal emphasis, however, is on coded modulation techniques, in which there is an explosion of current interest, both for research and for practical application. Both block-coded and trellis-coded modulation are covered, in a common framework. A few new techniques are presented.

A perspective on multiaccess channels

The information theoretic approach and the collision resolution approach to multiaccess channels are reviewed in terms of the underlying communication problems that both are modeling. Some perspective on the strengths and weakness of these approaches is given, and the need of a more combined approach focused on coding and decoding techniques is argued.

A simple derivation of the coding theorem and some applications

Upper bounds are derived on the probability of error that can be achieved by using block codes on general time-discrete memoryless channels. Both amplitude-discrete and amplitude-continuous channels are treated, both with and without input constraints. The major advantages of the present approach are the simplicity of the derivations and the relative simplicity of the results; on the other hand, the exponential behavior of the bounds with block length is the best known for all transmission rates between 0 and capacity. The results are applied to a number of special channels, including the binary symmetric channel and the additive Gaussian noise channel.

Variations on a theme by Huffman

In honor of the twenty-fifth anniversary of Huffman coding, four new results about Huffman codes are presented. The first result shows that a binary prefix condition code is a Huffman code iff the intermediate and terminal nodes in the code tree can be listed by nonincreasing probability so that each node in the list is adjacent to its sibling. The second result upper bounds the redundancy (expected length minus entropy) of a binary Huffman code by P_{1}+ \log_{2}[2(\log_{2}e)/e]=P_{1}+0.086 , where P_{1} is the probability of the most likely source letter. The third result shows that one can always leave a codeword of length two unused and still have a redundancy of at most one. The fourth result is a simple algorithm for adapting a Huffman code to slowly varying esthnates of the source probabilities. In essence, one maintains a running count of uses of each node in the code tree and lists the nodes in order of these counts. Whenever the occurrence of a message increases a node count above the count of the next node in the list, the nodes, with their attached subtrees, are interchanged.

Lower bounds to error probability for coding on discrete memoryless channels. II.

New lower bounds are presented for the minimum error probability that can be achieved through the use of block coding on noisy discrete memoryless channels. Like previous upper bounds, these lower bounds decrease exponentially with the block length N. The coefficient of N in the exponent is a convex function of the rate. From a certain rate of transmission up to channel capacity, the exponents of the upper and lower bounds coincide. Below this particular rate, the exponents of the upper and lower bounds differ, although they approach the same limit as the rate approaches zero. Examples are given and various incidental results and techniques relating to coding theory are developed. The paper is presented in two parts: the first, appearing here, summarizes the major results and treats the case of high transmission rates in detail; the second, to appear in the subsequent issue, treats the case of low transmission rates.

The Gaussian parallel relay network

We introduce the real, discrete-time Gaussian parallel relay network. This simple network is theoretically important in the context of network information theory. We present upper and lower bounds to capacity and explain where they coincide.