Robert J. Safranek

Signal compression based on models of human perception

The notion of perceptual coding, which is based on the concept of distortion masking by the signal being compressed, is developed. Progress in this field as a result of advances in classical coding theory, modeling of human perception, and digital signal processing, is described. It is proposed that fundamental limits in the science can be expressed by the semiquantitative concepts of perceptual entropy and the perceptual distortion-rate function, and current compression technology is examined in that framework. Problems and future research directions are summarized. >

Deinterlacing by successive approximation

We propose an algorithm for deinterlacing of interlaced video sequences. It successively builds approximations to the deinterlaced sequence by weighting various interpolation methods. A particular example given here uses four interpolation methods, weighted according to the errors each one introduces. Due to weighting, it is an adaptive algorithm. It is also time-recursive, since the motion-compensated part uses the previously interpolated frame. Furthermore, bidirectional motion estimation and compensation allow for better performance in the case of scene changes and covering/uncovering of objects. Experiments are run both on real-world and computer generated sequences. Finally, subjective testing is performed to evaluate the quality of the algorithm.

Matching and retrieval based on the vocabulary and grammar of color patterns

While it is recognized that images are described through color, texture and shapes of objects in the scene, general image understanding is still difficult. Thus, to perform image retrieval in a human-like manner one has to choose a specific domain, understand how users achieve similarity within that domain and then build a system that duplicates human performance. Since color and texture are fundamental aspects of human perception we propose a set of techniques for retrieval of color patterns. To determine how humans judge similarity of color patterns we performed a subjective study. Based on the results of the study five most relevant visual categories for the perception of pattern similarity were identified. We also determined the hierarchy of rules governing the use of these categories. Based on these results we designed a system which accepts one or more texture images as input, and depending on the query, produces a set of choices that follow human behavior in pattern matching. Processing steps in our model follow those of the human visual system, resulting in perceptually based features and distance measures. As expected, search results closely correlate with human choices.

The vocabulary and grammar of color patterns

We determine the basic categories and the hierarchy of rules used by humans in judging similarity and matching of color patterns. The categories are: (1) overall color; (2) directionality and orientation; (3) regularity and placement; (4) color purity; (5) complexity and heaviness. These categories form the pattern vocabulary which is governed by the grammar rules. Both the vocabulary and the grammar were obtained as a result of a subjective experiment. Experimental data were interpreted using multidimensional scaling techniques yielding the vocabulary and the hierarchical clustering analysis, yielding the grammar rules. Finally, we give a short overview of the existing techniques that can be used to extract and measure the elements of the vocabulary.

Methods for matching compressed video to ATM networks

Over the last few years three technologies have reached the stage of maturation where then can become synergistic. These are wideband, high speed networking, high quality video compression (MPEG-I and II), and high capacity affordable digital storage media. This paper addresses the interaction of these three technologies. In particular, it examines the problem of taking a compressed video data stream that is stored on a server, and transmitting it over an ATM channel which has a capacity smaller than that required by the data stream. The conventional approach to this problem would be to transcode by decoding the video data, and then re-encoding so as to meet the channel constraints. Currently this is not a cost effective solution since, while MPEG decoders are relatively inexpensive, encoders are not. Our approach to this problem is to partially decompress the video bitstream. Then, perform the transcoding in the quantized data domain. Finally, a valid bitstream is reassembled and transmitted. This approach has the advantage of providing nearly identical quality as the traditional transcoding approach, at a fraction of the hardware cost.

JPEG compliant encoder utilizing perceptually based quantization

The international JPEG (Joint Photographics Experts Group) standards for image compression deal with the compression of still images. It specifies the information contained in the compressed bit stream, and a decoder architecture that can reconstruct an image from the data in the bit stream. However, the exact implementation of the encoder is not standardized. The only requirement on the encoder is that it generate a compliant bit stream. This provides an opportunity to introduce new research results. The challenge in improving these standards based codecs is to generate a compliant bitstream that produces a perceptually equivalent image as the baseline system that has a higher compression ratio. This results in a lower encoded bit rate without perceptual loss in quality. The proposed encoder uses the perceptual model developed by Johnston and Safranek to determine, based on the input data, which coefficients are perceptually irrelevant. This information is used to remove (zero out) some coefficients before they are input to the quantizer block. This results in a larger percentage of zero codewords at the output of the quantizer which reduces the entropy of the resulting codewords.

A codec for HDTV

A high-quality digital video codec has been developed for the Zenith/AT&T HDTV system. It adaptively selects between two transmission modes with differing rates and robustness. The codec works on an image progressively scanned with 1575 scan lines every 1/30th of a second and achieves a compression ratio of approximately 50 to 1. The high compression ratio facilitates robust transmission of the compressed HDTV signal within an NTSC taboo channel. Transparent image quality is achieved using motion compensated transform coding coupled with a perceptual criterion to determine the quantization accuracy required for each transform coefficient. The codec has been designed to minimize complexity and memory in the receiver. >

Comparison of the coding efficiency of perceptual models

Over the past several years there have been many attempts to incorporate perceptual masking models into image compression systems. Unfortunately, there is little or no information on how these models perform in comparison to each other. The purpose of this paper is to examine how two different perceptual models perform when utilized in the came coding system. The models investigated are the Johnson-Safranek and the Watson. They both develop a contrast masking threshold for DCT based coders. The coder used for comparison is the Baseline Sequential mode of JPEG. Each model was implemented and used to generate image dependent masking thresholds for each 8x8 pixel block in the image. These thresholds were used to zero out perceptually irrelevant coefficients, while the remaining coefficients were quantized using a perceptually optimal quantization matrix. Both objective and subjective performance data was gathered. Bit rate saving versus standard JPEG was computed, and a subjective comparison of images encoded with both models and the nonperceptual JPEG was run. The perceptually based coders gave greater compression with no loss in subjective image quality.

Perceptually tuned sub-band image coder

In this paper we present a sub-band coder for true color images that uses an empirically derived perceptual masking model to set the allowable quantization noiselevel not only for each sub-band but also for each pixel in a given sub-band. The input image is converted into YIQ space and each channel is passed through a separable Generalized Quadrature Mirror Filterbank (GQMF). This separates the images frequency content into into 4 equal width bands in both the horizontal and vertical dimension, resulting in a representation consisting of 16 sub-bands for each channel. Using this representation, a perceptual masking model is derived for each channel. The model incorporates spatial-frequency sensitivity, contrast sensitivity, and texture masking. Based on the image dependent information in each sub-band and the perceptual masking model, noise-level targets are computed for each point in a subband. These noise-level targets are used to set the quantization levels in a DPCM quantizer. The output from the DPCM quantizer is then encoded, using an entropybased coding scheme, in either lxi , 1x2, or 2x2 pixel parts, based on the the statistics in each 4x4 sub-block of a particular sub-band. One set of codebooks, consisting of 100,000 entries, is used for all images. A block elimination algorithm takes advantage of the peaky spatial energy distribution of sub-bands to avoid using bits for quiescent parts of a given sub-band. The resultant bitrate depends on the complexity of the input image. For the images we use, high quality output requires bitrates from 0.25 to 1 .25 bits/pixel, while nearly transparent quality requires 0.5 to 2.5 bits/pixel.

A High Quality Digital HDTV Codec

A digital video codec has been developed for the Zenith/AT&T HDTV (high-definition TV) system for terrestrial broadcast over NTSC taboo channels. The codec works on an image progressively scanned with 1575 scan lines every 1/30th of a second and achieves a compression ratio of approximately 50 to 1. The transparent image quality is achieved using motion-compensated transform coding coupled with a perceptual criterion to determine the quantization accuracy required for each transform coefficient. The combination of a sophisticated encoded video format and advanced bit error protection techniques results in a highly robust reception and decoding of the compressed video signal. >