Alok Ahuja

Lossless Compression of Ultraspectral Sounder Data

The compression of ultraspectral sounder data is better to be lossless or near-lossless to avoid potential degradation of the geophysical retrieval in the associated ill-posed problem. Transform-based, prediction-based, and clustering-based methods for lossless compression of ultraspectral sounder data have been presented. It is shown that the compression ratios of ultraspectral sounder data via the standard state-of-the-art algorithms (e.g. 3D SPIHT, 2D JPEG2000, 2D CALIC, 2D JPEG-LS etc.) can be significantly improved when combining the BAR preprocessing scheme. We also report the promising compression results for the ultraspectral sounder data using various approaches such as lossless PCA and Predictive Partitioned Vector Quantization (PPVQ).

Data compression studies for NOAA Hyperspectral Environmental Suite (HES) using 3D integer wavelet transforms with 3D set partitioning in hierarchical trees

The next-generation NOAA/NESDIS GOES-R hyperspectral sounder, now referred to as the HES (Hyperspectral Environmental Suite), will have hyperspectral resolution (over one thousand channels with spectral widths on the order of 0.5 wavenumber) and high spatial resolution (less than 10 km). Hyperspectral sounder data is a particular class of data requiring high accuracy for useful retrieval of atmospheric temperature and moisture profiles, surface characteristics, cloud properties, and trace gas information. Hence compression of these data sets is better to be lossless or near lossless. Given the large volume of three-dimensional hyperspectral sounder data that will be generated by the HES instrument, the use of robust data compression techniques will be beneficial to data transfer and archive. In this paper, we study lossless data compression for the HES using 3D integer wavelet transforms via the lifting schemes. The wavelet coefficients are processed with the 3D set partitioning in hierarchical trees (SPIHT) scheme followed by context-based arithmetic coding. SPIHT provides better coding efficiency than Shapiros original embedded zerotree wavelet (EZW) algorithm. We extend the 3D SPIHT scheme to take on any size of 3D satellite data, each of whose dimensions need not be divisible by 2N, where N is the levels of the wavelet decomposition being performed. The compression ratios of various kinds of wavelet transforms are presented along with a comparison with the JPEG2000 codec.

Fast precomputed VQ with optimal bit allocation for lossless compression of ultraspectral sounder data

The compression of three-dimensional ultraspectral sounder data is a challenging task given its unprecedented size. We develop a fast precomputed vector quantization (FPVQ) scheme with optimal bit allocation for lossless compression of ultraspectral sounder data. The scheme consists of linear prediction, bit-depth partitioning, vector quantization, and optimal bit allocation. Linear prediction serves as a whitening tool to make the prediction residuals of each channel close to a Gaussian distribution, and then these residuals are partitioned based on bit depths. Each partition is further divided into several sub-partitions with various 2/sup k/ channels for vector quantization. Only the codebooks with 2/sup m/ codewords for 2/sup k/-dimensional normalized Gaussian distributions are precomputed. A new algorithm is developed for optimal bit allocation among subpartitions. Unlike previous algorithms that may yield a sub-optimal solution, the proposed algorithm guarantees to find the minimum of the cost function under the constraint of a given total bit rate. Numerical experiments upon the NASA AIRS data show that the FPVQ scheme gives high compression ratios for lossless compression of ultraspectral sounder data.

Predictive partitioned vector quantization for hyperspectral sounder data compression

The compression of three-dimensional hyperspectral sounder data is a challenging task given its unprecedented size and nature. Vector quantization (VQ) is explored for the compression of this hyperspectral sounder data. The high dimensional vectors are partitioned into subvectors to reduce codebook search and storage complexity in coding of the data. The partitions are made by use of statistical properties of the sounder data in the spectral dimension. Moreover, the data is decorrelated at first to make it better suited for vector quantization. Due to the data characteristics, the iterative codebook generation procedure converges much faster and also leads to a better reconstruction of the sounder data. For lossless compression of the hyperspectral sounder data, the residual error and the quantization indices are entropy coded. The independent vector quantizers for different partitions make this scheme practical for compression of the large volume 3D hyperspectral sounder data.

Lossless data compression for infrared hyperspectral sounders: an update

The compression of hyperspectral sounder data is beneficial for more efficient archive and transfer given its large 3-D volume. Moreover, since physical retrieval of geophysical parameters from hyperspectral sounder data is a mathematically ill-posed problem that is sensitive to the error of the data, lossless or near-lossless compression is desired. This paper provides an update into applications of state-of-the-art 2D and 3D lossless compression algorithms such as 3D EZW, 3D SPIHT, 2D JPEG2000, 2D JPEG-LS and 2D CALIC for hyperspectral sounder data. In addition, in order to better explore the correlations between the remote spectral regions affected by the same type of atmospheric absorbing constituents or clouds, the Bias-Adjusted Reordering (BAR) scheme is presented which reorders the data such that the bias-adjusted distance between any two neighboring vectors is minimized. This scheme coupled with any of the state-of-the-art compression algorithms produces significant compression gains.

Wavelet lossless compression of ultraspectral sounder data

This paper provides a study concerning the suitability of well-known image coding techniques originally devised for lossy compression of still natural images when applied to lossless compression of ultraspectral sounder data. An ultraspectral sounder generates an unprecedented amount of 3D data, consisting of two spatial and one spectral dimensions; with ultraspectral sounder data, better inference of atmospheric, cloud and surface parameters is feasible. Here we present the experimental results of five widespread wavelet-based coding techniques, namely EZW, IC, SPIHT, JPEG2000 and CCSDS-IDC. Since the considered still image coding techniques are 2D in nature, but the ultraspectral data is 3D, a preprocessing step is applied to convert the two spatial dimensions into a single one. We are also interested in analyzing the benefits of applying some pre-processing step (e.g., linear prediction or bias adjusted reordering) prior to the coding process in order to further exploit the spectral correlation, which is much stronger than the spatial correlation.

Ultraspectral sounder data compression using error-detecting reversible variable-length coding

Nonreversible variable-length codes (e.g. Huffman coding, Golomb-Rice coding, and arithmetic coding) have been used in source coding to achieve efficient compression. However, a single bit error during noisy transmission can cause many codewords to be misinterpreted by the decoder. In recent years, increasing attention has been given to the design of reversible variable-length codes (RVLCs) for better data transmission in error-prone environments. RVLCs allow instantaneous decoding in both directions, which affords better detection of bit errors due to synchronization losses over a noisy channel. RVLCs have been adopted in emerging video coding standards--H.263+ and MPEG-4--to enhance their error-resilience capabilities. Given the large volume of three-dimensional data that will be generated by future space-borne ultraspectral sounders (e.g. IASI, CrIS, and HES), the use of error-robust data compression techniques will be beneficial to satellite data transmission. In this paper, we investigate a reversible variable-length code for ultraspectral sounder data compression, and present its numerical experiments on error propagation for the ultraspectral sounder data. The results show that the RVLC performs significantly better error containment than JPEG2000 Part 2.

Effects of the starting channel for spectral reordering on the lossless compression of 3D ultraspectral sounder data

The unprecedented size of ultraspectral sounder data makes its compression a challenging task. Ultraspectral sounder data features strong correlations in disjoint spectral regions affected by the same type of absorbing gases. Previously, we proposed a reordering scheme to better explore these correlations of the ultraspectral sounder data. With this preprocessing scheme, the state-of-the-art compression algorithms such as CALIC, JPEG-LS and JPEG2000 significantly improve the compression ratios up to 15% on average. In this paper, we investigate the effects of different starting channels for spectral reordering on the lossless compression of 3D ultraspectral sounder data obtained from Atmospheric Infrared Sounder (AIRS) observations. It is shown that the compression ratios and reordering indices are dependent on the choice of the starting channel for reordering.

Lossless data compression studies for NOAA hyperspectral environmental suite using 3D integer wavelet transforms with 3D embedded zerotree coding

Hyperspectral sounder data is a particular class of data that requires high accuracy for useful retrieval of atmospheric temperature and moisture profiles, surface characteristics, cloud properties, and trace gas information. Therefore compression of these data sets is better to be lossless or near lossless. The next-generation NOAA/NESDIS GOES-R hyperspectral sounder, now referred to as the HES (Hyperspectral Environmental Suite), will have hyperspectral resolution (over one thousand channels with spectral widths on the order of 0.5 wavenumber) and high spatial resolution (less than 10 km). Given the large volume of three-dimensional hyperspectral sounder data that will be generated by the HES instrument, the use of robust data compression techniques will be beneficial to data transfer and archive. In this paper, we study lossless data compression for the HES using 3D integer wavelet transforms via the lifting schemes. The wavelet coefficients are then processed with the 3D embedded zerotree wavelet (EZW) algorithm followed by context-based arithmetic coding. We extend the 3D EZW scheme to take on any size of 3D satellite data, each of whose dimensions need not be divisible by 2N, where N is the levels of the wavelet decomposition being performed. The compression ratios of various kinds of wavelet transforms are presented along with a comparison with the JPEG2000 codec.

Real-Time DSP Implementation of 3D Wavelet Reversible Variable-length Coding for Ultraspectral Sounder Data Compression

Reversible variable-length codes (RVLCs) allow instantaneous decoding in both directions, which affords better detection of bit errors due to synchronization losses over a noisy channel. Earlier, we developed 3D wavelet reversible variable- length coding (3DWT-RVLC) for lossless compression of ultraspectral sounder data, which has significantly better error resilience than JPEG2000 Part 2 at only a small reduction in compression gain (1). To explore the feasibility of 3DWT-RVLC for real-time satellite onboard processing, we implement a memory-limited DSP version of 3DWT-RVLC. Experimental results for the 10 AIRS ultraspectral test granules show that the DSP-based 3DWT-RVLC yields an average compression ratio of 2.36, comparable to 2.51 of the original 3DWT-RVLC from our previous work.