Roger W. Heymann

Data compression studies for NOAA Hyperspectral Environmental Suite (HES) using 3D integer wavelet transforms with 3D set partitioning in hierarchical trees

The next-generation NOAA/NESDIS GOES-R hyperspectral sounder, now referred to as the HES (Hyperspectral Environmental Suite), will have hyperspectral resolution (over one thousand channels with spectral widths on the order of 0.5 wavenumber) and high spatial resolution (less than 10 km). Hyperspectral sounder data is a particular class of data requiring high accuracy for useful retrieval of atmospheric temperature and moisture profiles, surface characteristics, cloud properties, and trace gas information. Hence compression of these data sets is better to be lossless or near lossless. Given the large volume of three-dimensional hyperspectral sounder data that will be generated by the HES instrument, the use of robust data compression techniques will be beneficial to data transfer and archive. In this paper, we study lossless data compression for the HES using 3D integer wavelet transforms via the lifting schemes. The wavelet coefficients are processed with the 3D set partitioning in hierarchical trees (SPIHT) scheme followed by context-based arithmetic coding. SPIHT provides better coding efficiency than Shapiros original embedded zerotree wavelet (EZW) algorithm. We extend the 3D SPIHT scheme to take on any size of 3D satellite data, each of whose dimensions need not be divisible by 2N, where N is the levels of the wavelet decomposition being performed. The compression ratios of various kinds of wavelet transforms are presented along with a comparison with the JPEG2000 codec.

Lossless compression of three-dimensional hyperspectral sounder data using context-based adaptive lossless image codec with bias-adjusted reordering

Hyperspectral sounder data is used for retrieval of atmospheric temperature, moisture and trace gas profiles, surface temperature and emissivity, and cloud and aerosol optical properties. This large volume of data is 3-D in nature with many scan lines containing cross-track footprints, each with thousands of IR channels. Unlike hyperspectral imager data compression, hyperspectral sounder data compression is desired to be lossless or near-lossless to avoid substantial degradation of the geophysical retrieval. For this new class of data for compression studies, a lossless compression algorithm combining the context-based adaptive lossless image codec (CALIC) and a novel bias-adjusted reordering (BAR) scheme is presented. The 3-D data are arranged into two dimensions with the original 2-D spatial domain converted into one dimension using a continuous scan order. In the BAR scheme, the data are reordered such that the bias-adjusted distance between any two neighboring vectors is minimized. The result is then encoded using the CALIC algorithm with significant compression gains over using the CALIC algorithm alone.

Predictive partitioned vector quantization for hyperspectral sounder data compression

The compression of three-dimensional hyperspectral sounder data is a challenging task given its unprecedented size and nature. Vector quantization (VQ) is explored for the compression of this hyperspectral sounder data. The high dimensional vectors are partitioned into subvectors to reduce codebook search and storage complexity in coding of the data. The partitions are made by use of statistical properties of the sounder data in the spectral dimension. Moreover, the data is decorrelated at first to make it better suited for vector quantization. Due to the data characteristics, the iterative codebook generation procedure converges much faster and also leads to a better reconstruction of the sounder data. For lossless compression of the hyperspectral sounder data, the residual error and the quantization indices are entropy coded. The independent vector quantizers for different partitions make this scheme practical for compression of the large volume 3D hyperspectral sounder data.

Lossless data compression for infrared hyperspectral sounders: an update

The compression of hyperspectral sounder data is beneficial for more efficient archive and transfer given its large 3-D volume. Moreover, since physical retrieval of geophysical parameters from hyperspectral sounder data is a mathematically ill-posed problem that is sensitive to the error of the data, lossless or near-lossless compression is desired. This paper provides an update into applications of state-of-the-art 2D and 3D lossless compression algorithms such as 3D EZW, 3D SPIHT, 2D JPEG2000, 2D JPEG-LS and 2D CALIC for hyperspectral sounder data. In addition, in order to better explore the correlations between the remote spectral regions affected by the same type of atmospheric absorbing constituents or clouds, the Bias-Adjusted Reordering (BAR) scheme is presented which reorders the data such that the bias-adjusted distance between any two neighboring vectors is minimized. This scheme coupled with any of the state-of-the-art compression algorithms produces significant compression gains.

Ultraspectral sounder data compression using error-detecting reversible variable-length coding

Nonreversible variable-length codes (e.g. Huffman coding, Golomb-Rice coding, and arithmetic coding) have been used in source coding to achieve efficient compression. However, a single bit error during noisy transmission can cause many codewords to be misinterpreted by the decoder. In recent years, increasing attention has been given to the design of reversible variable-length codes (RVLCs) for better data transmission in error-prone environments. RVLCs allow instantaneous decoding in both directions, which affords better detection of bit errors due to synchronization losses over a noisy channel. RVLCs have been adopted in emerging video coding standards--H.263+ and MPEG-4--to enhance their error-resilience capabilities. Given the large volume of three-dimensional data that will be generated by future space-borne ultraspectral sounders (e.g. IASI, CrIS, and HES), the use of error-robust data compression techniques will be beneficial to satellite data transmission. In this paper, we investigate a reversible variable-length code for ultraspectral sounder data compression, and present its numerical experiments on error propagation for the ultraspectral sounder data. The results show that the RVLC performs significantly better error containment than JPEG2000 Part 2.

Effects of the starting channel for spectral reordering on the lossless compression of 3D ultraspectral sounder data

The unprecedented size of ultraspectral sounder data makes its compression a challenging task. Ultraspectral sounder data features strong correlations in disjoint spectral regions affected by the same type of absorbing gases. Previously, we proposed a reordering scheme to better explore these correlations of the ultraspectral sounder data. With this preprocessing scheme, the state-of-the-art compression algorithms such as CALIC, JPEG-LS and JPEG2000 significantly improve the compression ratios up to 15% on average. In this paper, we investigate the effects of different starting channels for spectral reordering on the lossless compression of 3D ultraspectral sounder data obtained from Atmospheric Infrared Sounder (AIRS) observations. It is shown that the compression ratios and reordering indices are dependent on the choice of the starting channel for reordering.

Lossless data compression studies for NOAA hyperspectral environmental suite using 3D integer wavelet transforms with 3D embedded zerotree coding

Hyperspectral sounder data is a particular class of data that requires high accuracy for useful retrieval of atmospheric temperature and moisture profiles, surface characteristics, cloud properties, and trace gas information. Therefore compression of these data sets is better to be lossless or near lossless. The next-generation NOAA/NESDIS GOES-R hyperspectral sounder, now referred to as the HES (Hyperspectral Environmental Suite), will have hyperspectral resolution (over one thousand channels with spectral widths on the order of 0.5 wavenumber) and high spatial resolution (less than 10 km). Given the large volume of three-dimensional hyperspectral sounder data that will be generated by the HES instrument, the use of robust data compression techniques will be beneficial to data transfer and archive. In this paper, we study lossless data compression for the HES using 3D integer wavelet transforms via the lifting schemes. The wavelet coefficients are then processed with the 3D embedded zerotree wavelet (EZW) algorithm followed by context-based arithmetic coding. We extend the 3D EZW scheme to take on any size of 3D satellite data, each of whose dimensions need not be divisible by 2N, where N is the levels of the wavelet decomposition being performed. The compression ratios of various kinds of wavelet transforms are presented along with a comparison with the JPEG2000 codec.

NOAA's future earth observations from advanced GOES satellites

Abstract IAF-95-B.2.01 This paper describes early progress made toward defining the next generation of GOES satellites. These satellites could be first launched in 2008. The paper is divided into four main sections. First, the actual process of starting a new satellite program is presented as it has occurred during the past two years. NOAA formed 12 internal teams to determine requirements and will contract with industry soon for feasibility studies. This real process in a complex agency and with many potential users is compared with the classic textbook approaches recommended for new program space starts by NASA/NOAA. These early results of requirements for GOES R are presented as the second section. Imager and sounder instruments currently on GOES are improved considerably. New instruments such as passive microwave, lightning mapper and ocean color are described. Finally, even new spacecraft configurations and more reliance on industry/government partnerships are suggested. Section three offers ideas about new spacecraft configurations for GOES N. These include both larger payload sizes or lighter payloads that might fly on lighter spacecraft. In addition, the use of commercial satellites for NOAA measurements is suggested. The fourth section gives advantages of geosynchronous orbit and the need for international participation in future geo-satellite missions. In a broader sense geo measurements will benefit the world and we welcome immediate and practical participation of interested parties in our planning.

Current status of ultraspectral sounder data compression

This paper presents current status of lossless compression of ultraspectral sounder data. The lossless compression results from the transform-based (e.g. JPEG2000, 3D SPIHT, and Lossless PCA), prediction-based (e.g. JPEG-LS, CALIC, and linear prediction using OOMP), and clustering-based (e.g. PVQ, DPVQ, PPVQ and FPVQ) methods are presented. The ultraspectral sounder data features strong correlations in disjoint spectral regions affected by the same type of absorbing gases. Some robust data preprocessing scheme (e.g. BAR) is also demonstrated to improve compression gains of existing state-of-the-art compression methods such as JPEG2000, 3D SPIHT, JPEG-LS, and CALIC.

Widespread satellite communication use and availability of advanced modulations and forward error correction coding

GOES-R, planned for launch around 2012, is currently under development by the National Oceanic and Atmospheric Administration (NOAA) of the United States. It will be the first in a new series of geostationary (GEO) environmental satellites to provide greater capabilities for weather, atmosphere, climate, and ocean monitoring. All the onboard sensors together may generate a combined raw sensor data rate of as much as 200 Mbps on the downlink, while the global rebroadcast data rate to the users after ground compression may be as much as 32 Mbps. To transmit such a high data rate through a channel of limited bandwidth, the adoption of a high-order modulation, such as QPSK, 8PSK, or 16QAM, is necessary. As a result, much higher transmit power than that for the binary modulation is needed in order to achieve the required bit error rate, which is particularly stringent for GOES-R due to the needed protection to the compressed data. Thus, the forward error correction (FEC) coding, which is a technique that can provide significant improvement of power efficiency, becomes crucial for GOES-R. This paper presents various methods of combining high-order modulations and FEC codes. We have proposed a baseline code waveform for GOES-R, which can satisfy both bandwidth and power efficiency requirements. In this paper, we also assess other commercially available code waveforms and compare their performances with that of our baseline waveform.