Maria A. Nieto-santisteban

Scientific data management in the coming decade

Scientific instruments and computer simulations are creating vast data stores that require new scientific methods to analyze and organize the data. Data volumes are approximately doubling each year. Since these new instruments have extraordinary precision, the data quality is also rapidly improving. Analyzing this data to find the subtle effects missed by previous studies requires algorithms that can simultaneously deal with huge datasets and that can find very subtle effects --- finding both needles in the haystack and finding very small haystacks that were undetected in previous measurements.

GrayWulf: Scalable Clustered Architecture for Data Intensive Computing

Data intensive computing presents a significant challenge for traditional supercomputing architectures that maximize FLOPS since CPU speed has surpassed IO capabilities of HPC systems and BeoWulf clusters. We present the architecture for a three tier commodity component cluster designed for a range of data intensive computations operating on petascale data sets named GrayWulf. The design goal is a balanced system in terms of IO performance and memory size, according to Amdahls Laws. The hardware currently installed at JHU exceeds one petabyte of storage and has 0.5 bytes/sec of I/O and 1 byte of memory for each CPU cycle. The GrayWulf provides almost an order of magnitude better balance than existing systems. The paper covers its architecture and reference applications. The software design is presented in a companion paper.

GALEX-SDSS CATALOGS FOR STATISTICAL STUDIES

We present a detailed study of the Galaxy Evolution Explorer’s (GALEX) photometric catalogs with special focus on the statistical properties of the All-sky and Medium Imaging Surveys. We introduce the concept of primaries to resolve the issue of multiple detections and follow a geometric approach to define clean catalogs with well understood selection functions. We cross-identify the GALEX sources (GR2+3) with Sloan Digital Sky Survey (SDSS; DR6) observations, which indirectly provides an invaluable insight into the astrometric model of the UV sources and allows us to revise the band merging strategy. We derive the formal description of the GALEX footprints as well as their intersections with the SDSS coverage along with analytic calculations of their areal coverage. The crossmatch catalogs are made available for the public. We conclude by illustrating the implementation of typical selection criteria in SQL for catalog subsets geared toward statistical analyses, e.g., correlation and luminosity function studies.

Cosmic-Ray Rejection and Readout Efficiency for Large-Area Arrays

We present an algorithm to optimally process uniformly sampled array image data obtained with a nondestructive readout. The algorithm discards full wells, removes cosmic-ray (particle) hits and other glitches, and makes a nearly optimum estimate of the signal on each pixel. The algorithm also compresses the data. The computer requirements are modest, and the results are robust. The results are shown and compared to results of Fowler-sampled and -processed data. Nonideal detector performance may require some additional code, but this is not expected to cost much processing time. Known types of detector faults are addressed.

Designing a Multi-Petabyte Database for LSST

The 3.2 giga-pixel LSST camera will produce approximately half a petabyte of archive images every month. These data need to be reduced in under a minute to produce real-time transient alerts, and then added to the cumulative catalog for further analysis. The catalog is expected to grow about three hundred terabytes per year. The data volume, the real-time transient alerting requirements of the LSST, and its spatio-temporal aspects require innovative techniques to build an efficient data access system at reasonable cost. As currently envisioned, the system will rely on a database for catalogs and metadata. Several database systems are being evaluated to understand how they perform at these data rates, data volumes, and access patterns. This paper describes the LSST requirements, the challenges they impose, the data access philosophy, results to date from evaluating available database technologies against LSST requirements, and the proposed database architecture to meet the data challenges.

Validation of Up-the-Ramp Sampling with Cosmic-Ray Rejection on Infrared Detectors

We examine cosmic-ray rejection methodology on data collected from InSb and Si:As detectors. The application of an up-the-ramp sampling technique with cosmic-ray identification and mitigation is the focus of this study. This technique is valuable for space-based observatories which are exposed to high-radiation environments. We validate the up-the-ramp approach on radiation-test data sets with InSb and Si:As detectors which were generated for SIRTF. The up-the-ramp sampling method studied in this paper is over 99.9% effective at removing cosmic rays and preserves the structure and photometric quality of the image to well within the measurement error.

The JWST data management system engineering database

The Engineering Data Processing (EDP) component of the James Webb Telescope (JWST) Data Management System (DMS) will collect calibrated engineering values for about 15,000 parameters, 300 million samples per day, with a potential daily database growth of 14 GB, 5 TB per year, 50 TB for a 10-year mission. While data will be mostly received in (time, parameter) order, fast access requires translation into (parameter, time) organization and sorting. Organization and indexing of the data will affect storage requirements as well as ingest and access efficiency. Fast access is critical to pipelines processing and calibrating science data.

Data compression for the next-generation space telescope

Summary form only given. The next-generation space telescope (NGST) will produce about 600 GB/day, assuming we use the NASA yardstick 8k/spl times/8k NIR camera (16 bits/pixel), save and transmit 64 non-destructive read-outs per image, and the camera is in continuous use (about 80 observations/day, 10/sup 3/ s each). However, with an L2 halo orbit, the NASA NGST study estimates a downlink rate of 5.35 GB/day using X-band. Clearly the volume of data to downlink must be reduced by at least a factor of 100. Astronomical images are noisy. This fact makes them difficult to compress by lossless compression algorithms such as Huffman, Lempel-Ziv, run-length, or arithmetic code. However, they also have the virtue of showing similar values among adjacent pixels. Techniques such as Rices algorithm (Rice et al., 1993) and derivatives (White and Becker, 1998; Stiavelli and White, 1997) can take advantage of this. We present the way in which some of these compression techniques would work with NGST images. Unfortunately, these lossless algorithms give us compression ratios that still exceed the telemetry guidelines. We have also looked into the feasibility of doing lossy compression by scaling the original image prior to the lossless compression. Under this scheme, we find substantial data reduction with a negligible effect on the data quality.