Joseph C. Jacob

Pegasus: A framework for mapping complex scientific workflows onto distributed systems

This paper describes the Pegasus framework that can be used to map complex scientific workflows onto distributed resources. Pegasus enables users to represent the workflows at an abstract level without needing to worry about the particulars of the target execution systems. The paper describes general issues in mapping applications and the functionality of Pegasus. We present the results of improving application performance through workflow restructuring which clusters multiple tasks in a workflow into single entities. A real-life astronomy application is used as the basis for the study.

Montage: a grid portal and software toolkit for science-grade astronomical image mosaicking

Montage is a portable software toolkit to construct custom, science-grade mosaics that preserve the astrometry and photometry of astronomical sources. The user specifies the dataset, wavelength, sky location, mosaic size, coordinate system, projection, and spatial sampling. Montage supports massive astronomical datasets that may be stored in distributed archives. Montage can be run on both single- and multi-processor computers, including clusters and grids. Standard grid tools are used to access remote data or run Montage on remote computers. This paper describes the architecture, algorithms, performance, and usage of Montage as both a software toolkit and a grid portal.

A comparison of two methods for building astronomical image mosaics on a grid

This paper compares two methods for running an application composed of a set of modules on a grid. The set of modules (collectively called Montage) generates large astronomical image mosaics by composing multiple small images. The workflow that describes a particular run of Montage can be expressed as a directed acyclic graph (DAG), or as a short sequence of parallel (MPI) and sequential programs. In the first case, Pegasus can be used to run the workflow. In the second case, a short shell script that calls each program can be run. In this paper, we discuss the Montage modules, the workflow run for a sample job, and the two methods of actually running the workflow. We examine the run time for each method and compare the portions that differ between the two methods.

Atmospheric CO 2 measurements with a 2 μm airborne laser absorption spectrometer employing coherent detection

We report airborne measurements of CO(2) column abundance conducted during two 2009 campaigns using a 2.05 μm laser absorption spectrometer. The two flight campaigns took place in the California Mojave desert and in Oklahoma. The integrated path differential absorption (IPDA) method is used for the CO(2) column mixing ratio retrievals. This instrument and the data analysis methodology provide insight into the capabilities of the IPDA method for both airborne measurements and future global-scale CO(2) measurements from low Earth orbit pertinent to the NASA Active Sensing of CO(2) Emissions over Nights, Days, and Seasons mission. The use of a favorable absorption line in the CO(2) 2 μm band allows the on-line frequency to be displaced two (surface pressure) half-widths from line center, providing high sensitivity to the lower tropospheric CO(2). The measurement repeatability and measurement precision are in good agreement with predicted estimates. We also report comparisons with airborne in situ measurements conducted during the Oklahoma campaign.

The Pegasus portal: web based grid computing

Pegasus is a planning framework for mapping abstract workflows for execution on the Grid. This paper presents the implementation of a web-based portal for submitting workflows to the Grid using Pegasus. The portal also includes components for generating abstract workflows based on a metadata description of the desired data products and application-specific services. We describe our experiences in using this portal for two Grid applications. A major contribution of our work is in introducing several components that can be useful for Grid portals and hence should be included in Grid portal development toolkits.

Montage: a grid-enabled engine for delivering custom science-grade mosaics on demand

This paper describes the design of a grid-enabled version of Montage, an astronomical image mosaic service, suitable for large scale processing of the sky. All the re-projection jobs can be added to a pool of tasks and performed by as many processors as are available, exploiting the parallelization inherent in the Montage architecture. We show how we can describe the Montage application in terms of an abstract workflow so that a planning tool such as Pegasus can derive an executable workflow that can be run in the Grid environment. The execution of the workflow is performed by the workflow manager DAGMan and the associated Condor-G. The grid processing will support tiling of images to a manageable size when the input images can no longer be held in memory. Montage will ultimately run operationally on the Teragrid. We describe science applications of Montage, including its application to science product generation by Spitzer Legacy Program teams and large-scale, all-sky image processing projects.

MID-INFRARED VARIABILITY FROM THE SPITZER DEEP WIDE-FIELD SURVEY

We use the multi-epoch, mid-infrared Spitzer Deep Wide-Field Survey to investigate the variability of objects in 8.1 deg^2 of the NOAO Deep Wide Field Survey Bootes field. We perform a Difference Image Analysis of the four available epochs between 2004 and 2008, focusing on the deeper 3.6 and 4.5 μm bands. Out of 474, 179 analyzed sources, 1.1% meet our standard variability selection criteria that the two light curves are strongly correlated (r > 0.8) and that their joint variance (σ_(12)) exceeds that for all sources with the same magnitude by 2σ. We then examine the mid-IR colors of the variable sources and match them with X-ray sources from the XBootes survey, radio catalogs, 24 μm selected active galactic nucleus (AGN) candidates, and spectroscopically identified AGNs from the AGN and Galaxy Evolution Survey (AGES). Based on their mid-IR colors, most of the variable sources are AGNs (76%), with smaller contributions from stars (11%), galaxies (6%), and unclassified objects, although most of the stellar, galaxy, and unclassified sources are false positives. For our standard selection criteria, 11%-12% of the mid-IR counterparts to X-ray sources, 24 μm AGN candidates, and spectroscopically identified AGNs show variability. The exact fractions depend on both the search depth and the selection criteria. For example, 12% of the 1131 known z>1 AGNs in the field and 14%-17% of the known AGNs with well-measured fluxes in all four Infrared Array Camera bands meet our standard selection criteria. The mid-IR AGN variability can be well described by a single power-law structure function with an index of γ ≈ 0.5 at both 3.6 and 4.5 μm, and an amplitude of S _0 ≃ 0.1 mag on rest-frame timescales of 2 yr. The variability amplitude is higher for shorter rest-frame wavelengths and lower luminosities.

Airborne Laser Absorption Spectrometer Measurements of Atmospheric CO2 Column Mole Fractions: Source and Sink Detection and Environmental Impacts on Retrievals

AbstractThis paper provides atmospheric CO2 column abundance measurement results from a summer 2011 series of flights of a 2.05-μm laser absorption spectrometer on the NASA DC-8 research aircraft. The integrated path differential absorption (IPDA) method is used for the CO2 column mole fraction retrievals. This instrument and the data analysis methodology developed to achieve retrievals over complex terrain and variable atmospheric conditions provide insight into the capabilities of the IPDA method for both airborne measurements and future global-scale CO2 measurements from low-Earth orbit pertinent to the proposed NASA Active Sensing of CO2 Emissions over Nights, Days, and Seasons (ASCENDS) mission. Demonstrated in this paper is the capability to measure CO2 drawdown caused by crop activity during a midday flight over the U.S. upper Midwest area. In addition, an example is provided of high spatial resolution measurements of CO2 plumes from individual stack clusters of the Four Corners Power Plant in nort...

Exploration of Large Digital Sky Surveys

We review some of the scientific opportunities and technical challenges posed by the exploration of the large digital sky surveys, in the context of a Virtual Observatory (VO). The VO paradigm will profoundly change the way observational astronomy is done. Clustering analysis techniques can be used to discover samples of rare, unusual, or even previously unknown types of astronomical objects and phenomena. Exploration of the previously poorly probed portions of the observable parameter space are especially promising. We illustrate some of the possible types of studies with examples drawn from DPOSS; much more complex and interesting applications are forthcoming. Development of the new tools needed for an efficient exploration of these vast data sets requires a synergy between astronomy and information sciences, with great potential returns for both fields.

Exploration of parameter spaces in a virtual observatory

Like every other field of intellectual endeavor, astronomy is being revolutionized by the advances in information technology. There is an ongoing exponential growth in the volume, quality, and complexity of astronomical data sets, mainly through large digital sky surveys and archives. The Virtual Observatory (VO) concept represents a scientific and technological framework needed to cope with this data flood. Systematic exploration of the observable parameter spaces, covered by large digital sky surveys spanning a range of wavelengths, will be one of the primary modes of research with a VO. This is where the truly new discoveries will be made, and new insights be gained about the already known astronomical objects and phenomena. We review some of the methodological challenges posed by the analysis of large and complex data sets expected in the VO-based research. The challenges are driven both by the size and the complexity of the data sets (billions of data vectors in parameter spaces of tens or hundreds of dimensions), by the heterogeneity of the data and measurement errors, including differences in basic survey parameters for the federated data sets (e.g., in the positional accuracy and resolution, wavelength coverage, time baseline, etc), various selection effects, as well as the intrinsic clustering properties (functional form, topology) of the data distributions in the parameter spaces of observed attributes. Answering these challenges will require substantial collaborative efforts and partnerships between astronomers, computer scientists, and statisticians.