David R. Chapman

A Hybrid Scheduling Algorithm for Data Intensive Workloads in a MapReduce Environment

The specific choice of workload task schedulers for Hadoop MapReduce applications can have a dramatic effect on job workload latency. The Hadoop Fair Scheduler (FairS) assigns resources to jobs such that all jobs get, on average, an equal share of resources over time. Thus, it addresses the problem with a FIFO scheduler when short jobs have to wait for long running jobs to complete. We show that even for the FairS, jobs are still forced to wait significantly when the MapReduce system assigns equal sharing of resources due to dependencies between Map, Shuffle, Sort, Reduce phases. We propose a Hybrid Scheduler (HybS) algorithm based on dynamic priority in order to reduce the latency for variable length concurrent jobs, while maintaining data locality. The dynamic priorities can accommodate multiple task lengths, job sizes, and job waiting times by applying a greedy fractional knapsack algorithm for job task processor assignment. The estimated runtime of Map and Reduce tasks are provided to the HybS dynamic priorities from the historical Hadoop log files. In addition to dynamic priority, we implement a reordering of task processor assignment to account for data availability to automatically maintain the benefits of data locality in this environment. We evaluate our approach by running concurrent workloads consisting of the Word-count and Terasort benchmarks, and a satellite scientific data processing workload and developing a simulator. Our evaluation shows the HybS system improves the average response time for the workloads approximately 2.1x faster over the Hadoop FairS with a standard deviation of 1.4x.

A near fundamental decadal data record of airs infrared surface brightness temperatures

Aqua provides a direct measurement of the surface infrared radiation for a near decade (2003-2011) from a single satellite platform with two completely independent instruments (AIRS and MODIS). We produced a 0.5°×1.0° lat-lon daily gridded near fundamental decadal data record (FDDR) of AIRS all sky brightness temperatures (BT) for all 2378 AIRS channels including the 4μ and 12μ surface windows for the 9-year period. We show (i) that decadal difference trends between the spectrally convolved AIRS and MODIS IR surface channels 3.96μ, 4.05μ and 12.03μ are as small as 0.01K, 0.016K and 0.001K per year respectively, suitable to detect year to year decadal trends of AIRS global surface BTs as small as 0.1K over the decade. (ii) The analysis of gridded AIRS surface BT channel 4.16μ and 12.18μ reveals no global annual warming trend between 2003-2011 and negligible trends for both hemispheres in the 9-year period. (iii) Both AIRS channels 4.16μ and 12.18μ show a statistically significant global annual BT warming trend of 0.6K and 0.87K respectively in the Arctic (60N-90N) with yearly changes varying by as much as 1.6K. (iv) Finally, we show that AIRS global surface BT spectral channel 4.16μ has a correlation with annual, seasonal and monthly GISS global station surface temperature data records 0.96, 0.9, 0.82 respectively. We conjecture that direct satellite level 1B AIRS IR radiance sounding observations constitute a FDDR and that well calibrated IR surface BTs can serve as a surrogate for directly observing global climate surface temperatures.

Noise reduction in gridded airs brightness temperature grids using the MODIS Obscov algorithm

We have adapted the MODIS Observation Coverage (Obscov) algorithm to the gridding of the AIRS L1b calibrated brightness temperatures, and developed a “Numeric-Obscov” implementation capable of utilizing the AIRS Tophat PSF dataset [1, 2]. We show that Obscov significantly reduces AIRS gridded-artifacts from a day to a season over winter 2005. The Obscov algorithm provides a physically accurate gridding model using spatial response information within the IFOV. Previously, Obscov has been used with an approximate triangular PSF, to allow for billions of integrals to be calculated per daily grid. We evaluated the Obscov algorithm via inter-comparison with MODIS-Aqua. Over a day, RMS noise is improved at high latitudes at 0.5°×1°. When averaged over a week or more, RMS noise is reduced globally by almost 40%.

Sub-cellular feature detection and automated extraction of collocalized actin and myosin regions

We describe a new distance-based metric to measure the strength of collocalization in multi-color microscopy images for user-selected regions. This metric helps to standardize, objectify, quantify, and even automate light microscopy observations. Our new algorithm uses this metric to automatically identify and annotate a donut shaped actomyosin stress fiber bundle evident in vascular smooth muscle cells on certain types of surfaces. Both the metric and the algorithm have been implemented as an open source plugin for the popular ImageJ toolkit. They are available for download at http://code.google.com/p/actin-myosin-plugin/. Using cells stained for the cytoskeletal proteins actin and myosin, we show how characteristics of the identified stress fiber bundle are indicative of the kind of surface the cell is placed upon, and prove that weak spots in this structure are correlated with local membrane extensions. Given the relationship between membrane extension, cell migration, vascular disease, embryonic development, and cancer metastasis we provide that these tools to enable biological research that could improve our quality of life.