Kwo-Sen Kuo

The Microwave Radiative Properties of Falling Snow Derived from Nonspherical Ice Particle Models. Part I: An Extensive Database of Simulated Pristine Crystals and Aggregate Particles, and Their Scattering Properties

AbstractA 3D growth model is used to simulate pristine ice crystals, which are aggregated using a collection algorithm to create larger, multicrystal particles. The simulated crystals and aggregates have mass-versus-size and fractal properties that are consistent with field observations. The growth/collection model is used to generate a large database of snow particles, and the single-scattering properties of each particle are computed using the discrete dipole approximation to account for the nonspherical geometries of the particles. At 13.6 and 35.5 GHz, the bulk radar reflectivities of nonspherical snow particle polydispersions differ from those of more approximate spherical, homogeneous, ice–air particle polydispersions that have the same particle size distributions, although the reflectivities of the nonspherical particles are roughly approximated by polydispersions of spheres of 0.1–0.2 g cm−3 density. At higher microwave frequencies, such as 165.5 GHz, the bulk extinction (and scattering) coefficie...

The microwave radiative properties of falling snow derived from nonspherical ice particle models. Part II: Initial testing using radar, radiometer and In situ observations

AbstractIn this study, two different particle models describing the structure and electromagnetic properties of snow are developed and evaluated for potential use in satellite combined radar–radiometer precipitation estimation algorithms. In the first model, snow particles are assumed to be homogeneous ice–air spheres with single-scattering properties derived from Mie theory. In the second model, snow particles are created by simulating the self-collection of pristine ice crystals into aggregate particles of different sizes, using different numbers and habits of the collected component crystals. Single-scattering properties of the resulting nonspherical snow particles are determined using the discrete dipole approximation. The size-distribution-integrated scattering properties of the spherical and nonspherical snow particles are incorporated into a dual-wavelength radar profiling algorithm that is applied to 14- and 34-GHz observations of stratiform precipitation from the ER-2 aircraftborne High-Altitude ...

Evaluating the impact of data placement to spark and SciDB with an Earth Science use case

We investigate the impact of data placement on two Big Data technologies, Spark and SciDB, with a use case from Earth Science where data arrays are multidimensional. Simultaneously, this investigation provides an opportunity to evaluate the performance of the technologies involved. Two datastores, HDFS and Cassandra, are used with Spark for our comparison. It is found that Spark with Cassandra performs better than with HDFS, but SciDB performs better yet than Spark with either datastore. The investigation also underscores the value of having data aligned for the most common analysis scenarios in advance on a shared nothing architecture. Otherwise, repartitioning needs to be carried out on the fly, degrading overall performance.

Addressing the big-earth-data variety challenge with the hierarchical triangular mesh

We have implemented an updated Hierarchical Triangular Mesh (HTM) as the basis for a unified data model and an indexing scheme for geoscience data to address the variety challenge of Big Earth Data. In the absence of variety, the volume challenge of Big Data is relatively easily addressable with parallel processing. The more important challenge in achieving optimal value with a Big Data solution for Earth Science (ES) data analysis, however, is being able to achieve good scalability with variety. With HTM unifying at least the three popular data models, i.e. Grid, Swath, and Point, used by current ES data products, data preparation time for integrative analysis of diverse datasets can be drastically reduced and better variety scaling can be achieved. HTM is also an indexing scheme, and when applied to all ES datasets, data placement alignment (or co-location) on the shared nothing architecture, which most Big Data systems are based on, is guaranteed and better performance is ensured. With HTM most geospatial set operations become integer interval operations with further performance advantages.

Implications of data placement strategy to Big Data technologies based on shared-nothing architecture for geosciences

It is found that data placement on the networked nodes of a cluster based on the shared-nothing architecture (SNA) should align in the physical (i.e. spatiotemporal) space for most geoscience Big Data analysis systems in order to minimize data movements and thus achieve optimal performance and efficiency. This is due to the fact that data analysis in geosciences predominantly requires spatiotemporal coincidence. If individual datasets are considered separately in their placement on the cluster nodes, these systems often have to move data between nodes when an analysis involves two or more datasets. In this paper, we first report our discoveries from a data placement alignment experiment with two Big Data technologies, SciDB and Spark+HDFS, and then elucidate some of the far-reaching implications of this discovery.

Implementing connected component labeling as a user defined operator for SciDB

We have implemented a flexible User Defined Operator (UDO) for labeling connected components of a binary mask expressed as an array in SciDB, a parallel distributed database management system based on the array data model. This UDO is able to process very large multidimensional arrays by exploiting SciDBs memory management mechanism that efficiently manipulates arrays whose memory requirements far exceed available physical memory. The UDO takes as primary inputs a binary mask array and a binary stencil array that specifies the connectivity of a given cell to its neighbors. The UDO returns an array of the same shape as the input mask array with each foreground cell containing the label of the component it belongs to. By default, dimensions are treated as non-periodic, but the UDO also accepts optional input parameters to specify periodicity in any of the array dimensions. The UDO requires four stages to completely label connected components. In the first stage, labels are computed for each subarray or chunk of the mask array in parallel across SciDB instances using the weighted quick union (WQU) with half-path compression algorithm. In the second stage, labels around chunk boundaries from the first stage are stored in a temporary SciDB array that is then replicated across all SciDB instances. Equivalences are resolved by again applying the WQU algorithm to these boundary labels. In the third stage, relabeling is done for each chunk using the resolved equivalences. In the fourth stage, the resolved labels, which so far are “flattened” coordinates of the original binary mask array, are renamed with sequential integers for legibility. The UDO is demonstrated on a 3-D mask of 0(10n) elements, with 0(108) foreground cells and o(106) connected components. The operator completes in 19 minutes using 84 SciDB instances.

Feature extraction and tracking for large-scale geospatial data

Feature extraction and tracking is a fundamental operation used in many geoscience applications. In this paper, we present a scalable method for computing and tracking features on distributed memory machines for large-scale geospatial data. We carefully apply new communication schemes to minimize the data exchanged among the computing nodes in building and updating the global connectivity information of features. We present a theoretical complexity analysis, and show that our method can significantly reduce the communication cost compared to the traditional method.

Leveraging data intensive computing to support Automated Event Services

Our AES is an ideal example of a new generation of scientific analysis tools that are empowered by the rapid growth of facilities tailored for data-intensive computing. AES will greatly reduce the effort on the part of investigators to systematically search for interesting correlations and test hypotheses while also freeing researchers from the burden of managing the exploding volume of data. By supporting the ability to exchange event specifications and query results, AES greatly aids collaboration among investigators. We anticipate that AES will ultimately lead to entirely novel lines of investigation.