Sifei Lu

A Framework for Cloud-Based Large-Scale Data Analytics and Visualization: Case Study on Multiscale Climate Data

In this paper, we present a cloud framework to provide cloud clustering, workflow scheduling and management, fault tolerance and distributed data storage, data analytics and visualisation services. Using a practical case study, we show that in the process of analyzing multiscale climate data, typical problems plaguing data analysts are faced. These include large datasets and limited computational resources, data complexity and limited knowledge, and varying data structures/formats and the need to integrate different tools. The implementation of our framework to climate studies was a success. This can be seen in its ability to perform spatio-temporal data analysis and visualization of a large multi-dimensional climate dataset with reduced processing time. The framework demonstrates great flexibility and simplicity for end users intending to perform data analysis by aiding the integration of data and tools and enabling interactive visualization on-the-fly. This is coupled with effective utilization of computational resources and data storage systems.

An Iterative Optimization Framework for Adaptive Workflow Management in Computational Clouds

As more and more data can be generated at a faster-than-ever rate nowadays, it becomes a challenge to processing large volumes of data for complex data analysis. In order to address performance and cost issues of big data processing on clouds, we present a novel design of adaptive workflow management system which includes a data mining based prediction model, workflow scheduler, and iteration controls to optimize the data processing via iterative workflow tasks. We proposed a new heuristic algorithm, called Upgrade Fit, which dynamically and continuously reallocates multiple types of cloud resources to fulfill the performance and cost requirements. The iterative workflow tasks can be bursty bags of tasks to be executed repetitively for data processing. A real application of weather forecast workflow has been used to evaluate the capability of our system for large volume image data processing. Experimental system has been set up and the results indicate that the system can effectively handle multiple types of cloud resources and optimize the performance iteratively.

RiskVis: Supply chain visualization with risk management and real-time monitoring

With increased complexity, supply chain networks (SCNs) of modern era face higher risks and lower efficiency due to limited visibility. Hence, there is an immediate need to provide end-to-end supply chain visibility for efficient management of complex supply chains. This paper proposes a visualization scheme based on multi-hierarchical modular design and develops a supply chain visualization platform with risk management and real-time monitoring, named RiskVis, for realizing better Supply Chain Risk Management (SCRM). A Supply Chain Visualizer (SCV) with a graphical visualization platform is mounted as a part of a SCRM management decision-making dashboard and it provides senior management a clearer view of supply chain operations in a local/regional/global setting. The platform not only displays spatio-temporal connectivity patterns of entities in a supply chain; it also accommodates real-time risk-related data collection and risk monitoring. The proposed platform offers the flexibility to be customized based on the users requirements - to process and store the supply chain data in the server, visualize the supply chain data, network map, risk alert, and other information needed for SCRM. Supply chain decision makers can deploy it on the desktop or embed it into the companys enterprise applications in a front office environment for better managing risks of their supply chains.

Evaluating hardware-assisted virtualization for deploying HPC-as-a-service

Virtualization has been the main driver behind the rise of Cloud computing. Despite Cloud computings tremendous benefits to many applications (e.g., enterprise, Web, game/ multimedia, life sciences, and data analytics), its success in High Performance Computing (HPC) domain has been limited. The oft-cited reason is, apparently, latency caused by virtualization. Meanwhile, the rising popularity of virtualization has compelled CPU vendors to incorporate virtualization technology (VT) in chips. This hardware VT is believed to accelerate context switching, speed up memory address translation, and enable I/O direct access; those are basically sources of virtualization overheads. This paper reports the evaluation on computation and communication performance of different virtualized environments, i.e., Xen and KVM, leveraging hardware VT. Different network fabrics, namely Gigabit Ethernet and InfiniBand, were employed and tested in the virtualized environments and their results were compared against those in the native environments. A real-world HPC application (an MPI-based hydrodynamic simulation) was also used to assess the performance. Outcomes indicate that hardware-assisted virtualization can bring HPC-as-a-Service into realization.

Design and Development of an Adaptive Workflow-Enabled Spatial-Temporal Analytics Framework

Cloud computing is a suitable platform for execution of complex computational tasks and scientific simulations that are described in the form of workflows. Such applications are managed by Workflow Management System (WfMS). Because existing WfMSs are not able to autonomically provision resources to real-time applications and schedule them while supporting fault tolerance and data privacy, we present a highly-scalable workflow-enabled analytics system that manages inter-dependable analytics tasks adaptively with varying operational requirements on a common platform and enables visualization of multidimensional datasets of real world phenomena. In this paper, we present the architecture of such a WfMS and evaluate it in terms of performance for execution of workflows in Clouds. A real world application of climate-associated dengue fever prediction was evaluated on public, private, and hybrid Clouds and experienced effective speedup in all the environments.

A Dynamic Hybrid Resource Provisioning Approach for Running Large-Scale Computational Applications on Cloud Spot and On-Demand Instances

Testing and executing large-scale computational applications in public clouds is becoming prevalent due to cost saving, elasticity, and scalability. However, how to increase the reliability and reduce the cost to run large-scale applications in public clouds is still a big challenge. In this paper, we analyzed the pricing schemes of Amazon Elastic Compute Cloud (EC2) and found the disturbance effect that the price of the spot instances can be heavily affected due to the large number of spot instances required. We proposed a dynamic approach which schedules and runs large-scale computational applications on a dynamic pool of cloud computational instances. We use hybrid instances, including both on-demand instances for high priority tasks and backup, and spot instances for normal computational tasks so as to further reduce the cost without significantly increasing the completion time. Our proposed method takes the dynamic pricing of cloud instances into consideration, and it reduces the cost and tolerates the failures for running large-scale applications in public clouds. We conducted experimental tests and an agent based Scalable complex System modeling for Sustainable city (S3) application is used to evaluate the scalability, reliability and cost saving. The results show that our proposed method is robust and highly flexible for researchers and users to further reduce cost in real practice.

Hierarchical Parallelization and Runtime Scheduling for Pregel-Like Graph Processing Systems

Graph processing has become popular for various big data analytic applications. Googles Pregel framework enables vertex-centric graph processing in distributed environment based on Bulk Synchronous Parallel (BSP) model. However, the BSP model is inefficient for many complex graph algorithms requiring graph traversals, as only a small number of vertices really update states in each super step. In this paper, we propose an hierarchical parallelization mechanism, taking the advantages of both synchronous (warp-level) and asynchronous (task-level) parallelization approaches. In addition, a runtime task scheduling mechanism is proposed, relying on real-time monitoring or prediction of resource utilization. Experiments have verified that the hierarchical parallelization mechanism can expose greater parallelism, and thus, increase resource utilization significantly. Moreover, the runtime scheduling mechanism can avoid aggressive resource competition, and thus, further enhance the performance of the parallelized graph processing.

Social media for supply chain risk management

With the rapid increase of online social network users worldwide, social media feeds have become a rich and valuable information resource and attract great attention across diversified domains. In social media data, there are abundant contents of two-way and interactive communication about products, demand, customer services and supply. This makes social media a valuable channel for listening to the voices from the market and measuring supply chain risks and new market trends for companies. In this study, we surveyed the potential value of social media in supply chain risk management (SCRM) and examined how they can be applied to SCRM systematically. We found that while such medium is very useful in supply chain risk management, it also brings along a new risk to supply chains, so called social media risk, as supply chain incidents may be rapidly transmitted and magnified through social media platforms worldwide. Accordingly, a new framework is proposed that assists the hiring of social media to serve supply chain risk management tasks.

A cloud-based workflow management solution for collaborative analytics

The concept of collaborative analytics is to accommodate reuse and collaboration in data analysis process through sharing of analytics methods, algorithms, and computation resources. However, realizing collaborative analytics is challenging due to the large data sets, high throughput and computational intensive requirements. In this demonstration, we present a cloud-based workflow management solution that allows collaborative analytics to run in the cloud computing environment. Our solution provides sharing of analytics resources, recommendation of analytic workflows, dynamic scheduling and provisioning for scalable data analytics, high availability through fault-tolerance, real-time monitoring and tracking of collaborative analytics status. Examples of a generic data mining analysis and climate change analytics are given to show that our work can be applied for a wide variety of study in the real-life world.

Performance and Monetary Cost of Large-Scale Distributed Graph Processing on Amazon Cloud

Graph analytics has become essential to uncover relationship insights in complex systems. As graphs grow in scale, several graph-parallel frameworks including Pregel, GraphLab, and PowerGraph are developed based on commodity computers and/or Cloud instances. According to recent research and empirical performance evaluation, system optimization on PowerGraph allow it to outperform others significantly for processing natural graphs with skewed degree distribution. However, the performance characters, resource usage pattern and monetary cost have never been explored in-depth. In this paper, PowerGraph are evaluated with three different algorithms on Amazon EC2 instances with upto 768 CPU cores. We find that thegraph processing performance does not always increase with the increasing Cloud resources. Due to synchronization overheads, resources are not fully utilized. Graph processing tasks may prefer different execution strategies with specified number and type of Cloud instances to achieve high cost-efficiency, playing the trade-off between monetary cost and execution performance.