Brandon Myers

REEF: Retainable Evaluator Execution Framework

Resource Managers like Apache YARN have emerged as a critical layer in the cloud computing system stack, but the developer abstractions for leasing cluster resources and instantiating application logic are very low-level. This flexibility comes at a high cost in terms of developer effort, as each application must repeatedly tackle the same challenges (e.g., fault-tolerance, task scheduling and coordination) and re-implement common mechanisms (e.g., caching, bulk-data transfers). This paper presents REEF, a development framework that provides a control-plane for scheduling and coordinating task-level (data-plane) work on cluster resources obtained from a Resource Manager. REEF provides mechanisms that facilitate resource re-use for data caching, and state management abstractions that greatly ease the development of elastic data processing work-flows on cloud platforms that support a Resource Manager service. REEF is being used to develop several commercial offerings such as the Azure Stream Analytics service. Furthermore, we demonstrate REEF development of a distributed shell application, a machine learning algorithm, and a port of the CORFU [4] system. REEF is also currently an Apache Incubator project that has attracted contributors from several instititutions.1 http://reef.incubator.apache.org

Compiled Plans for In-Memory Path-Counting Queries

Dissatisfaction with relational databases for large-scale graph processing has motivated a new class of graph databases that offer fast graph processing but sacrifice the ability to express basic relational idioms. However, we hypothesize that the performance benefits amount to implementation details, not a fundamental limitation of the relational model. To evaluate this hypothesis, we are exploring code-generation to produce fast in-memory algorithms and data structures for graph patterns that are inaccessible to conventional relational optimizers.

Apache REEF: Retainable Evaluator Execution Framework

Resource Managers like YARN and Mesos have emerged as a critical layer in the cloud computing system stack, but the developer abstractions for leasing cluster resources and instantiating application logic are very low level. This flexibility comes at a high cost in terms of developer effort, as each application must repeatedly tackle the same challenges (e.g., fault tolerance, task scheduling and coordination) and reimplement common mechanisms (e.g., caching, bulk-data transfers). This article presents REEF, a development framework that provides a control plane for scheduling and coordinating task-level (data-plane) work on cluster resources obtained from a Resource Manager. REEF provides mechanisms that facilitate resource reuse for data caching and state management abstractions that greatly ease the development of elastic data processing pipelines on cloud platforms that support a Resource Manager service. We illustrate the power of REEF by showing applications built atop: a distributed shell application, a machine-learning framework, a distributed in-memory caching system, and a port of the CORFU system. REEF is currently an Apache top-level project that has attracted contributors from several institutions and it is being used to develop several commercial offerings such as the Azure Stream Analytics service.

POGIL Activities for Computer Organization and Architecture: (Abstract Only)

Research shows that active learning can increase student performance and engagement, but access to materials is a notable barrier to using research-based instruction strategies in CS and Engineering. We present results of a project-in-progress that aims to create, pilot, revise, and disseminate POGIL activities for Computer Organization and Architecture. POGIL is a research-based instruction strategy that comprises self-managed teams, guided inquiry (or, exploration), and development of process skills, such as critical thinking and assessment. The strategy has been shown to improve student performance and engagement in courses in scientific disciplines and, more recently, CS. This poster presents how we have applied the methodology for POGIL activities to Computer Organization and Architecture and highlights one activity in depth. From 2 pilots with 36 and 70 students we produced revisions and timings for 6 activities. We also discuss lessons learned in a) facilitation, such as the importance of roles and the tradeoffs of class-level synchrony and b) authorship, such as the appropriate choice of model (or, subject of inquiry) and level of guidance in the exploration phase of an activity. The intended outcome of this project is to make these activities publicly available on cspogil.org.

Integrating query processing with parallel languages

In this thesis we propose new techniques for using parallel languages to improve query processing. Optimizing a query plan and its particular implementation is important for efficient processing on modern systems. First, we present our work on a parallel representation of queries using partitioned global address space languages that enables new optimizations. Next, we propose future work on cooperative optimization of query plans and imperative programs in the context of parallel applications that include queries.