M. David Allen

PLUS: A provenance manager for integrated information

It can be difficult to fully understand the result of integrating information from diverse sources. When all the information comes from a single organization, there is a collective knowledge about where it came from and whether it can be trusted. Unfortunately, once information from multiple organizations is integrated, there is no longer a shared knowledge of the data and its quality. It is often impossible to view and judge the information from a different organization; when errors occur, notification does not always reach all users of the data. We describe how a multi-organizational provenance store that collects provenance from heterogeneous systems addresses these problems. Unlike most provenance systems, we cope with an open world, where the data usage is not determined in advance and can take place across many systems and organizations.

Capturing Provenance in the Wild

All current provenance systems are “closed world” systems; provenance is collected within the confines of a well understood, pre-planned system. However, when users compose services from heterogeneous systems and organizations to form a new application, it is impossible to track the provenance in the new system using currently available work. In this work, we describe the ability to compose multiple provenance-unaware services in an “open world” system and still collect provenance information about their execution. Our approach is implemented using the PLUS provenance system and the open source MULE Enterprise Service Bus. Our evaluations show that this approach is scalable and has minimal overhead.

What do we do now? Workflows for an unpredictable world

Workflow systems permit organization of many individual subtasks into a cohesive whole, in order to accomplish a specific mission. For many government and business missions, these systems are used to manage repetitive processes, such as large data-processing and exploitation pipelines. Government missions with strong interactions with the real world are extremely dynamic, as are all missions dealing with error-prone or changing data streams. We contribute a vision for discovery of new steps in adaptive workflow systems, suitability functions that can discover candidate alternatives, and a way forward for sourcing options for decision-makers, without the strong assumptions required by previous work. As data-processing workflows are shared, the sharing entities may find that certain parts of the workflow must be adapted to the new environment of mission. Extremely dynamic environments call for capabilities that support agile operations and pipeline sharing by making it possible to choose relevant actions when a situation invalidates the assumptions of current execution. We adapt some work in schema matching towards this problem, citing key differences between the two sets of challenges.

Engineering Choices for Open World Provenance

This work outlines engineering decisions required to support a provenance system in an open world where systems are not under any common control and use many different technologies. Real U.S. government applications have shown us the need for specialized identity techniques, flexible storage, scalability testing, protection of sensitive information, and customizable provenance queries. We analyze tradeoffs for approaches to each area, focusing more on maintaining graph connectivity and breadth of capture, rather than on fine-grained/detailed capture as in other works. We implement each technique in the PLUS system, test its real-time efficiency, and describe the results.

Making Query Execution Over Encrypted Data Practical

The benefits of data outsourcing continue to grow, however owners of sensitive data cannot take full advantage due to its risk profile. Encrypted query processing promises to change this situation and allow data owners to securely outsource their sensitive data: data is encrypted, installed in a database on a remote (e.g., cloud) server, and standard queries are processed against the remote encrypted data. Correct query answers are returned without ever exposing plaintexts or decryption keys at the server. This chapter addresses three key challenges to realizing, as a practical option, the promise of encrypted query processing: handling query operations which cannot execute in ciphertext, implementing a working system, and achieving acceptable query performance.

Fit for purpose: engineering principles for selecting an appropriate type of data exchange standard

Abstract Data standards are a powerful, real-world tool for enterprise interoperability, yet there exists no well-grounded methodology for selecting among alternative standards approaches. We focus on a specific sub-problem within a community’s data sharing challenge and identify four major standards-based approaches to that task. We present characteristics of a data sharing community that one should consider in selecting a standards approach—such as relative power, motivation level, and technical sophistication of different participants—and illustrate with real-world examples. These characteristics and other factors are then analyzed to develop decision rules for selecting among the four approaches. Independent of the data exchange problem, we suggest two general practices in choosing a standards approach: (1) vertical decomposition of interoperability issues, in order to define a narrow, formal, tractable problem, and (2) option-exclusion rules, as they are much simpler than stating optimal-choice rules.

Fit for Purpose: Toward an Engineering Basis for Data Exchange Standards

Data standards are a powerful, real-world tool for enterprise interoperability, yet there exists no rigorous methodology for selecting among alternative standards approaches. This paper is a first step toward creating a detailed engineering basis for choosing among standards approaches. We define a specific sub-problem within a community’s data sharing challenge, and focus on it in depth. We describe the major choices (kinds of standards) applied to that task, examining tradeoffs. We present characteristics of a data sharing community that one should consider in selecting a standards approach—such as relative power, motivation level, and technical sophistication of different participants—and illustrate with real-world examples. We then show that one can state simple decision rules (based on engineering experience) that system engineers without decades of data experience can apply. We also comment on the methodology used, extracting lessons (e.g., “negative rules are simpler”) that can be used in similar analyses on other issues.

Intuitive Interaction with Encrypted Query Execution in DataStorm

The encrypted execution of database queries promises powerful security protections, however users are currently unlikely to benefit without significant expertise. In this demonstration, we illustrate a simple workflow enabling users to design secure executions of their queries. The Data Storm system demonstrated simplifies both the design and execution of encrypted execution plans, and represents progress toward the challenge of developing a general planner for encrypted query execution.