Len Bass

Detecting cloud provisioning errors using an annotated process model

In this paper, we demonstrate the feasibility of annotating a process model with assertions to detect errors in cloud provisioning in near real time. Our proposed workflow is: a) construct a process model of the desired provisioning activities using log data, b) use the process model to determine appropriate annotation triggers and annotate the process model with assertions, c) use the process model to monitor the deployment logs as they are generated, d) trigger the assertion checking based on process activities and log entries, and e) check the assertions to determine errors. For a production deployment tool, Asgard, we have implemented the steps involving constructing a process model, using the model to determine appropriate annotation triggers, triggering the annotation checking based on Asgard log files, and detecting errors. Our prototype has detected errors that cross deployment tool boundaries and go undetected by Asgard; it further has detected other errors substantially more quickly than Asgard would have.

Incorporating Uncertainty into In-Cloud Application Deployment Decisions for Availability

Cloud consumers have a variety of deployment related techniques, such as auto-scaling policies and recovery strategies, for dealing with the uncertainties in the cloud. Uncertainties can be characterized as stochastic (such as failures, disasters, and workload spikes) and subjective (such as choice among various deployment options). Cloud consumers must consider both stochastic and subjective uncertainties. Analytic support for consumers in selecting appropriate techniques and setting the required parameters in the face of different types of uncertainty is currently limited. In this paper, we propose a set of application availability analysis models that capture subjective uncertainties in addition to stochastic uncertainties. We built and validated the models by using industry best practices on deployment, and actual commercial products for disaster recovery and live migration. Our results show that the models permit more informed and quantitative availability analysis than industry best practices under a wide range of scenarios.

Eliciting operations requirements for applications

The DevOps community advocates communication between the operations staff and the development staff as a means of ensuring that the developers understand the issues associated with operations. This paper argues that “communication” is too vague and that there are a variety of specific and well known sources that developers can examine to determine requirements to support the installation and operations of an application product. These sources include standards, process descriptions, studies about sources of failure in configuration and upgrade, and models that include both product and process.

Rollup: Non-Disruptive Rolling Upgrade with Fast Consensus-Based Dynamic Reconfigurations

Rolling upgrade consists of upgrading progressively the servers of a distributed system to reduce service downtime.Upgrading a subset of servers requires a well-engineered cluster membership protocol to maintain, in the meantime, the availability of the system state. Existing cluster membership reconfigurations, like CoreOS etcd, rely on a primary not only for reconfiguration but also for storing information. At any moment, there can be at most one primary, whose replacement induces disruption. We propose Rollup, a non-disruptive rolling upgrade protocol with a fast consensus-based reconfiguration. Rollup relies on a candidate leader only for the reconfiguration and scalable biquorums for service requests. While Rollup implements a non-disruptive cluster membership protocol, it does not offer a full-fledged coordination service. We analyzed Rollup theoretically and experimentally on an isolated network of 26 physical machines and an Amazon EC2 cluster of 59 virtual machines. Our results show an 8-fold speedup compared to a rolling upgrade based on a primary for reconfiguration.

Making Real Time Data Analytics Available as a Service

Conducting (big) data analytics in an organization is not just about using a processing framework (e.g. Hadoop/Spark) to learn a model from data currently in a single file system (e.g. HDFS). We frequently need to pipeline real time data from other systems into the processing framework, and continually update the learned model. The processing frameworks need to be easily invokable for different purposes to produce different models. The model and the subsequent model updates need to be integrated with a product that may require a real time prediction using the latest trained model. All these need to be shared among different teams in the organization for different data analytics purposes. In this paper, we propose a real time data-analytics-as-service architecture that uses RESTful web services to wrap and integrate data services, dynamic model training services (supported by big data processing framework), prediction services and the product that uses the models. We discuss the challenges in wrapping big data processing frameworks as services and other architecturally significant factors that affect system reliability, real time performance and prediction accuracy. We evaluate our architecture using a log-driven system operation anomaly detection system where staleness of data used in model training, speed of model update and prediction are critical requirements.

Securing a deployment pipeline

At the RELENG 2014 Q&A, the question was asked, “What is your greatest concern?” and the response was “someone subverting our deployment pipeline”. That is the motivation for this paper. We explore what it means to subvert a pipeline and provide several different scenarios of subversion. We then focus on the issue of securing a pipeline. As a result, we provide an engineering process that is based on having trusted components mediate access to sensitive portions of the pipeline from other components, which can remain untrusted. Applying our process to a pipeline we constructed involving Chef, Jenkins, Docker, Github, and AWS, we find that some aspects of our process result in easy to make changes to the pipeline, whereas others are more difficult. Consequently, we have developed a design that hardens the pipeline, although it does not yet completely secure it.

Improving Availability of Cloud-Based Applications through Deployment Choices

Deployment choices are critical in determining the availability of applications running in a cloud. But choosing good deployment for various software application components into virtual machines is a challenging task because of potential sharing of components among applications and potential interference from multi-tenancy. This paper presents an approach for improving the availability guarantee of software applications by optimizing the availability, performance and monetary cost trade-offs of different deployment choices. Our approach explicitly considers different classes of application requests during the decision process. The results of our experimental evaluation show that the approach can effectively improve the availability guarantees with little or negligible increase in the performance and monetary cost of the deployment choice.

Achieving Reliable High-Frequency Releases in Cloud Environments

Continuous delivery and deployment are dramatically shortening release cycles from months to hours. Cloud applications with high-frequency releases often rely heavily on automated tools and cloud infrastructure APIs to deploy new software versions. The authors report on reliability issues and how these tools and APIs contribute to them. They also analyze the trade-offs between using heavily baked and lightly baked virtual-image approaches, on the basis of experiments with Amazon Web Service OpsWorks APIs and the Chef configuration management tool. Finally, they propose error-handling practices for continuous-delivery facilities.

Formal specifications better than function points for code sizing

Size and effort estimation is a significant challenge for the management of large-scale formal verification projects. We report on an initial study of relationships between the sizes of artefacts from the development of seL4, a formally-verified embedded systems microkernel. For each API function we first determined its COSMIC Function Point (CFP) count (based on the seL4 user manual), then sliced the formal specifications and source code, and performed a normalised line count on these artefact slices. We found strong and significant relationships between the sizes of the artefact slices, but no significant relationships between them and the CFP counts. Our finding that CFP is poorly correlated with lines of code is based on just one system, but is largely consistent with prior literature. We find CFP is also poorly correlated with the size of formal specifications. Nonetheless, lines of formal specification correlate with lines of source code, and this may provide a basis for size prediction in future formal verification projects. In future work we will investigate proof sizing.

Discovering and Visualizing Operations Processes with POD-Discovery and POD-Viz

Understanding the behavior of an operations process and capturing it as an abstract process model has been shown to improve dependability significantly [1]. In particular, process context can be used for error detection, diagnosis, and even automated recovery. Creating the process model is an essential step in determining process context and, consequently, improving dependability. This paper describes two systems. The first, POD-Discovery, simplifies the creation of such an abstract process model from operations logs. An activity that previously required many manual steps can now be done largely automatically and in minutes. Using the discovered model, the second system, POD-Viz, provides operators with the ability to visualize the current state of an operations process in near-real-time and to replay a set of events to understand how the process context changed over time. This allows operators to trace the progress of an operations process easily, and helps in analyzing encountered errors.