Fábio Oliveira

Testing Idempotence for Infrastructure as Code

Due to the competitiveness of the computing industry, software developers are pressured to quickly deliver new code releases. At the same time, operators are expected to update and keep production systems stable at all times. To overcome the development–operations barrier, organizations have started to adopt Infrastructure as Code (IaC) tools to efficiently deploy middleware and applications using automation scripts. These automations comprise a series of steps that should be idempotent to guarantee repeatability and convergence. Rigorous testing is required to ensure that the system idempotently converges to a desired state, starting from arbitrary states. We propose and evaluate a model-based testing framework for IaC. An abstracted system model is utilized to derive state transition graphs, based on which we systematically generate test cases for the automation. The test cases are executed in light-weight virtual machine environments. Our prototype targets one popular IaC tool (Chef), but the approach is general. We apply our framework to a large base of public IaC scripts written by operators, showing that it correctly detects non-idempotent automations.

Weaver: language and runtime for software defined environments

Continuous delivery of software and related infrastructure environments is a challenging proposition. Typical enterprise environments, comprising distributed software and its supporting infrastructure, exhibit non-obvious, often implicit dependencies and requirements. Further increasing this challenge is that knowledge about configuration is fragmented and informally recorded. Given this situation, we propose Weaver, a domain-specific language designed to formally specify blueprints, desired state descriptions of environments. An associated runtime executes blueprints to create or modify environments through a set of target-specific platform providers that supply cloud-specific implementations. New and existing automation to implement and maintain the desired state can be associated with a blueprint specified in Weaver. Furthermore, Weaver supports the definition of conditions to validate a blueprint at design time and deployment time, as well as to continuously validate a deployed environment. We demonstrate the use of Weaver to deploy IBM Connections, an enterprise social software platform.

AESON: A Model-Driven and Fault Tolerant Composite Deployment Runtime for IaaS Clouds

Infrastructure-as-a-Service (IaaS) cloud environments expose to users the infrastructure of a data center while relieving them from the burden and costs associated with its management and maintenance. IaaS clouds provide an interface by means of which users can create, configure, and control a set of virtual machines that will typically host a composite software service. Given the increasing popularity of this computing paradigm, previous work has focused on modeling composite software services to automate their deployment in IaaS clouds. This work is concerned with the runtime state of composite services during and after deployment. We propose AESON, a deployment runtime that automatically detects node (virtual machine) failures and eventually brings the composite service to the desired deployment state by using information describing relationships between the service components. We have designed AESON as a decentralized peer-to-peer publish/subscribe system leveraging IBMs Bulletin Board (BB), a topic-based distributed shared memory service built on top of an overlay network.

Delivering software with agility and quality in a cloud environment

agility and quality in a cloud environment F. Oliveira T. Eilam P. Nagpurkar C. Isci M. Kalantar W. Segmuller E. Snible Cloud computing and the DevOps movement are two pillars that facilitate software delivery with extreme agility. “Born on the cloud” companies, such as Netflix®, have demonstrated rapid growth to their business and continuous improvement to the service they provide, by reportedly applying DevOps principles. In this paper, we claim that to fulfill the vision of fast software delivery, without compromising the quality of the provided services, we need a new approach to detecting problems, including problems that may have occurred during the continuous deployment cycle. A native DevOps-centric approach to problem resolution puts the focus on a wider range of possible error sources (including code commits), makes use of DevOps metadata to clearly define the source of the problem, and leads to a quick problem resolution. We propose such a continuous quality assurance approach, and we demonstrate it by preliminary experiments in our public Container Cloud environment and in a private OpenStack® cloud environment.

Automated testing of chef automation scripts

Infrastructure as Code (IaC) is a novel approach for deployment of middleware and applications. IaC typically builds on automation scripts to put the system into a specific state. The series of steps in an automation should be idempotent to guarantee repeatability and convergence. These are key factors if automations are run periodically to override out-of-band changes and prevent drifts from the desired state. Rigorous testing must ensure that the system reliably converges from arbitrary initial/intermediate states to a desired state. We tackle this issue and demonstrate our tool for automated testing of automation scripts. Our tool is tailored to Opscodes Chef, one of the most popular IaC frameworks to date. Various testing parameters can be configured, and the Web-based user interface allows to inspect the system state changes during execution. Detailed test reports are created at the end of a test suite, which facilitate tracking down the root cause of failures and issues of non-idempotence.

Context-based analytics: establishing explicit links between runtime traces and source code

Diagnosing problems in large-scale, distributed applications runningin cloud environments requires investigating different sources ofinformation to reason about application state at any given time. Typical sources of information available to developers and operatorsinclude log statements and other runtime information collectedby monitors, such as application and system metrics. Just as importantly, developers rely on information related to changes to the source code andconfiguration files (program code) when troubleshooting. This information is generally scattered, and it is up to the troubleshooterto inspect multiple implicitly-connected fragments thereof. Currently, different tools need to be used in conjunction, e.g., logaggregation tools, source-code management tools, and runtime-metricdashboards, each requiring different data sources and workflows. Notsurprisingly, diagnosing problems is a difficult proposition. In this paper, we propose Context-Based Analytics, an approach that makes the links between runtime informationand program-code fragments explicit by constructing a graph based on anapplication-context model. Implicit connections between informationfragments are explicitly represented as edges in the graph. We designeda framework for expressing application-context models and implemented a prototype. Further, we instantiated our prototype framework with an application-contextmodel for two real cloud applications, one from IBM and another from a major telecommunications provider. We applied context-based analytics to diagnose twoissues taken from the issue tracker of the IBM application and foundthat our approach reduced the effort of diagnosing these issues. In particular, context-based analytics decreased the number of required analysis steps by 48% and the number ofneeded inspected traces by 40% on average as compared to a standard diagnosis approach.