Ram Chillarege

Early warning of failures through alarm analysis a case study in telecom voice mail systems

This paper presents three principles for alarm analysis to develop early warnings of failure. The concepts are illustrated with real data and experience drawn from Comverse voice mail systems installed by telecom service providers worldwide. The case study demonstrates practicality of the approach and insight to further advance this body of research.

The effect of system workload on error latency: an experimental study

In this paper, a methodology for determining and characterizing error latency is developed. The method is based on real workload data, gathered by an experiment instrumented on a VAX 11/780 during the normal workload cycle of the installation. This is the first attempt at jointly studying error latency and workload variations in a full production system. Distributions of error latency were generated by simulating the occurrence of faults under varying workload conditions. A family of error latency distributions so generated illustrate that error latency is not so much a function of when in time a fault occurred but rather a function of the workload that followed the failure. The study finds that the mean error latency varies by a 1 to 8 (hours) ratio between high and low workloads. The method is general and can be applied to any system.

An experimental study of memory fault latency

The difficulty with the measurement of fault latency is due to the lack of observability of the fault occurrence and error generation instants in a production environment. The authors describe an experiment, using data from a VAX 11/780 under real workload, to study fault latency in the memory subsystem accurately. Fault latency distributions are generated for stuck-at-zero (s-a-0) and stuck-at-one (s-a-1) permanent fault models. The results show that the mean fault latency of an s-a-0 fault is nearly five times that of the s-a-1 fault. An analysis of variance is performed to quantify the relative influence of different workload measures on the evaluated latency. >

Comparing four case studies on Bohr-Mandel characteristics using ODC

This paper uses four case studies to examine the difference in properties of Bohr-Mandel bugs. The mechanism used to differentiate Bohr versus Mandel bugs are the ODC Triggers that was developed in a previous study on this subject. In this study, the method is extended to reflect on two additional dimensions. First, on the customer perceived impact. And, second, on how these change between their manifestation in production or the field usage versus late stage development or quality assurance testing. This paper: ; Compares Bohr and Mandel bugs rates between customer/field usage and pre-release system testing. ; Finds that Mandel bugs predominantly have a Reliability-Availability-Serviceability (RAS) Impact. ; Finds that Mandel bugs, rarely, if ever have a Functional Impact. ; Finds these studies predict Mandel bug rates consistent with other studies. ; Finds that pre-release testing found very few Mandel bugs (<;10%).

Diagnosing Development Turbulence in Agile Processes

When the smooth flow of an Agile development project is disrupted by one or more offending process problems, turbulence sets in. Diagnosing the exact source of turbulence and rapidly remedying it is key to success. The challenge is compounded by fact that process problems are often multi-fault. Furthermore, similar symptoms can originate from different process problems, and intuition becomes inadequate for accurate diagnosis. This paper illustrates how turbulence needs to be diagnosed and addressed.

Using ODC to diagnose an Agile enterprise application development project

This paper presents a case study using Orthogonal Defect Classification (ODC) as a diagnostic tool to understand the quality and productivity issues in a Agile enterprise web development project. The project spanned a little over two years and had an effort in the range of 100,000 hours. The use of Agile was new to this engineering team. However, the senior people in the team had several years of experience with different process models. The project used a Scrum development model and ran three week Sprints. The objective of the ODC analysis was to understand the dynamics of the development as it actually occurred - as opposed to what was on paper or believed to have been implemented. We also wanted to understand the quality of the product produced, its stability and causes of customer pain. The development process had a separate Quality Assurance (QA) function, and a development led test and inspection effort. We wanted to understand their individual effectiveness, and how the structure effected production level quality. This case study was conducted by the senior engineering team along with ODC expertise from our firm. A subset of the application (around 4 major components) were chosen for this study. This allowed for an examination of component level issues, versus systemic issues across the entire application.

Trigger and Impact Profiles Enable Outcome Focused Defect Discovery to Manage Software Quality

Late life failures in software occur primarily due to certain Triggers that are inadequately present in the development and test processes. Triggers surface faults that are otherwise dormant driven by workloads, configurations, traffic, etc. This case study presents Trigger and Impact distributions across five stages of testing in development and early customer trials. These distribution changes and their implications enable the study, design and prediction of an outcome focused method to gain high quality. The insight from the data sheds light on the possibilities for late life failures and their consequences.

Targeting Specific Customer Satisfactions Issues with ODC Analysis

Often, the task of quality improvement is complicated by having defect backlogs and process gaps that seem too hard to tackle in a short period of time. This is particularly true of most development organizations that have performed root cause analysis or process assessments which identify long list of potential improvement opportunities. The list include identification of one or more of a variety of issues - such as architecture weaknesses, test coverage inadequacy, inadequate regression opportunity, weak code review or inadequacy of appropriate skills.