Andrew P. Snow

The challenge of accurate software project status reporting: a two-stage model incorporating status errors and reporting bias

Software project managers perceive and report project status. Recognizing that their status perceptions might be wrong and that they may not faithfully report what they believe, leads to a natural question-how different is true software project status from reported status? Here, the authors construct a two-stage model which accounts for project manager errors in perception and bias that might be applied before reporting status to executives. They call the combined effect of errors in perception and bias, project status distortion . The probabilistic model has roots in information theory and uses discrete project status from traffic light reporting. The true statuses of projects of varying risk were elicited from a panel of five experts and formed the model input. The same experts estimated the frequency with which project managers make status errors, while the authors created different bias scenarios in order to investigate the impact of different bias levels. The true status estimates, error estimates, and bias levels allow calculation of perceived and reported status. The results indicate that at the early stage of the development process most software projects are already in trouble, that project managers are overly optimistic in their perceptions, and that executives receive status reports very different from reality, depending on the risk level of the project and the amount of bias applied by the project manager. Key findings suggest that executives should be skeptical of favorable status reports and that for higher risk projects executives should concentrate on decreasing bias if they are to improve the accuracy of project reporting.

The effects of optimistic and pessimistic biasing on software project status reporting

Anecdotal evidence suggests that project managers (PMs) sometime provide biased status reports to management. In our research project we surveyed PMs to explore possible motivations for bias, the frequency with which bias occurs, and the strength of the bias typically applied. We found that status reports were biased 60% of the time and that the bias was twice as likely to be optimistic as pessimistic. By applying these results to an information-theoretic model, we estimated that only about 10-15% of biased project status reports were, in fact, accurate and these occurred only when pessimistic bias offset project management status errors. There appeared to be no significant difference in the type or frequency of bias applied to high-risk versus low-risk projects. Our work should provide a better understanding of software project status reporting.

A Framework for Assessing the Reliability of Software Project Status Reports

Abstract In this article, we propose a probabilistic model that accounts for project manager fallibility in determining project status, and the tendency of project managers to slant the status to be better than that actually perceived. We call the fallibility in determining status error, the tendency to slant perceived status bias, and the combined effect distortion. The results indicate that distortion can produce very large differences between true and reported status. In addition, the results indicate that the magnitude of this distortion is heavily influenced by the level of bias applied by the project manager. This investigation supports the notion that the reliability of status reports is not only dependent upon the skills of the project manager, but also on the culture of the organization. The model also provides a framework for further investigating status report reliability through empirical studies to determine error and bias estimates.

Meeting SLA Availability Guarantees through Engineering Margin

This research deals with availability of Service Level Agreements (SLA) between information technology service providers, such as carriers, and user corporations. A combination of different reliability/maintainability scenarios and time intervals have been used to generate availability distributions in order to assess the efficacy of availability guarantees. Markov and semi-Markov models are used to provide a comparative analysis of the availability distributions and probability of service level violation. In the semi-Markov model, long tail and short tail lognormal distributions are used for the repair time. A Monte Carlo simulation program is developed and from the generated distributions it is found that there are significant chances of SLA violation, given a wide range of different reliability/maintainability levels used to achieve typical goals such as 0.99999 availability for a variety of time intervals. The results are fairly sensitive to the tail of the repair distribution, meaning it is essential to understand repair distribution in order to assess chances of SLA violation. Results indicate that the way to dramatically decrease the probability of SLA violation is through availability engineering margin. Delivering availability beyond 0.99999 through redundancy and responding more rapidly to failures greatly diminishes the chances of violating a 0.99999 SLA.

Using artificial neural networks to enhance CART

Accuracy is a critical factor in predictive modeling. A predictive model such as a decision tree must be accurate to draw conclusions about the system being modeled. This research aims at analyzing and improving the performance of classification and regression trees (CART), a decision tree algorithm, by evaluating and deriving a new methodology based on the performance of real-world data sets that were studied. This paper introduces a new approach to tree induction to improve the efficiency of the CART algorithm by combining the existing functionality of CART with the addition of artificial neural networks (ANNs). Trained ANNs are utilized by the tree induction algorithm by generating new, synthetic data, which have been shown to improve the overall accuracy of the decision tree model when actual training samples are limited. In this paper, traditional decision trees developed by the standard CART methodology are compared with the enhanced decision trees that utilize the ANN’s synthetic data generation, or CART+. This research demonstrates the improved accuracies that can be obtained with CART+, which can ultimately improve the knowledge that can be extracted by researchers about a system being modeled.

Towards an optimal network survivability threshold

Without empirical insights into the survivability and resiliency of networks, operators and maintainers can only speculate about the efficacy of key network infrastructures. Outage reporting is a key aspect of integrated network management, and requires special consideration by network operators. For instance, the outage is to be deemed a threat to network survivability and resiliency. This paper considers the appropriateness of a network survivability reporting threshold by investigating the optimality of different thresholds, and threshold types. We wish to search for an optimal threshold, which minimizes the number of reports, and maximizes the amount of communication loss exposed by the reports. As a consequence, we define here a threshold figure of merit, which is the ratio of the communications loss and the number of outages exposed relative to a baseline reporting threshold, and seek to maximize this metric. Over 19 000 actual network outages are used to investigate optimal reporting thresholds. Results indicate that network operator perceptions of network survivability and resiliency depend not only on the magnitude of the threshold, but also on the threshold type.

Assessing Wireless Network Dependability through Knowledge Extraction via Decision Trees

Critical infrastructures such as wireless network systems demand dependability. Dependability attributes reported here include availability, reliability, maintainability and survivability (ARMS). This research uses computer simulation and knowledge extraction to introduce a new approach to measure dependability of wireless networks. Earlier research has used computer simulation for estimating wireless network dependability. This work introduces a new methodology which uses discrete time event simulation in-put/output to train an artificial neural network and then extract knowledge via decision trees. A comparison of decision tree extraction technique results are discussed, including those from neural (TREPAN) and non neural networks (C4.5). Significant insights are gained into increasing wireless infrastructure dependability through such knowledge extraction techniques; however the neural approach is superior from a parsimonious and comprehensibility perspective.

Collateral damage from anticipated or real disasters: skewed perceptions of system and business continuity risk?

Managers perceive risk, and based upon the perception develop proactive or reactive strategies. The former strategy results in either under investment, optimal investment, or over investment in response to the perceived risk. The latter is a default strategy if a decision is made to ignore a risk as too low for inclusion in disaster recovery plans. Some disaster events may increase in frequency and serve as a wakeup call and raise perceived risk to the appropriate level. Other events may be rare and inflate perceived risk beyond that which is rational because of the event‘s catastrophic size and impact. The size and frequency of disasters are the determining factors in risk perception. In addition, large-scale disasters are episodic and can result in overestimation of risk. What is a systems manager to do? This paper tries to assess this conundrum by reviewing recent disasters and discussing perceived and apparent risks associated with each. The paper concludes that there are no easy answers and that system managers should perhaps assess risk levels in consultation with risk management firms and insurance companies. In addition, specific recommendations are made, including geographic dispersion and insuring that redundant architectures are not defeated by violating key assumptions.

Understanding Danger to Critical Telecom Infrastructure: A Risky Business

Malicious acts aimed directly against humans, or indirectly at their critical infrastructures is a real and present danger. However, how does society quantify danger levels? And is danger different from risk? Classic risk assessments require probability assessments, a highly speculative task for rare events. A “Danger Index”, proposed by Randall Larsen in a recent book, is investigated here. In this exploratory research paper, this metric is examined in the context of current views of risk, and investigates its potential application to telecommunication settings. The danger index is deemed to have potential for assessing critical telecommunication infrastructure protection, avoiding difficult, and often impractical, probability assessments.