Kim Weyns

A review of methods for evaluation of maturity models for process improvement

Maturity models are widely used in process improvement. The users of a maturity model should be confident that the weak points of the assessed processes can be found, and that the most valuable changes are introduced. Therefore, the evaluation of maturity models is an important activity. In this paper, a mapping study of the literature on the evaluation of maturity models is presented. Two databases are searched resulting in a set of relevant papers. The identified papers can be classified according to six categories, namely the maturity model under evaluation, type of evaluation, relation of the evaluators/authors to the maturity model, level of objectivity, main purpose of the paper, and size of study. Further, a framework of different evaluations of maturity models is developed, and the relevant papers are mapped to the framework. Finally, the relevant research on the evaluation of the maturity models in the Capability Maturity Model family is discussed in more detail. The result of this mapping study is a clear overview of how the evaluation of maturity models has been done, and some discussions are provided for further research on the evaluation of commonly used or newly developed maturity models. Copyright

Sensitivity of Website Reliability to Usage Profile Changes

To measure the reliability of a website from a users point of view, the uncertainly on the usage of the website has to be taken into account. In this paper we investigate the influence of this uncertainly on the reliability estimate for a web server. For this purpose a session based Markov model is used to model the usage extracted from the servers logfiles. From these logfiles a complete user profile can be extracted together with an estimate of the uncertainty on this user profile. This paper investigates the applicability of this kind of Markov model on web server reliability and discusses the difficulties with data extraction from the logfiles. Advantages and disadvantages of this approach are discussed and the approach is applied to data from a university departments web server to demonstrate its applicability.

A review of research on risk analysis methods for IT systems

Context: At the same time as our dependence on IT systems increases, the number of reports of problems caused by failures of critical IT systems has also increased. This means that there is a need for risk analysis in the development of this kind of systems. Risk analysis of technical systems has a long history in mechanical and electrical engineering. Objective: Even if a number of methods for risk analysis of technical systems exist, the failure behavior of information systems is typically very different from mechanical systems. Therefore, risk analysis of IT systems requires different risk analysis techniques, or at least adaptations of traditional approaches. This means that there is a need to understand what types of methods are available for IT systems and what research that has been conducted on these methods. Method: In this paper we present a systematic mapping study on risk analysis for IT systems. 1086 unique papers were identified in a database search and 57 papers were identified as relevant for this study. These papers were classified based on 5 different criteria. Results: This classification, for example, shows that most of the discussed risk analysis methods are qualitative and not quantitative and that most of the risk analysis methods that are presented in these papers are developed for IT systems in general and not for specific types of IT system. Conclusions: The results show that many new risk analysis methods have been proposed in the last decade but even more that there is a need for more empirical evaluations of the different risk analysis methods. Many papers were identified that propose new risk analysis methods, but few papers discuss a systematic evaluation of these methods or a comparison of different methods based on empirical data.

Sensitivity of software system reliability to usage profile changes

Usage profiles are an important factor in software system reliability estimation. To assess the sensitivity of a systems reliability to changes in the usage profile, a Markov based system model is used. With the help of this model, the statistical sensitivity to many independent changes can be estimated. The theory supports both absolute and relative changes and can be used for systems with or without a terminal state. With this approach it is possible to very quickly estimate the uncertainty on the predicted reliability calculated from a Markov model based upon the uncertainty on the usage profile. Finally the theory is applied to an example to illustrate its use and to show its validity.

A maturity model for IT dependability in emergency management

In many organisations a gap exists between IT management and emergency managemement. This paper illustrates how process improvement based on a maturity model can be used to help organisations to evaluate and improve the way they include IT dependability information in their emergency management. This paper presents the IDEM3 (IT Dependability in Emergency Management Maturity Model) process improvement framework which focuses especially on the cooperation between IT personnel, emergency managers, and users, to proactively prevent IT dependability problems when the IT systems are most critical in emergency situations. This paper describes the details of the framework, how the framework was developed and its relation to other maturity models in related fields.

A Maturity Model for IT Dependability in Emergency Management

In many organisations a gap exists between IT management and emergency managemement. This paper illustrates how process improvement based on a maturity model can be used to help organisations to evaluate and improve the way they include IT dependability information in their emergency management. This paper presents the IDEM3 (IT Dependability in Emergency Management Maturity Model) process improvement framework which focuses especially on the cooperation between IT personnel, emergency managers, and users, to proactively prevent IT dependability problems when the IT systems are most critical in emergency situations. This paper describes the details of the framework, how the framework was developed and its relation to other maturity models in related fields. (Less)

Service level agreements in Municipal IT dependability management

Service Level Agreements (SLA) are considered a good practice not only for IT outsourcing but also for IT management within an organisation. In this paper we study the usage of SLAs in municipal IT management. Municipal IT management traditionally involves a large organisation, often with a low IT maturity, but with high requirements on software quality and information security. In this study, a series of interviews was conducted with IT managers from a number of Swedish municipalities which shows a need for more support for municipalities trying to write and improve their SLAs. Based on a number of SLAs collected, this study proposes a specific SLA template for municipal IT management that focuses especially on information security issues. In a final step, the proposed template was evaluated in a focus group meeting with practitioners from Swedish municipalities.

Evaluation of a Maturity Model for IT Dependability in Emergency Management

The IDEM3 maturity model is a process improvement framework that can be used by an organisation to assess and improve their IT dependability management processes. The framework focuses on the coordination of IT management and safety management within an organisation. In this paper, an evaluation plan for the maturity model is presented to evaluate its applicability, assessment accuracy, and practical value. Based on this plan, two evaluations were carried out in two case studies at two Swedish hospitals. The evaluations indicate that the IDEM3 maturity model can be a valuable tool for an organisation to quickly identify the main strengths and weaknesses of the organisation in the field of IT dependability management. Therefore, this study provides a strong argument for a further evaluation based on the developed evaluation plan. (Less)

Identification of IT Incidents for Improved Risk Analysis by Using Machine Learning

Today almost every system or service is dependent on IT systems, and failure of these systems have serious and negative effects on the society. IT incidents are critical for the society as they can stop the function of critical systems and services. Therefore, it is important to analyze these systems for potential risks before becoming dependent on them. Moreover, in a software engineering context risk analysis is an important activity for the development and operation of safe software intensive systems. However, the increased complexity and size of software-intensive systems put additional requirements on the effectiveness of the risk analysis process. This means that the risk analysis process needs to be improved and it is believed that this can be done by having an overview of already occurred IT incidents. This study investigates how difficult it is to find relevant risks from available sources and the effort required to set up such a system. It also investigates the accuracy of the found risks. In this study 58% of texts that potentially can contain information about IT incidents were correctly identified from an experiment dataset by using the presented method. It is concluded that the identifying texts about IT incidents with automated methods like the one presented in this study is possible, but it requires some effort to set up.