Richard Baskerville

Generalizing Generalizability in Information Systems Research

Generalizability is a major concern to those who do, and use, research. Statistical, sampling-based generalizability is well known, but methodologists have long been aware of conceptions of generalizability beyond the statistical. The purpose of this essay is to clarify the concept of generalizability by critically examining its nature, illustrating its use and misuse, and presenting a framework for classifying its different forms. The framework organizes the different forms into four types, which are defined by the distinction between empirical and theoretical kinds of statements. On the one hand, the framework affirms the bounds within which statistical, sampling-based generalizability is legitimate. On the other hand, the framework indicates ways in which researchers in information systems and other fields may properly lay claim to generalizability, and thereby broader relevance, even when their inquiry falls outside the bounds of sampling-based research.

A critical perspective on action research as a method for information systems research

This paper reviews the origins, techniques and roles associated with action research into information systems (IS). Many consider the approach to be the paragon of post-positivist research methods, yet it has a cloudy history among the social sciences. The paper summarizes the rigorous approach to action research and suggests certain domains of ideal use (such as systems development methodology). For those faced with conducting, reviewing or examining action research, the paper discusses various problems, opportunities and strategies.

Growing systems in emergent organizations

requirements Complete and unambiguous specifications New system projects as achievements Tools/Techniques for emergent ISD goals: Reliable back channels for ISD professionals, emergent IT organizations, prototyping, proper rewards system, and existing tools such as open systems, end-user development and object-oriented designs and implementation tools. third goal is an implied response to the revocation of third and fourth items in the old goal set. Always analysis. Under emergent assumptions, the analysis of IS applications must be continuous. Since the organization is emerging, the fundamental IS must continuously change and adapt. In order to implement this adaptation, requirements and specifications are constantly renegotiated. Analysis activities are no longer captured within the early stages of a system’s life cycle. Instead, these activities are an ongoing service of the organizational ISD group. It is important to realize that this ongoing service must not be cyclical (periods of analysis followed by periods of implementation), but is generally a constant ISD activity in parallel with systems operation and maintenance. The results of this ongoing analysis are continuously fed into the maintenance activities. Because of organizational emergence, the underlying ISD service continuously monitors and reappraises the IS support for every business process and organizational activity. Under this goal, analysis is not a component of an ISD project, but an ongoing ISD organizational maintenance activity. Dynamic requirements negotiations. Because the organization is emerging around the users, IS requirements can never be fully specified because users are always in conflict with them. Thus user satisfaction is improbable. Indeed, under this assumption, a setting where users are fully satisfied would be an alarming anomaly. Requirements are no longer determined as part of a project, but become a negotiated outcome of the changing characteristics of an emergent organization and the resources for enhancing or altering the existing IS. An emergent ISD goal is not user satisfaction, but a “healthy” degree of conflict between users and their IS. As requirements conflicts rise, increased negotiation and IS enhancement activities are prescribed. As requirements conflicts fall, ISD activities are decreased. The conflict, negotiation and enhancement are continuous service activities provided to support ongoing business processes. These activities are not necessarily associated with any ISD project. Incomplete and usefully ambiguous specifications. If abstract requirements are largely imaginary, and unambiguous specifications are ineffectual, analysts must come to terms with ambiguity. Because the requirements are in motion, specifications must be kept in a state in which these can be easily adapted for enhancing or modifying the existing system. The goal is a set of specifications wherein each specification is open-ended and easily modified. Complete and unambiguous specifications are only possible for organizations that are totally stable, and waste valuable resources in an emergent setting. System enhancement and modification activities begin to be undertaken even though the specifications are incomplete and ambiguous. These activities “succeed” because they are themselves never completed (the organization is likely to emerge from under the planned enhancements or modifications). Traditionally, the IS is a consequence of the specification. Under the emergent view, the specification is just as equally a consequence of the IS emergence. This parallel emergence leads to both an IS and an ISD process that are incomplete and usefully ambiguous. These last two characteristics represent an excellent foundation for further organizational emergence. Continuous redevelopment. Under emergent assumptions, this goal supplants the current ISD project mentality under which all systems terminate at their obsolescence point. The goal of ISD is to preserve all existing IS applications by continuously enhancing and modifying these to match organizational requirements. The goal of ISD is to prevent system obsolescence and thereby eliminate system termination (and the implied new ISD project). The U.S. railroad system provides a metaphor to illustrate how this ISD approach operates. Today’s railroad systems no longer resemble the railroads of a century ago. The engines, rolling stock, tracks, stations, and signaling have all been replaced with modern elements. There has not been a nationwide development project to replace the entire railroad system. Instead, the railroad system has emerged to match the needs of the nation and the limits of the technology. This emergence is a consequence of continuous enhancements: new tracks added in some areas and new rolling stock purchased when needed, for example. The net effect is an adaptive railroad system. Continuous redevelopment implies that information systems are continuously enhanced and modified such that they are never totally outdated and irreparable. COMMUNICATIONS OF THE ACM August 1999/Vol. 42, No. 8 121 SYSTEMS SHOULD BE UNDER CONSTANT DEVELOPMENT, CAN NEVER BE FULLY SPECIFIED AND ARE SUBJECT TO CONSTANT ADJUSTMENT AND ADAPTATION. There are two interesting implications of continuous redevelopment. The first implication arises from the viewpoint of life-cycle termination as an anomaly. When an IS becomes too expensive to maintain and must be replaced, there is an implied failure on the part of ISD management. ISD management failed to keep the IS maintained in a state that permitted its further redevelopment. In other words, the IS was allowed to decay beyond its economic rescue point. In an emergent setting, the decayed IS probably imposed a long period of rising stable systems drag that limited the organizational ability to emerge. Had the system been continuously redeveloped, the drag would have been reduced and the system life span extended indefinitely. In most traditional ISD organizations, the resources that might be used for continuous redevelopment are paradoxically occupied with system replacement projects. The second interesting implication regards legacy systems and the infamous Y2K problem. These two interconnected problems have risen in importance over the last decade. To a degree, both of these result from the preservation of the 1960s and 1970s ISD project mentality into the 1980s and 1990s. The new systems projects consumed the resources that might have otherwise been applied in gradually redeveloping, enhancing, and modifying these old systems. Under continuous redevelopment, these systems, like the national railroad system, could not be legacy systems. Over the 1980s and 1990s, these legacy systems should have evolved, but didn’t. Today’s ISD managers are now confronted with (and blamed for) the failures of their predecessors. Adaptability orientation. The essential impact of the emergent goal set on ISD relates to the adaptability of IS. Recognizing that IS must undergo continuous redevelopment, the ISD approach and the underlying IS architecture must be conducive to redevelopment. Ease of modification must be deeply embedded in every IS. This easy modification implies that every system includes explicit ISD mechanisms by which the system can adapt. An interesting implication of this goal is the merger of IS and ISD. Development of an IS is exactly the same activity as maintenance, and is equally an essential component of IS operation. The distinction between IS and ISD disappears because the emergence of IS is embodied by the goal set of emergent ISD—an emergent IS is ISD. Ways of Supporting the New ISD Goals The existing vehicles for supporting an effort to reach the emergent organization goals include easily maintainable specifications, open systems interconnection architectures, prototyping, and end-user development. Easily maintainable specifications, like object-oriented designs, make it easier and cheaper to respecify IT systems when change is needed. Open systems architectures enable IT components to be easily rearranged and incorporated with newly developed components. Prototypes, particularly operational prototypes, are typically built with tools that enable easy changes. End-user development uses productivity tools to create inexpensive applications that can be thought of as disposable systems. These existing tools have a role in supporting emergent organizations, but these alone do not go far enough. Several IT organizational capabilities can also help. Back channel communications for ISD professionals. Back channels, such as guaranteed privacy for email, chat rooms, and groupware, permit developers to establish versions of the organizational identity or reality that conflict with other versions. This conflict is important for autopoiesis and emergence. These channels should extend beyond the ISD group and into the users with whom they may interact in order to continuously redevelop systems. Emergent IT organizations. The IT organization itself must be highly emergent. One element that can promote this emergence is virtual teams that extend to include users. These teams lack the history that confines their adaptation, and eliminate the boundary between user and developer. Another important element is the elimination of the “project” as the primary means of organizing IT activities. An emergent IT organization replaces projects with “streams” of redevelopment activity that are continuous as long as the particular IT system requirement is present. A new project represents the failure of the IT organization to properly adapt the systems in its charge. Proper rewards system. The IT organization that supports emergent organizations must value system adaptation. Initially developing adaptable systems is important. However, most of the organization’s important development activities are merged with its maintenance activities. Maintenance needs to become innovative

Grounded action research: a method for understanding IT in practice

Abstract This paper shows how the theory development portion of action research can be made more rigorous. The process of theory formulation is an essential part of action research, yet this process is not well understood. A case study demonstrates how units of analysis and techniques from grounded theory can be integrated into the action research cycle in order to add rigor and reliability to the theory formulation process.

Amethodical systems development: the deferred meaning of systems development methods

Abstract This paper contributes a deeper understanding of the concept of methodical information systems development. The method concept is an assumption underlying much of the research into systems analysis, design and implementation. A postmodern deconstruction technique is used to discover a deferred concept: amethodical systems development. The methodical and amethodical views are developed in terms of their assumptions and their ideal characteristics. Our understanding of these two opposing views of systems development is important as a means to refocus our aims in research, practice and education in information systems development.

Controlling action research projects

Action research (AR), which emphasises collaboration between researchers and practitioners, is a qualitative research method that has much potential for the information systems (IS) field. AR studies of IS phenomena are now beginning to be published in the IS research literature. However, the rigour of many AR studies in IS can be improved. When AR has been published, the findings have frequently been emphasised at the expense of the process. In this article, we look at the process in AR projects, and look at some of the key choices and alternatives in controlling AR. We discuss three aspects of control: the procedures for initiating an AR project, those for determining authority within the project, and the degree of formalisation. We analyse seven recent AR projects in IS and from this analysis distil recommendations for determining these control structures.

Future directions for behavioral information security research

Information Security (InfoSec) research is far reaching and includes many approaches to deal with protecting and mitigating threats to the information assets and technical resources available within computer based systems. Although a predominant weakness in properly securing information assets is the individual user within an organization, much of the focus of extant security research is on technical issues. The purpose of this paper is to highlight future directions for Behavioral InfoSec research, which is a newer, growing area of research. The ensuing paper presents information about challenges currently faced and future directions that Behavioral InfoSec researchers should explore. These areas include separating insider deviant behavior from insider misbehavior, approaches to understanding hackers, improving information security compliance, cross-cultural Behavioral InfoSec research, and data collection and measurement issues in Behavioral InfoSec research.

The Theoretical Foundations of Knowledge Management

Knowledge management has emerged as an important field for practice and research in information systems. This field is building on theoretical foundations from information economics, strategic management, organizational culture, organizational behavior, organizational structure, artificial intelligence, quality management, and organizational performance measurement. These theories are being used as foundations for new concepts that provide a rationale for managing knowledge, define the process of managing knowledge, and enable us to evaluate the results of this process. Based on articles published between 1995 and 2005, new concepts are emerging, including knowledge economy, knowledge alliance, knowledge culture, knowledge organization, knowledge infrastructure, and knowledge equity. An analysis of the theoretical foundations of knowledge management reveals a healthy arena with a strong foundation and clear directions for future work.

A comprehensive framework for evaluation in design science research

Evaluation is a central and essential activity in conducting rigorous Design Science Research (DSR), yet there is surprisingly little guidance about designing the DSR evaluation activity beyond suggesting possible methods that could be used for evaluation. This paper extends the notable exception of the existing framework of Pries-Heje et al [11] to address this problem. The paper proposes an extended DSR evaluation framework together with a DSR evaluation design method that can guide DSR researchers in choosing an appropriate strategy for evaluation of the design artifacts and design theories that form the output from DSR. The extended DSR evaluation framework asks the DSR researcher to consider (as input to the choice of the DSR evaluation strategy) contextual factors of goals, conditions, and constraints on the DSR evaluation, e.g. the type and level of desired rigor, the type of artifact, the need to support formative development of the designed artifacts, the properties of the artifact to be evaluated, and the constraints on resources available, such as time, labor, facilities, expertise, and access to research subjects. The framework and method support matching these in the first instance to one or more DSR evaluation strategies, including the choice of ex ante (prior to artifact construction) versus ex post evaluation (after artifact construction) and naturalistic (e.g., field setting) versus artificial evaluation (e.g., laboratory setting). Based on the recommended evaluation strategy(ies), guidance is provided concerning what methodologies might be appropriate within the chosen strategy(ies).

Agile requirements engineering practices and challenges: an empirical study

This paper describes empirical research into agile requirements engineering (RE) practices. Based on an analysis of data collected in 16 US software development organizations, we identify six agile practices. We also identify seven challenges that are created by the use of these practices. We further analyse how this collection of practices helps mitigate some, while exacerbating other risks in RE. We provide a framework for evaluating the impact and appropriateness of agile RE practices by relating them to RE risks. Two risks that are intractable by agile RE practices emerge from the analysis. First, problems with customer inability and a lack of concurrence among customers significantly impact agile development. Second, risks associated with the neglecting non‐functional requirements such as security and scalability are a serious concern. Developers should carefully evaluate the risk factors in their project environment to understand whether the benefits of agile RE practices outweigh the costs imposed by the challenges.