Kelly Neville

Using cognitive task analysis to explore issues in the procurement of intelligent decision support systems

Government statistics and various news reports suggest that upwards of half of all large-scale information technology (IT) development projects fail to meet expectations for facilitating cognitive work. Many of the failures point to the neglect of human-centering considerations during the development of sociotechnical systems. The groups of people who create IT themselves constitute a sociotechnical system. Therefore, laws of cognitive work apply to the cognitive work of IT development, and these laws include the “reductive tendency” for people to form simplified understandings when confronted with domains of dynamics and complexity. In this article, we report a study in which we “turned the tables” on IT systems development. Rather than using cognitive task analysis to study some work domain for which an envisioned IT system would be developed, we used cognitive task analysis to study the work domain of IT systems development itself. Through documentation analysis and critical decision method procedures, we sought to reveal specific challenges with regard to human-centering, and ways in which principles, methods, and tools of ergonomics (human factors, cognitive systems engineering) might help the developers of IT systems better address the human and social aspects of cognitive work. The findings highlight the outstanding challenges and barriers to the procurement and development of usable, useful, and understandable IT for sociotechnical systems. Challenges include the following: the need for better coordination mechanisms; the need to locate cognitive systems engineers, as advocates for workers, in key leadership roles; the need to reconceive concepts and methods of requirements and requirements specification; and the need for better negotiation of the trade-offs of cost/schedule considerations with human-centering considerations.

Challenges to the development of pedagogically driven engineering requirements for complex training systems

Complex operating environments are becoming prevalent within industry and the military. Concomitantly, the development of advance technologies is increasing, enabling the simulation of complex environments and presenting exciting training opportunities. What is lacking is strong guidance from training science for how to focus and enhance the use of simulation technologies for training system design. The focus of this paper is to continue the discussion of this long-standing problem by focusing on three causes: the lack of integration of training research, training practice and technology development; the difficulty of translating research findings into useful design artefacts; and fundamental, culturally based differences between the stakeholders involved in training system development, including programme managers, engineers, psychologists and domain experts. The causes are addressed both to raise awareness and to begin envisioning solutions.

Application of a Work-Centered Design Method to Support Counterspace Operations

A number of challenges hinder the development of systems that support users in the conduct of their work. Challenges include the widespread use and acceptance of design methods that are system-centered rather than work-centered; the time required to develop a work-centered system design; the imprecise nature of translating work domain analysis results into a work-centered design; and unsatisfactory means for coordinating design and design-implementation processes. The Work-centered Infomediary Layer (WIL) design model and method have been developed to address challenges such as these and to facilitate work-centered design in general. In this paper, we describe the application of WIL to the design of a work-centered support system for defensive counterspace (DCS) operators. The design model and method are presented, followed by a description of the resulting system design and ways in which it was shaped and defined through use of the WIL method.

Multidisciplinary Systems Development: Challenges and Lessons Learned

Systems developed by multidisciplinary teams should benefit from the varied backgrounds and diverse contributions of their developers. However, multidisciplinary systems development teams face challenges that can impair team effectiveness and limit the contributions of participating disciplines. These challenges may be major contributing factors to the long-term struggle of the cognitive and human factors engineering disciplines to contribute meaningfully to systems development. To gain insight into these challenges, we reviewed four of our own recent multidisciplinary systems development projects. In each project, cognitive engineers participated with systems and software engineers across the entire development effort. By describing challenges we faced, this paper is intended to draw attention to types of change that may facilitate the participation of multiple disciplines, and of cognitive and human factors engineering in particular, in future systems development efforts.

Going Paperless in Mission Readiness Management: Challenges to Designing a Readiness Management Work Support System

This paper describes a recordkeeping and readiness management support system designed to replace and extend a paper-based system used in a large military organization. The focus of the paper is a set of system design challenges that are particularly relevant to the present effort. These challenges involve gaining user acceptance; anticipating changes and variations in work practices; understanding and accommodating macroergonomic factors; and facilitating information access and use.

An Adaptive Performance Support System for Satellite Ground Systems Operators

Rapid advances in space technology create an unparalleled international dependence on satellite and other spacecraft capabilities. Critical issues arise as a result of inefficient, outdated and costly satellite operations. While more US satellites are deployed with different designs and missions to occupy orbits, both individual and team satellite operator skills and performance support have not kept pace, creating wide disparity and increased deficiencies. The Adaptive Decision Enabling and Proficiency Toolkit (ADEPT) performance support system (PSS) was developed to reduce inadequacies and errors in satellite operations and to facilitate the work of satellite crews. In particular, ADEPT was built to support crews who will manage the Space Tracking and Surveillance System (STSS). The STSS is to become fully operational within the next five years. Operational ground systems are already in place and are being used to manage existing defense system program (DSP) technology until the low earth orbit (LEO) STSS satellites are launched into orbit. STSS will be the majority of LEO systems to be launched with special capabilities and missions to support, which will require a dramatic increase in workload and time pressure, as well as significant changes in the daily duties of the satellite operator. These types of changes led to the development of ADEPT. The goal of the paper will be to describe the development of situated performance support capabilities and the factors that influenced the design of those capabilities. The paper will also describe how the use of a work-centered design approach led to a toolkit that supports skill-based, rule-based, and knowledge-based behavior in the course of satellite operations work. Finally, the paper will present the full ADEPT concept, including descriptions and pictures of the ADEPT tools, and a discussion of how the tools support improved information flow and management for future challenging work conditions.

Using Three-Dimensional Graphics to Enhance the Acquisition of Complex Satellite Systems Knowledge

This paper describes the design of a training system in which three-dimensional (3-D) graphics are employed to help satellite operations trainees learn about satellite systems. The design takes into account training practices and support requirements of the satellite systems instructors who will be using it and ways in which trainees should be able to interact with and use the 3-D graphics. The training system must significantly improve the rate of satellite system knowledge acquisition beyond the rate obtained using current training practices. To this end, training principles that facilitate the acquisition of complex, functional, and integrated knowledge guided the design of a training system infrastructure that puts 3-D graphics to optimal use.

A Da ta Management and Visualization Tool That Benefits Mission Readiness

A significant amount of resources and effort go toward ensuring that all unit personnel maintain a mission ready status, specifically in US Air Force Space Command (AFSPC). Personnel folders used to track and document readiness activities (called Individual Qualification Folders [IQFs]) are paper-based, and this presents additional difficulties on top of basic workload demands. Limitations associated with using paper-based IQFs have the potential to impact the mission readiness of an individual or entire unit. In this paper, we describe the cognitive engineering approach, design process, and concept of operations for an electronic IQF and mission-readiness support system called Records Evaluation, Analysis, and Data Interpretation (READI). READI is a server-based, PC-compatible system designed for use by AFSPC unit operations support staffs (e.g., training, evaluation, and crew force management staffs). It is intended to support basic IQF maintenance and operations support staff work associated with ensuring the mission readiness of unit personnel. The supported work areas encompass 1) recurring personnel training and evaluation; 2) tracking, managing, and scheduling mission-ready requirements and activities to ensure mission-ready requirements are met; and 3) ensuring the quality of IQFs, IQF content, and mission-ready activities and materials. In addition to supporting operations support staffs in AFSPC, READI is relevant to operations support staffs throughout the DoD and potentially to training management outside of the DoD.

Designing Work-Centered Performance Support Solutions for Operators of a Complex and Evolving System

This paper describes the design of a performance support toolset to be used in a complex and changing satellite operations domain. In this paper, we focus on a subset of the design challenges encountered during the process of designing this toolset. This subset of design challenges includes enhancing intra-team communications in a time-limited environment; improving information comprehension; supporting users in both low and a high tempo operations; and supporting users in future conditions and during the transition to those future conditions. This paper is intended to demonstrate the valuable contributions of the research literature and of the domain experts who will be using the toolset to the design of work-centered solutions. In addition, this paper is intended to serve as a resource for other design efforts facing similar challenges.

A Cognitive Task Analysis of Coordination in a Distributed Tactical Team: Implications for Expertise Acquisition

A Cognitive Task Analysis (CTA) was conducted to examine distributed team coordination expertise. Knowledge associated with air wing strike team coordination challenges was elicited from E-2C Naval Flight Officers (NFOs) with varying levels of experience, and was assessed using a methodology consisting of multiple qualitative assessment techniques. Results include: (1) insight into the organizational structure of NFO team coordination knowledge; (2) rich representations of NFO team coordination knowledge that may be used to populate training and performance support tools; (3) experience-related differences in the use of knowledge and skill to support team coordination; and (4) knowledge and skill categories that support team coordination. Each of these results may contribute to the design of performance support tools and training guidelines, strategies, and content that enhances team coordination.