David D. Woods

FINDING DECISION SUPPORT REQUIREMENTS FOR EFFECTIVE INTELLIGENCE ANALYSIS TOOLS

Within ARDAs GI2Vis program, we developed a unique framework for the definition of decision support requirements for intelligence analysis tools. This framework, based on a first-of-a-kind integration of a model of inferential analysis and principles for designing effective human-computer teams from Cognitive Systems Engineering, has defined the essential support functions to be provided to the intelligence analyst(s). This model has proven to be extremely useful in assessing the support provided by a large set of visualization tools. This assessment has identified clusters of support functions that are addressed by many tools as well as key missing support functions. In this way, the Support Function Model has been used to identify gaps in the support function coverage of existing tools. This can serve as a valuable focusing mechanism for future design and development efforts. In addition, we believe this would be a useful mechanism to enhance cross-discussions among research teams involved in Cognitive Task Analysis efforts within the Intelligence Community. Having others integrate their analytic results with this framework would provide the mechanism for expansion of this model to become a more robust tool and have an even greater impact on the Intelligence Community.

Patterns in Cooperative Cognition

The goal of this panel is to discuss and begin to converge on what constitutes generalizable patterns in cooperative cognition. The premise is that by pulling together findings and observations that hold across different research perspectives, domains, and methodologies, we will further our understanding of the fundamentally cooperative nature of cognition. This understanding is central to the mission of human factors research, which is to provide design guidance and insight for new classes of innovative support tools.

Judging Sufficiency: How Professional Intelligence Analysts Assess Analytical Rigor

This study examines how professional intelligence analysts judge the rigor behind an analysis. The study investigates the challenges that inhibit the understanding of rigor in intelligence analysis and explores cues used by analysts to identify analytic rigor—or lack of rigor. Nine professional intelligence analysts participated in a modified elicitation by critiquing method study, embedded in a scenario walkthrough. Findings from the study indicate that, while professional intelligence analysts can make perceptive assessments about the quality of an analysis process based on product quality, these perceptions are apt to change with insight into the analytic process.

Supporting the Cognitive Work of Information Analysis and Synthesis: A Study of the Military Intelligence Domain

Information Analysis and Synthesis (henceforth “IAS”) is a type of cognitive work that plays a key role in many high-performance, complex, and mission-critical domains. These can range from tactical military intelligence to scientific or technological forecasting, business and financial intelligence to national strategic counterterrorism, and include areas as disparate as geopolitical policy analysis to computer network intrusion detection. The specific subject of this study is the military intelligence domain as one instantiation of IAS. Several innovative ethnographic and cognitive task analysis methods were used to observe team-based distributed work done by actual domain practitioners. The main investigative effort took the form of a scaled-world study, leveraging a real world tactical intelligence training exercise as a natural laboratory for investigating the contrasts between weaker and stronger IAS. Specifically, we examined the role of instructors in providing broadening checks to the team analytic process, and mapped the findings to an existing framework.

Cooperative Problem-Solving Challenges for the Movement of Aircraft on the Ground

In this paper, we discuss issues surrounding aircraft surface movement that were uncovered through a series of ethnographic investigations, including (a) observations of surface movement operations, (b) structured interviews with airline ramp control and dispatch personnel, and (c) critical incident reports. These results are part of the “problem identification” stage of a study aimed at the design of cognitive tools to improve the safety and efficiency of aircraft surface movement. This study is focusing on such issues from an airline operations control perspective (i.e., ramp control and dispatch). We have identified areas of opportunity for aiding airline staff and controllers in this information-intensive cognitive work, which will lead to identifying the nature of cognitive tools and procedures that could improve prioritization, planning, and coordination during surface movement

New Approaches to Overcoming E-Mail Overload

As e-mail has become the preferred medium for communication, the inbox and mail folders become one hub for organizing activities and schedule. The combination of a natural rise in message volume and the large amounts of unsolicited bulk messages (spam) have led some to suggest that the usefulness of e-mail is at an end; users feel overwhelmed by the sheer numbers of individual messages and the effort to manage the inbox. This paper frames the issue of e-mail/message overload as a specific example of data overload and uses previous results to suggest three design concepts – cognitive buoyancy, e-mail constellations, and the intelligent subject line – for use in e-mail.

Message Overload from the Inbox to Intelligence Analysis: How Spam and Blogs Point to New Tools

Patterns of responses to “message overload” can be seen in the ways in which people adapt messaging systems and capabilities. Blogging is an effective and increasingly popular decentralized form of group communication that is proving useful in helping people find and share what is informative. We look to blogging for clues to new solutions to the problem of “data overload” in the world of email. These design solutions to email overload go beyond efforts to block spam, and are based on shifting the basic unit of organization toward communication relationships that allow patterns in communications to emerge.

Shifts in Functions of a New Technology over Time: An Analysis of Logged Electronic Intensive Care Unit Interventions

A study of logged interventions at an electronic intensive care unit (e-ICU) was conducted to examine how functions changed over a two-year period. In total, 2301 log entries of e-ICU interventions from 2005 and 2007 were uniquely coded as to function. A Chi square goodness of fit analysis revealed that 7 out of 11 functions (64%) significantly changed over the two years that were measured. There were increases seen by the log data in the e-ICU nurses calling ICU nurses to supply missing information, recommend the use of best practices, and providing education to ICU nurses. Additionally, increases were seen for e-ICU physicians receiving requests for ordering actions to be taken on patients. Decreases were seen in e-ICU nurses communicating critical lab results and vital sign changes to ICU nurses. We discuss how these shifts relate to several of our predicted archetypical patterns for how new technologies change over time, both in terms of their primary functions as well as changes to positive and negative “unintended” consequences on secondary functions.

Evaluating the Effectiveness of a Joint Cognitive System: Metrics, Techniques, and Frameworks

An implication of Cognitive Systems Engineering is that joint cognitive systems (JCS; also known as complex socio-technical systems) need to be evaluated for its effectiveness in performing the complex cognitive work requirements. This requires using measures that go well beyond “typical” performance metrics such as the number of subtask goals achieved per person per unit of time and the corresponding simple baseline comparisons or workload assessment metrics. This JCS perspective implies that the system must be designed and evaluated from the perspective of the shift in role of the human supervisor. This imposes new types of requirements on the human operator. Previous research in CSE and our own experience has lead us to identify a set of generic JCS support requirements that apply to cognitive work by any cognitive agent or any set of cognitive agents, including teams of people and machine agents. Metrics will have to reflect such phenomena as “teamwork” or “resilience” of a JCS. This places new burdens on evaluation techniques and frameworks, since metrics should be generated from a principled approach and based on fundamental principles of interest to the designers of the JCS. An implication of the JCS perspective is that complex and cognitive systems need to be evaluated for usability, usefulness, and understandability; each of which goes well beyond raw performance. However, conceptually-grounded evaluation frameworks, corresponding operational techniques, and corresponding measures for these are limited. Therefore, in order to advance the state of the field, we have gathered a set of researchers and practitioners to present recent evaluation work to stimulate discussion.

A simulation study of computer-supported inferential analysis under data overload

A simulation study of inferential analysis under data overload was conducted with professional intelligence analysts. Using a process tracing methodology, patterns in information sampling and sources of inaccurate statements were identified when analysts were asked to analyze something outside their base of expertise, were tasked with a tight deadline, and had a large data set. The main contribution from this study is a better understanding of potential vulnerabilities in inferential analysis in challenging situations. These vulnerabilities are informative because they point to a set of design criteria that human-centered solutions to data overload should meet in order to be useful. These evaluation criteria are interesting, in part, because they are so difficult to address. They are not amenable to simple, straightforward adjustments or feature additions to current tools. Meeting these design criteria will require innovative design concepts.