Gary L. Klein

The TIPSTER SUMMAC Text Summarization Evaluation

The TIPSTER Text Summarization Evaluation (SUMMAC) has established definitively that automatic text summarization is very effective in relevance assessment tasks. Summaries as short as 17% of full text length sped up decision-making by almost a factor of 2 with no statistically significant degradation in F-score accuracy. SUMMAC has also introduced a new intrinsic method for automated evaluation of informative summaries.

SUMMAC: a text summarization evaluation

The TIPSTER Text Summarization Evaluation (SUMMAC) has developed several new extrinsic and intrinsic methods for evaluating summaries. It has established definitively that automatic text summarization is very effective in relevance assessment tasks on news articles. Summaries as short as 17% of full text length sped up decision-making by almost a factor of 2 with no statistically significant degradation in accuracy. Analysis of feedback forms filled in after each decision indicated that the intelligibility of present-day machine-generated summaries is high. Systems that performed most accurately in the production of indicative and informative topic-related summaries used term frequency and co-occurrence statistics, and vocabulary overlap comparisons between text passages. However, in the absence of a topic, these statistical methods do not appear to provide any additional leverage: in the case of generic summaries, the systems were indistinguishable in accuracy. The paper discusses some of the tradeoffs and challenges faced by the evaluation, and also lists some of the lessons learned, impacts, and possible future directions. The evaluation methods used in the SUMMAC evaluation are of interest to both summarization evaluation as well as evaluation of other ‘output-related’ NLP technologies, where there may be many potentially acceptable outputs, with no automatic way to compare them.

Supporting Complex Decision Making Through Option Awareness

This article describes the theoretical underpinnings and preliminary experimental support for option awareness (OA): the perception and comprehension of the relative desirability of available options, as well the underlying factors and trade-offs that explain that desirability. The authors’ research has produced a body of theory and experimental findings supporting the potential for OA to beneficially augment situation awareness (SA) and help decision makers identify the most robust options: those that are most likely to turn out well under the widest range of possible future conditions. OA incorporates perspectives from rationalistic and naturalistic models of decision making, as both are used concurrently in the types of complex high-technology work the authors have examined, including emergency management, infectious disease containment, and air traffic control. The authors have developed approaches to support OA through the use of exploratory modeling and visual analytics. These systems were tested over the course of four human-in-the-loop experiments. The results demonstrate the value of this approach to improve decision accuracy, confidence, and speed for decision makers facing scenarios at varying levels of difficulty. The methodology described here provides a framework to move forward with research on supporting OA in complex and uncertain scenarios in a variety of task domains.

Weighing decisions: Aiding emergency response decision making via option awareness

Emergency response is an especially challenging domain for decision support, as often high-impact decisions must be made quickly, usually under high levels of uncertainty about the situation. The level of uncertainty and the time pressure require a new approach to using modelbased decision support tools during ongoing emergency events. This paper discusses the latest in a series of experiments examining the use of a decision-space visualization helping users identify the most robust options in response to complex emergency scenarios. The results provide additional insight into the value of decision-space information and option awareness for users working in complex, emerging, and uncertain task environments.

Towards methodological approaches to meet the challenges of human, social, cultural, and behavioral (HSCB) modeling

The international developments of the recent years resulted in a radical change of tasks the armed forces are conducting. Supporting M&S methods and tools can no longer focus on attrition, movement, and warfighting operations, but need to address reconstruction, crisis prevention, police tasks, and related efforts that are conducted in collaboration with or in support of civil agencies and organizations. A “whole of society” approach is needed, focusing on human, social, cultural, and behavioral (HSCB) modeling. This paper summarizes the position papers of a group of international experts in this emerging domain looking a methodological support to define a body of knowledge, establish communities of interests, integrate operationally relevant data, and work towards a supporting framework, and was prepared in preparation of a panel discussion during the Winter Simulation Conference 2010.

Modeling Knowledge Using a Crowd Of Experts

This paper describes a series of methodological enhancements made to the model-building method in DESIM (Descriptive to Executable Simulation Modeling), which is a participatory modeling technique that converts qualitative models drawn from subject-matter experts into quantitative models that can be used for decision support. The knowledge elicitation methods of three studies are reviewed and evaluated in terms of the quality of the models they produced and their ability to generate actionable information for decision makers. The results, which can be applied across other participatory modeling techniques, demonstrate the practical impacts of different knowledge elicitation strategies for modeling expert knowledge, including the time and cost of acquiring qualified experts and the level of hands-on involvement of experts in model building.

Decision space visualization: lessons learned and design principles

While the situation space consists of facts about what is currently happening, the decision space consists of analytical information that supports comparing the relative desirability of one decision option versus another. We have focused on new approaches to display decision space information that aids cognition and confidence. As a result of our earlier empirical work, we have developed a set of principles for visualizing decision space information. This paper describes those principles and illustrates their use.

Challenges and Prospects for the Paradigm of Naturalistic Decision Making

This is a report on developments in naturalistic decision making (NDM) with respect to current challenges and prospects discussed at the 2015 NDM International Conference. Emphasis is placed on orienting scientific resources to address the challenges expressed by the Human Systems Priority Steering Council. Participants and presenters at the NDM conference were asked to discuss ways in which the NDM paradigm can be extended and applied to address current and emerging national, international, and societal challenges.

Approaches to Cognitive Bias in Serious Games for Critical Thinking

This Panel discusses decision and learning theory and application in the design of serious games to train people for bias-free critical thinking, particularly in analytical domains. The theoretical understanding of cognitive bias will determine the shape of the cognitive work that is entrained by the games. Panelists will describe an approach to bias mitigation training that is premised on a notion that reasoning strategies can have heuristic value depending on circumstances.

Elements needed to support a crisis management collaboration framework

The national airspace security mission objective is to prevent or counter attacks on air vehicles, including attempts to use aircraft as weapons. Execution of this mission requires coordination and collaboration by both civil and military government agencies as well as the private sector, with tightly choreographed activities from a number of independent organizations in multiple locations. However, there is not yet an accepted multi-agency collaboration framework. The elements needed to support such a framework are proposed in this paper, the result of a MITRE Sponsored Research (MSR) project. It addresses the multiple characteristics of joint crisis management collaboration: unrelated organizations, non-collocation, and time sensitivity.