Mark S. Pfaff

Modeling Knowledge Using a Crowd Of Experts

This paper describes a series of methodological enhancements made to the model-building method in DESIM (Descriptive to Executable Simulation Modeling), which is a participatory modeling technique that converts qualitative models drawn from subject-matter experts into quantitative models that can be used for decision support. The knowledge elicitation methods of three studies are reviewed and evaluated in terms of the quality of the models they produced and their ability to generate actionable information for decision makers. The results, which can be applied across other participatory modeling techniques, demonstrate the practical impacts of different knowledge elicitation strategies for modeling expert knowledge, including the time and cost of acquiring qualified experts and the level of hands-on involvement of experts in model building.

The Heuristic Quality Scale

This paper presents the Heuristic Quality Scale (HQS), an instrument designed to assess the quality of a published heuristic set. The purpose of this instrument is 2-fold: first, to aid the evaluator in selecting between prospective specialized heuristic sets for a given evaluation; second, to suggest considerations and guidelines for the creation and publication of new heuristic sets. The HQS was developed over a series of studies evaluating and refining iterations of the instrument while also polling the user experience community at large to identify which factors truly predict the quality of a particular set of heuristics. The contribution of the HQS is that it is a low-cost heuristic evaluation (HE) to apply to heuristic sets themselves. It produces a reliable subjective measure of the quality of a heuristic set and helps the usability professional rank and compare competing options for conducting HEs.

Advances of Research in Affective Processes in Communication and Collaboration

Affective processes have been an important research area for human factors and ergonomics. Although there is an obvious connection between affect and communication and collaboration, little research has been conducted in the human factors community until recently. In this panel, the panelists will discuss recent advances in affective research in communication and collaboration systems. Theoretical perspectives in human computer interaction, human agent interaction, and teamwork that take affective process into account will be discussed. Methodological issues will also be addressed, such as the measurements of affect, research design, and data analysis methods. Finally the applications of the theories and methods in different systems, such as human robot interaction, healthcare, and multi-tasking teams, will be discussed.

Characterizing Crowdsourced Data Collected Using DESIM (Descriptive to Executable Simulation Modeling)

Across many case studies, the Descriptive to Executable Simulation Modeling (DESIM) method has demonstrated the ability to capture and model qualitative knowledge from multiple subject-matter experts (SMEs), convert those models to an executable form using a crowdsourcing approach, and interactively visualize the outputs. This method helps decision makers leverage collective expertise to perform complex “What if?” analysis. This paper takes advantage of a large-scale multiple-model application of DESIM to illustrate the nature and interpretation of the data produced throughout its multiple phases. Lessons learned from this study provide direction toward future evaluation and improvements to this method.

Software Development Practices, Barriers in the Field and the Relationship to Software Quality

Context: Critical software systems developed for the government continue to be of lower quality than expected, despite extensive literature describing best practices in software engineering. Goal: We wanted to better understand the extent of certain issues in the field and the relationship to software quality. Method: We surveyed fifty software development professionals and asked about practices and barriers in the field and the resulting software quality. Results: There is evidence of certain problematic issues for developers and specific quality characteristics that seem to be affected. Conclusions: This motivates future work to address the most problematic barriers and issues impacting software quality.

Identifying Option Awareness Requirements for Public Health Crisis Decision Making

This paper presents the results of a series of task analyses with decision makers responsible for managing public health crises. The goal was to uncover cognitive requirements for achieving option awareness, in addition to situation awareness, in complex and emergent decision-making scenarios. Five expert decision makers in multiple roles related to public health were interviewed to elicit knowledge requirements relating to emerging public health crises. Analysis revealed patterns of responses showing that in addition to the known necessity of high situation awareness in handling complex events, additional forms of awareness about the relative desirability of options affect the decision making of experts. The results have implications for the design of systems supporting decision making under high complexity and uncertainty, as well as for training to accelerate the growth of novices into expert decision makers.

Collaborative Information Seeking Around Big Data

Big data analytics poses many socio-computational problems with respect to collaborative information seeking. While collaboration presents a way to alleviate the data deluge, research into this area is only relatively recent. The diverse range of skills and knowledge among a data analytics team presents an array of problems, including a wide spectrum of domain expertise, lack of shared understanding between roles, and challenges with the physical and computational aspects of multiple people seeking information within the multiple systems required for big data analytics. This chapter discusses recent research on collaborative big data analytics to discuss present progress, lessons learned, and gaps to be filled with future research. It proposes that the framework of the transactive memory system is a viable way to view people working around big data, as it supports collaborative sensemaking and the production of common ground among heterogeneous teams.