Preston A. Kiekel

Measuring Team Knowledge: A Window to the Cognitive Underpinnings of Team Performance

The authors report an effort aimed at developing and evaluating measures of taskwork and teamwork team knowledge for teams in which members differ in knowledge backgrounds. These measures were used in a study with 36 teams to explore the cognitive underpinnings of team performance variations due to cross-training regime. The authors demonstrate that these measures are valid and provide team performance information that complements outcome and behavioral measures. Teams exposed to full cross-training acquired more taskwork and teamwork knowledge than control teams or teams exposed to a conceptual version of cross-training. Measures of team knowledge provide information regarding team task performance critical for system design and training programs.

Measuring Team Knowledge During Skill Acquisition of a Complex Task

The design of team training programs and other team interventions could benefit from an understanding of team cognition. The research presented in this article evaluates methods for eliciting and assessing team knowledge during acquisition of a complex task. Knowledge measures are evaluated in terms of their ability to predict team performance and also in terms of how they reflect skill acquisition. The study was conducted in the context of a synthetic 3-person team task that involved operating an uninhabited air vehicle (UAV). Eleven teams of 3 Air Force ROTC cadets participated in 3 experimental sessions lasting from 3 to 6 hr. During these sessions, teams were trained on the task and were observed as they performed ten 40-min missions. During the missions, team performance and team process behaviors were measured, as well as the fleeting team knowledge associated with situation awareness. In addition, long-term team knowledge regarding both taskwork and teamwork were measured offline in four sessions. ...

Some Promising Results of Communication-Based Automatic Measures of Team Cognition

Some have argued that the most appropriate measure of team cognition is a holistic measure directed at the entire team. In particular, communication data are useful for measuring team cognition because of the holistic nature of the data, and because of the connection between communication and declarative cognition. In order to circumvent the logistic difficulties of communication data, the present paper proposes several relatively automatic methods of analysis. Four data types are identified, with low-level physical data vs. content data being one dimension, and sequential vs. static data being the other. Methods addressing all four of these data types are proposed, with the exception of static physical data. Latent Semantic Analysis is an automatic method used to assess content, either statically or sequentially. PRONET is useful to address either physical or content-based sequential data, and we propose CHUMS to address sequential physical data. The usefulness of each method to predict team performance data is assessed.

Measuring Speech Flow of Co-Located and Distributed Command and Control Teams During a Communication Channel Glitch

Team cognition can be observed in the flow of communications among team members. This is shown in the context of a simulated unmanned aerial vehicle ground control station. Automatic measures of low-level team communication flow were used to assess high-level constructs of team cognition. Measures show support for the expected results of manipulations in this task. Co-location and channel degradation effects were successfully predicted by CHUMS, ProNet, and a cross-correlation function-based Dominance measure. Results grant concurrent validity to the measures, and highlight substantive effects of the manipulations. In particular, in geographically distributed teams, communication patterns are less stable, and the route planner exerts less communicative influence. Some co-location effects drop with task experience. During a mission containing a five-minute one-way communication channel cut, all teams communicate more like distributed teams, and team members do create alternate pathways to retain information flow.

Evaluation of Latent Semantic Analysis-Based Measures of Team Communications Content

Team process is thought to mediate team member inputs and team performance. Among the team behaviors identified as process variables, team communications have been widely studied. We view team communications as a team behavior and also as team information processing, or team cognition. Within the context of a Predator Uninhabited Air Vehicle (UAV) synthetic task, we have developed several methods of communications content assessment based on Latent Semantic Analysis (LSA). These methods include: Communications Density (CD) which is the average task relevance of a teams communications, Lag Coherence (LC) which measures task-relevant topic shifting over UAV missions, and Automatic Tagging (AT) which categorizes team communications. Each method is described in detail. CD and LC are related to UAV team performance. AT-human is comparable to human-human agreement on content coding. The results are promising for the assessment of teams based on LSA applied to communication content.

Dynamical Perspectives on Team Cognition

A theoretical perspective based on dynamical systems theory is applied to the concept of team cognition. Important postulates based on this perspective include a division of labor based on the elements of information involved in performing a team task, as well as the notion of coordination among information elements occurring through elemental couplings via a coupling medium. These features of our perspective are outlined, with the important notion of mediational means receiving the most attention. The dynamics of discourse-based mediation are considered in light of empirical data involving changes in the content of discourse over time in an uninhabited air vehicle reconnaissance task.

Comparing and validating measures of team knowledge

Current measures of team knowledge are overly simplistic and fail to address some of the more interesting aspects of the construct. We present new methods and measures that address some of these shortcomings and apply them to eleven three-person teams who participated in ten missions of a synthetic team task involving uninhabited air vehicle operations. In addition to team knowledge, team performance and team process behaviors are also measured. Knowledge measures are compared and evaluated in terms of their ability to predict team performance and reflect skill acquisition. Measures of teamwork knowledge and the team situation model succeeded in reflecting knowledge changes with task acquisition. A taskwork relatedness ratings measure, taken at both the individual and team levels, along with the team situation model measure, were predictive of team performance. Specifically, high performing teams had more accurate situation models and more knowledge of the task from the perspective of other team members, as opposed to lower performing teams. Measures of knowledge at the team level provide information on the team cognition underlying team behavior and performance and have implications for the design of training programs and sociotechnical systems.

Addressing Limitations of the Measurement of Team Cognition

Team cognition is more than the sum of the cognition of the individual team members. Instead, it emerges from the interplay of individual cognition and team process behaviors. Team cognition has been implicated as a major factor underlying team performance and thus, its measurement is critical for team training and design. Measures of team cognition, however, are limited in a number of ways. For instance, measures are taken at an individual level and aggregated, rather than pursuing data collection at the more holistic level of the team. Further, measures do not capture the heterogeneous knowledge backgrounds of team members. We have begun to address these and other limitations by developing new measures and applying them in four studies of team performance in military synthetic task environments. We highlight the results of these studies, which support the validity of our measures of taskwork knowledge, teamwork knowledge, and team situation awareness.

Improving Teams' Interpositional Knowledge Through Cross Training

Recent investigations of team training have demonstrated advantages of cross training team members in the positions of other team members. Such benefits have been attributed to increases in interpositional knowledge. In an attempt to reduce the time demands of cross training, a conceptual cross-training condition that targeted teamwork knowledge was compared to traditional full cross-training and two control conditions. Three-person teams were assigned to a training condition and participated in two synthetic helicopter missions. Outcomes, team process behaviors, team situation awareness, taskwork knowledge, and teamwork knowledge were measured. Results indicated weak support for the benefits of full cross-training on team performance, yet minimal support for conceptual cross-training. Further, teams cross-trained in the traditional manner acquired more teamwork and taskwork interpositional knowledge than teams in any other condition. Both types of interpositional knowledge were correlated with team performance.