Matthew S. Prewett

A systematic review of the reliability of objective structured clinical examination scores

Medical Education 2011: 45: 1181–1189

Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis

In a meta-analysis of 43 studies, we examined the effects of multimodal feedback on user performance, comparing visual-auditory and visual-tactile feedback to visual feedback alone. Results indicate that adding an additional modality to visual feedback improves performance overall. Both visual-auditory feedback and visual-tactile feedback provided advantages in reducing reaction times and improving performance scores, but were not effective in reducing error rates. Effects are moderated by task type, workload, and number of tasks. Visual-auditory feedback is most effective when a single task is being performed (g = .87), and under normal workload conditions (g = .71). Visual-tactile feedback is more effective when multiple tasks are begin performed (g = .77) and workload conditions are high (g = .84). Both types of multimodal feedback are effective for target acquisition tasks; but vary in effectiveness for other task types. Implications for practice and research are discussed.

The Team Personality–Team Performance Relationship Revisited: The Impact of Criterion Choice, Pattern of Workflow, and Method of Aggregation

Using meta-analytic evidence, this study tested trait- and task-based theoretical approaches to team personality management, using both team behaviors and team outcomes as criteria. Trait theories state that maximization of the team trait is harmful for Extroversion (complementary team fit) but beneficial for Conscientiousness, Agreeableness, and Emotional Stability (supplementary fit). Task-based theories state that tasks with few work exchanges are best reflected by mean trait scores, whereas tasks with frequent work exchanges are best represented by other types of scores (e.g., minimum score). Correlations between different aggregations of team personality and team performance were coded, as well as the study criterion choice and the pattern of workflow (as moderators). Partial support for both trait and task theories were found. Team Conscientiousness and Agreeableness provided supplementary fit primarily with team behaviors, but there was mixed evidence that Extroversion provided complementary fit. Furthermore, minimum and variance measures of the team trait related to team performance in tasks with frequent work exchanges, but not in tasks with few work exchanges. Results suggest several limitations with existing measurement methods, which are discussed.

Managing workload in human-robot interaction: A review of empirical studies

Working with artificial agents is a challenging endeavor, often imposing high levels of workload on human operators who work within these socio-technical systems. We seek to understand these workload demands through examining the literature in major content areas of human-robot interaction. As research on HRI continues to explore a host of issues with operator workload, there is a need to synthesize the extant literature to determine its current state and to guide future research. Within HRI socio-technical systems, we reviewed the empirical literature on operator information processing and action execution. Using multiple resource theory (MRT; Wickens, 2002) as a guiding framework, we organized this review by the operator perceptual and responding demands which are routinely manipulated in HRI studies. We also reviewed the utility of different interventions for reducing the strain on the perceptual system (e.g., multimodal displays) and responses (e.g., automation). Our synthesis of the literature demonstrates that much is known about how to decrease operator workload, but there are specific gaps in knowledge due to study operations and methodology. This work furthers our understanding of workload in complex environments such as those found when working with robots. Principles and propositions are provided for those interested in decreasing operator workload in applied settings and also for future research.

A Meta-Analysis of Vibrotactile and Visual Information Displays for Improving Task Performance

Many studies have investigated the effect of vibrotactile cues on task performance, but a wide range of cue and task types have made findings difficult to interpret without a quantitative synthesis. This report addresses that need by reviewing the effectiveness of vibrotactile cues in a meta-analysis of 45 studies. When added to a baseline task or to existing visual cues, vibrotactile cues enhanced task performance. When vibrotactile cues replaced visual cues; however, some effects were attenuated and others moderated by cue information complexity. To summarize such moderating effects, vibrotactile alerts are an effective replacement for visual alerts, but vibrotactile direction cues are not effective when replacing visual direction cues. This meta-analysis of vibrotactile applications underscores the benefits of vibrotactile and multimodal displays, highlights conditions in which vibrotactile cues are particularly effective, and identifies areas in need of further investigation.

The benefits of multimodal information: a meta-analysis comparing visual and visual-tactile feedback

Information display systems have become increasingly complex and more difficult for human cognition to process effectively. Based upon Wickens Multiple Resource Theory (MRT), information delivered using multiple modalities (i.e., visual and tactile) could be more effective than communicating the same information through a single modality. The purpose of this meta-analysis is to compare user effectiveness when using visual-tactile task feedback (a multimodality) to using only visual task feedback (a single modality). Results indicate that using visual-tactile feedback enhances task effectiveness more so than visual feedback (g = .38). When assessing different criteria, visual-tactile feedback is particularly effective at reducing reaction time (g = .631) and increasing performance (g = .618). Follow up moderator analyses indicate that visual-tactile feedback is more effective when workload is high (g = .844) and multiple tasks are being performed (g = .767). Implications of results are discussed in the paper.

Normative Scoring of Multidimensional Pairwise Preference Personality Scales Using IRT: Empirical Comparisons With Other Formats

In this article, we offer some suggestions as to why tetrads and pentads have become the dominant formats for administering multidimensional forced choice (MFC) items but, in turn, raise questions regarding the underlying psychometric model and means of addressing item quality and scoring accuracy. We then focus our attention on multidimensional pairwise preference (MDPP) items and present an item response theory–based approach to constructing and modeling MDPP responses directly, assessing information at the item and scale levels, and a way of computing standard errors for trait scores and estimating scale reliability. To demonstrate the viability of this method for applied use, we show that the correspondence between MDPP scores derived from direct modeling with those obtained using single statement and unidimensional pairwise preference measures administered in a laboratory setting. Trait score correlations and criterion related validities are compared across testing formats and rating sources (i.e., self and other), and the usefulness of our model-based approach is further demonstrated by some illustrative results involving computerized adaptive tests (CAT).

Group Parent-Child Interaction Therapy: A Randomized Control Trial for the Treatment of Conduct Problems in Young Children

OBJECTIVE Although efficacious interventions exist for childhood conduct problems, a majority of families in need of services do not receive them. To address problems of treatment access and adherence, innovative adaptations of current interventions are needed. This randomized control trial investigated the relative efficacy of a novel format of parent-child interaction therapy (PCIT), a treatment for young children with conduct problems. METHOD Eighty-one families with 3- to 6-year-old children (71.6% boys, 85.2% White) with diagnoses of oppositional defiant or conduct disorder were randomized to individual PCIT (n = 42) or the novel format, Group PCIT. Parents completed standardized measures of childrens conduct problems, parenting stress, and social support at intake, posttreatment, and 6-month follow-up. Therapist ratings, parent attendance, and homework completion provided measures of treatment adherence. Throughout treatment, parenting skills were assessed using the Dyadic Parent-Child Interaction Coding System. RESULTS Parents in both group and individual PCIT reported significant improvements from intake to posttreatment and follow-up in their childrens conduct problems and adaptive functioning, as well as significant decreases in parenting stress. Parents in both treatment conditions also showed significant improvements in their parenting skills. There were no interactions between time and treatment format. Contrary to expectation, parents in Group PCIT did not experience greater social support or treatment adherence. CONCLUSIONS Group PCIT was not inferior to individual PCIT and may be a valuable format to reach more families in need of services. Future work should explore the efficiency and sustainability of Group PCIT in community settings. (PsycINFO Database Record

Teamwork in the trauma room evaluation of a multimodal team training program.

Introduction: Poor teamwork leads to preventable medical errors, and thus negatively impacts medical care. One way to improve teamwork is training. A multimodality team training program was designed to impact the attitudes and behavior of first-year residents who will encounter medical situations in the trauma room. The training program included low-fidelity role plays, lectures, and high-fidelity simulation with feedback. Materials and Methods: The training program was a one-day workshop that was conducted twice, once for each of the two groups over two days at the beginning of the academic year in July. A total of 41 first-year interns (10 Emergency Medicine and 31 Surgery) were recruited for participation. Participants completed a Situational judgment test (SJT) on trauma teamwork before training. The training began with a low-fidelity simulation that served as an icebreaker to team concepts. Subsequently, a lecture with discussion provided key points regarding teamwork in the trauma room. A high-fidelity simulation then allowed participation in one of four trauma room scenarios with medical expert debriefing. The course concluded with a course summary and an assessment of participant attitudes regarding training along with a second administration of SJT. Results: Participant reactions to the training were positive overall. Results of SJT showed a positive effect for team training in three of the four possible comparisons. Conclusion: The program was well received by the residents. Results suggest that a comprehensive training approach using role play, lecture, and simulation can positively affect behavioral choices for teamwork in the trauma room.

Autonomy and Automation Reliability in Human-Robot Interaction: A Qualitative Review

The effectiveness and reliability of automation aids are critical topics in the area of human-robot interaction (HRI). As more tasks are subsumed by robots and autonomous systems, it is important to examine the relationships between these entities and their human operators. Research to date has covered various manipulations of autonomy, but this broad body of research is in need of focus and consistency. The current study presents a qualitative overview of research regarding levels and reliability of autonomy/control and the effects they have on important HRI-relevant outcome variables. Results indicate that autonomy and automation aids operate uniquely for different tasks, and that there are many complex factors that can affect not only performance but also usability, confidence, and safety. Unresolved issues in the field and challenges and opportunities for future research are also presented.