Kevin J. Williams

Assessment of the Intrarater and Interrater Reliability of an Established Clinical Task Analysis Methodology

Background Task analysis may be useful for assessing how anesthesiologists alter their behavior in response to different clinical situations. In this study, the authors examined the intraobserver and interobserver reliability of an established task analysis methodology. Methods During 20 routine anesthetic procedures, a trained observer sat in the operating room and categorized in real-time the anesthetists activities into 38 task categories. Two weeks later, the same observer performed task analysis from videotapes obtained intraoperatively. A different observer performed task analysis from the videotapes on two separate occasions. Data were analyzed for percent of time spent on each task category, average task duration, and number of task occurrences. Rater reliability and agreement were assessed using intraclass correlation coefficients. Results Intrarater reliability was generally good for categorization of percent time on task and task occurrence (mean intraclass correlation coefficients of 0.84–0.97). There was a comparably high concordance between real-time and video analyses. Interrater reliability was generally good for percent time and task occurrence measurements. However, the interrater reliability of the task duration metric was unsatisfactory, primarily because of the technique used to capture multitasking. Conclusions A task analysis technique used in anesthesia research for several decades showed good intrarater reliability. Off-line analysis of videotapes is a viable alternative to real-time data collection. Acceptable interrater reliability requires the use of strict task definitions, sophisticated software, and rigorous observer training. New techniques must be developed to more accurately capture multitasking. Substantial effort is required to conduct task analyses that will have sufficient reliability for purposes of research or clinical evaluation.

Quantitative description of the workload associated with airway management procedures

STUDY OBJECTIVESnTo measure the workload associated with specific airway management tasks.nnnSETTING AND INTERVENTIONnWritten survey instrument.nnnPATIENTSn166 Stanford University and 75 University of California, San Diego, anesthesia providers.nnnMEASUREMENTS AND MAIN RESULTSnSubjects were asked to use a seven-point Likert-type scale to rate the level of perceived workload associated with different airway management tasks with respect to the physical effort, mental effort, and psychological stress they require to perform in the typical clinical setting. The 126 subjects completing questionnaires (overall 52% response rate) consisted of 43% faculty, 26% residents, 23% community practitioners, and 8% certified registered nurse-anesthetists (CRNAs). Faculty physicians generally scored lower workload measures than residents, whereas community practitioners had the highest workload scores. Overall, workload ratings were lowest for laryngeal mask airway (LMA) insertion and highest for awake fiberoptic intubation. Airway procedures performed on sleeping patients received lower workload ratings than comparable procedures performed on awake patients. Direct visualization procedures received lower workload ratings than fiberoptically guided procedures.nnnCONCLUSIONSnThese kinds of data may permit more objective consideration of the nonmonetary costs of technical anesthesia procedures. The potential clinical benefits of the use of more complex airway management techniques may be partially offset by the impact of increased workload on other clinical demands.

Developing a Technique to Measure Anesthesiologists' Real-Time Workload

Workload is a construct used to describe the extent to which an operator has engaged the cognitive and physical resources required for task performance. As task difficulty increases, operators allocate more resources to maintain acceptable performance. The purpose of the present study is to develop a technique to measure workload continuously, with high-resolution in real-time, and in a way that accounts for the contribution of multiple individual task components of the job of administering anesthesia. Workload associated with 51 clinical tasks performed by anesthesiologists was assessed using a written survey instrument, developed for this study. Participants were 241 anesthesia providers. Interval scale values were computed for each of the tasks using direct estimatation procedures. These values were then used to produce workload density maps for actual anesthesia cases by weighting the tasks performed during the case by their workload scale values. Moment-by-moment data on the specific tasks performed by anesthesiologists were obtained in real-time by trained observers during actual anesthetic cases. This type of analysis can be used to evaluate the costs associated with technical anesthesia procedures, increase our understanding of the anesthesiologists job, and guide rational optimization of procedures, equipment, scheduling, and training.

The effects of simultaneous learning and performance goals on performance: An inductive exploration

This study was the first to examine the simultaneous use of multiple learning and performance goals on single task performance. We report the results of two studies (one study used self-set goals; the other study used assigned goals) to examine the effects of the concurrent pursuit of proximal and distal learning and performance goals. The first study was correlational in nature and the second used an experimental manipulation. In both we found a negative curvilinear relationship between total goal difficulty and performance. We discovered that an intermediate level of total goal difficulty was optimal. The results were also tied to self-efficacy and task strategies where additional curvilinear relationships were found. These results expand goal-setting theory and suggest new research directions. Our study has implications for managers who want to find the best combination of goals for their employees.

On the Predictive Efficiency of Past Performance and Physical Ability: The Case of the National Football League

This study investigated the criterion-related validity of past performance and physical ability tests over time in a physically demanding context, the National Football League (NFL). Results suggested that an indicator of past performance, collegiate performance, engendered a stronger relationship with future NFL performance than a variety of physical ability tests administered during the NFL Combine. Unlike physical ability, past performance remained a valid predictor across four years of the criterion domain; however, the validity coefficients eventually deteriorated over time, following a simplex pattern commonly found with general mental ability tests. Implications germane to staffing research and practice are discussed.

Allocation of Responsibility for Injuries from a “Hidden” Hazard

Safety researchers have investigated how people assign blame for injuries sustained during the use of or exposure to consumer products. In this study, we examine attributions made by people given product-use scenarios that describe a girl whose age is manipulated to be from 18 months to 16 years and who suffers serious brain damage after choking on marshmallows made available to her by her mother. Supplementary information intended to be either positive or detrimental to the manufacturer and its safety practices was either present or absent from the scenario. Approximately half of the participants in the positive frame condition also received sample product warnings purportedly developed and used by the manufacturer. In general, participants allocated more responsibility to the girls parents than to the manufacturer. As predicted, allocation of blame to the girl varied directly with her age. The greater the age of the child, the greater the blame she received. Supplementary information that casts the manufacturer and the manufacturers practices in a positive light shifted blame away from the manufacturer toward the girl and her parents. Conversely, supplementary information detrimental to the manufacturer shifted blame away from the girl and her parents and toward the manufacturer. The warnings had no measurable effect on allocation of blame. The implications of these results for consumers, legal professionals, and researchers are discussed.

Allocating Blame for Airbag Deployment Injuries: Separating Manufacturers' Blame from Personal Responsibility

This study examined how people allocate blame for injuries sustained from the deployment of a driver-side airbag. Participants read one of several versions of a fictitious scenario in which the driver of an automobile is injured by a deploying airbag after a driver swerves into oncoming traffic to avoid striking a child who has run into the road. The scenarios depicted a driver sitting within the airbags deployment zone and varied in the following ways: the stature of the injured driver (small or large); severity of the injury resulting from the deployment of the airbag (permanent blindness in one eye versus quadriplegia); vehicle speed at impact (15 m.p.h. above versus driving at the posted speed limit); and the safety-worthiness of the vehicle (an elaborate system of safety features versus the absence of these features). When assigning blame for the injuries sustained in the crash, participants appeared sensitive to both the quality of the vehicles safety system and the driving behavior of the injured party. The manufacturer of the “safe” vehicle was held significantly less responsible than the manufacturer of the vehicle lacking these safety features. However, driver behavior also exerted a significant effect on allocation of blame. Injured drivers depicted as traveling significantly above the speed limit were assigned significantly more blame than their counterparts depicted as driving at the speed limit. This finding suggests that people take other factors into account, including personal responsibility, when assigning blame. Perhaps the most important finding of this research, and one that supports previous research on this topic, is that safety pays. When companies are perceived as making a good faith attempt to look out for the safety of their customers, their customers, in return, may be less likely to hold them responsible when injuries do occur.

Separating the Effects of Warning and Information Distribution Practices: A Case of Cascading Responsibility

In this study we examine how participants allocate blame after reading one of several variants of a fictitious product-use scenario that describes a construction worker who is injured after falling through an acrylic panel used in the construction of greenhouses. The safety practices and policies exhibited by the manufacturer, distributor, construction company owner and worker were cast either in a positive or negative light. Among the variables studied was the quality of the product warning and the methods used to distribute safety materials (e.g., installation manual) to the end user (panel installers). The results showed that blame allocation fell “downstream”, where most blame fell on the party that failed to distribute the safety information to the next step of the chain. The effects of distribution breakdown on blame was moderated by the type of warning provided by the manufacturer in that greater blame was placed on the negligent party when a good warning is used, and part of the blame for the accident was shifted to the manufacturer when it provided a poor warning, regardless of where the breakdown occurred. The implications of these results for consumers, legal professionals, and researchers are discussed.

Emphasizing Non-Obvious Hazards Using Multi-Frame Pictorials and Color on Allocation of Blame

This study examined allocations of blame for injuries sustained from the consumption of a product with a non-obvious hazard. Participants were given product-use scenarios that described a girl whose age was manipulated to be from 18 months to 16 years and who suffered serious brain damage after choking on a marshmallow made available to her by her mother. Supplementary information intended to be either positive or detrimental to the manufacturer and its safety practices was either present or absent from the scenario. When present and positive, the manufacturer put a warning on the product about the non-obvious hazard. The warning was manipulated by having color or not and a multi-frame or single-frame pictorial. Results replicated findings reported initially by Kalsher et al. (1999). When supplementary information was positive or not provided, participants directed more blame toward the parents of the young victim and less to the manufacturer. The opposite pattern was shown when negative supplementary information about the manufacturer and its safety practices was provided, suggesting that people perceived the manufacturer as irresponsible in their practices.

Responsibility allocation for child injury: victim age and positive vs. negative framing of manufacturer's safety policy

Two experiments examined allocation of responsibility in the context of a fictitious, but realistic, product-use scenario in which a young girl suffers serious brain injury after consuming a product with a non-obvious hazard (marshmallows). The research investigated whether the responsibility allocated to the various parties would depend on the age of the child and whether the manufacturer took, or failed to take, precautions. Scenarios given to participants stated the age of the girl as 1½ years, 4 years, 8 years, or 16 years and had positive, negative, or no supplemental information about the manufacturer and its safety practices. Both experiments showed that the parents were considered most responsible for a young childs injury, but the allocation decreased with the older child. When negative information about the manufacturers safety practices was given, allocations of responsibility for the girls injury to the manufacturer increased significantly. In Experiment 2, the presence of warnings in the positive supplemental information condition reduced the manufacturers responsibility for the oldest (16-year old) child. Negative impressions due to poor safety practices by manufacturers can lead to increased levels of responsibility allocated for injury. Primary caretakers are responsible for the safety of young children, but as they get older, children are viewed as being more responsible for their own safety. These results have implications for product-development decisions including labelling. They also point out a role for human factors professionals before and during product-related forensic litigation.