George Pachev

Comparing the use of global rating scale with checklists for the assessment of central venous catheterization skills using simulation

The use of checklists is recommended for the assessment of competency in central venous catheterization (CVC) insertion. To explore the use of a global rating scale in the assessment of CVC skills, this study seeks to compare its use with two checklists, within the context of a formative examination using simulation. Video-recorded performances of CVC insertion by 34 first-year medical residents were reviewed by two independent, trained evaluators. Each evaluator used three assessment tools: a ten-item checklist, a 21-item checklist, and a nine-item global rating scale. Exploratory principal component analysis of the global rating scale revealed two factors, accounting for 84.1% of the variance: technical ability and safety. The two checklist scores correlated positively with the weighted factor score on technical ability (0.49 [95% CI 0.17–0.71] for the 10-item checklist; 0.43 [95% CI 0.10–0.67] for the 21-item checklist) and negatively with the weighted factor score on safety (−0.17 [95% CI −0.48–0.18] for the 10-item checklist; −0.13 [95% CI −0.45–0.22] for the 21-item checklist). A checklist score of <80% was strong indication of incompetence. However, a high checklist score did not preclude incompetence. Ratings using the global rating scale identified an additional 11 candidates (32%) who were deemed incompetent despite scoring >80% on both checklists. All these candidates committed serious errors. In conclusion, the practice of universal adoption of checklists as the preferred method of assessment of procedural skills should be questioned. The inclusion of global rating scales should be considered.

Randomized trial of problem-based versus didactic seminars for disseminating evidence-based guidelines on asthma management to primary care physicians.

Introduction: This randomized controlled trial (RCT) investigated the effectiveness of and satisfaction with small‐group problem‐based learning (PBL) versus a didactic lecture approach to guideline dissemination in asthma management controlling for confounders common in comparative educational interventions. Methods: Sites were selected as either lecture or PBL using simple randomization. All participants were exposed to similar educational resources to ensure treatment equivalency. Instruments included standardized program/speaker evaluation forms and a validated case‐based questionnaire with a visual analogue scale measuring the level of confidence of responses. The latter was presented immediately pre‐ and post‐intervention and 3 months later. The statistician was blinded to intervention groups. Results: Overall, 52 family physicians agreed to participate, 23 in the PBL sessions (mean 4.6 per group) and 29 in the didactic lecture sessions (mean 7.25). There was no significant difference between the groups with respect to the knowledge gained at each test administration. Participants rated the lecturer or facilitator equally well as having established a positive learning environment. PBL participants rated the perceived educational value of the program higher than did lecture participants (4.36 vs. 3.93; p =.04). Both groups experienced a significant increase in asthma‐related knowledge post‐intervention. Attrition rates for the 3‐month post‐test were 14% for PBL participants versus 32% for lecture‐based participants. Discussion: PBL was as effective in knowledge uptake and retention as lecture‐based continuing medical education (CME) programs. Further study is warranted to investigate whether the assessment of higher educational value or an increase in response rate to delayed testing is replicable in other RCTs addressing common confounders and if these factors influence future CME participation, changes in physician clinical behavior, or patient health outcomes.

Workplace injury management: using new technology to deliver and evaluate physician continuing medical education

Objective: Physicians typically receive little continuing medical education (CME) about their role in workplace injury management as well as on workplace injuries and disease. Although new technologies may help educate physicians in these areas, careful evaluation is required, given the understudied nature of these interventions. The objective of this study is to evaluate two promising new technologies to deliver CME (online learning and videoconferencing) and to compare the effectiveness of these delivery methods to traditional CME interventions (large urban traditional conference lectures and small group local face-to-face outreach) in their impact on physician knowledge related to workplace injury management. Methods: This study utilized a prospective, controlled evaluation of two educational programs for BC physicians: 1) The Diagnosis and Management of Lateral Epicondylitis; and 2) Is Return-to-Work Good Medicine? Each educational module was delivered in each of four ways (Outreach Visit, Videoconference Session, Conference Lecture, Online) and physicians self-selected their participation—both in terms of topic and delivery method. Questionnaires related to knowledge as well as learner attitude and satisfaction were administered prior (pre-test) and following (post-test) all educational sessions. Results: 581 physician encounters occurred as a result of the educational interventions and a significant percentage of the physicians participated in the research per se (i.e. there were 358 completed sets of pre-test and post-test ‘Knowledge’ questionnaires). Overall the results showed that the developed training programs increased physicians’ knowledge of both Lateral Epicondylitis and the physician’s role in Return-To-Work planning as reflected in improved post-test performance when compared to pre-test scores. Furthermore, videoconferencing and online training were at least as effective as conference lectures and instructor-led small group outreach sessions in their impact on physician knowledge. Conclusions: Use of effective videoconferencing and online learning activities will increase physician access to quality CME related to workplace injury management and will overcome access barriers intrinsic to types of CME interventions based on instructor-student face-to-face interactions.

Reliability and acceptance of the mini-clinical evaluation exercise as a performance assessment of practicing physicians.

Background The purpose of this study was to evaluate the reliability and acceptance of the mini-Clinical Evaluation Exercise (mini-CEX) as an assessment of practicing primary care physicians. Method Six raters were recruited to conduct the assessments. After a training session, their ability to discriminate between levels of performance was evaluated using videotaped clinical scenarios. Fifteen physicians were assessed in an office setting by the raters who scored multiple clinical encounters using a validated mini-CEX form for each encounter. Participants were given a postassessment survey regarding the process. Results Raters distinguished between performance levels on the videotaped scenarios (P < .001). A total of 188 physician–patient interactions were assessed. The generalizability coefficient for 10 encounters was 0.92. In the postassessment survey, the raters (94%) and physicians assessed (75%) both felt that the mini-CEX is an acceptable assessment. Conclusions The mini-CEX seems to be a reliable and acceptable instrument for the assessment of practicing physicians.

Objective Assessment of Laparoscopic Skills Dual-Task Approach

Background. Assessment of surgical performance is often accomplished with traditional methods that often provide only subjective data. Trainees who perform well on a simulator in a controlled environment may not perform well in a real operating room environment with distractions. This project uses the ideas of dual-task methodology and applies them to the assessment of performance of laparoscopic surgical skills. The level of performance on distracting secondary tasks while trying to perform a primary task becomes an indirect but objective measure of the surgical skill of the trainee. Methods. Nine surgery residents and 6 experienced laparoscopic surgeons performed 3 primary tasks on a laparoscopic virtual reality simulator (camera position, grasping, and cholecystectomy) while being distracted by 3 secondary tasks (counting beeps, selective responses, and mental arithmetic). Completion time and error rates were recorded for each combination of tasks. Results. When performed separately, time to completion and error rates for primary and secondary tasks were similar for learners and experts. When performing the tasks simultaneously, learners had more errors than experts. Error rates increased for learners when distracting tasks became more difficult or required more attention. Expert surgeons maintained consistent error rates despite the increasing difficulty of task combinations. Conclusions. The use of dual-task methodology may help trainers to identify which surgical trainees require more preparation before entering the real operating room environment. Expert surgeons are capable of maintaining performance levels on a primary task in the face of distractions that may occur in the operating room.