Jeffrey Turnbull

Evaluating communication skills in the objective structured clinical examination format: reliability and generalizability

In most objective structured clinical examinations (OSCEs), communication skills are assessed as an ‘add‐on’ to history‐taking stations, rather than in stations designed to assess communication skills in the broadest sense. This study investigated the feasibility of developing such stations. In part one, 60 clinical clerks and 36 residents were rated in four 10‐min emotionally charged situations portrayed by standardized patients. Inter‐rater reliability was demonstrated (r = 0.59–0.63) and a highly significant effect of educational level was found. Generalizability between communication stations was low (0.17–0.20).

Educating future physicians for Ontario

In 1987, Ontarios physicians conducted a strike, ultimately not successful, over the issue of “extra billing.” The fact that the Ontario public did not support this action reflected a major gap between the professions view of itself and the publics view of the profession. In 1990, the provinces five medical schools launched a collaborative project to determine more specifically what the people of Ontario expect of their physicians, and how the programs that prepare future physicians should be changed in response. The authors report on the first five years of that ongoing project. Consumer groups were asked to state their views concerning the current roles of physicians, future trends that would affect these roles, changes in roles they wished to see, and suggestions for changes in medical education. Methods used included focus groups, key informant interviews, an extensive literature review, and surveys, including a survey of health professionals. Concurrently, inter-university working groups prepared tools and strategies for strengthening faculty development, assessing student performance, and preparing future leadership for Ontarios medical education system. Eight specific physician roles were identified: medical expert, communicator, collaborator, health advocate, learner, manager (“gatekeeper”), scholar, and “physician as person.” Educational strategies to help medical students learn to assume these eight roles were then incorporated into the curricula of the five participating medical schools. The authors conclude that the project shows that it is feasible to learn specifically what society expects of its physicians, to integrate this knowledge into the process of medical education reform, and to implement major curriculum changes through a collaborative, multi-institutional consortium within a single geopolitical jurisdiction.

Performance-based assessment of clinical ethics using an objective structured clinical examination.

PURPOSE: To further examine the objective structured clinical examination (OSCE) as a performance-based assessment method for clinical ethics. METHOD: In the spring of 1993, a volunteer sample of 88 final-year medical students from all five Ontario medical schools took a four-station OSCE that used standardized patients and involved decisions to forego life-sustaining treatment. Performance was scored on a checklist of behaviors unique to each case. Data were analyzed for reliability using intraclass correlation coefficients and the Spearman-Brown prophecy formula. RESULTS: Reliability of the test was only .28 as a result of a low average inter-station correlation of .07. To achieve a test reliability of .8, 41 stations (almost seven hours of testing time) would be required. CONCLUSION: Because of its low test reliability, the OSCE is not a feasible stand-alone method for summative evaluation of clinical ethics. This performance-based evaluation method should be combined with other, more reliable evaluation methods. The OSCE has promise for formative evaluation.

Clinical work sampling A new approach to the problem of in-training evaluation.

OBJECTIVE: Existing systems of in-training evaluation (ITE) have been criticized as being unreliable and invalid methods for assessing student performance during clinical education. The purpose of this study was to assess the feasibility, reliability, and validity of a clinical work sampling (CWS) approach to ITE. This approach focused on the following: (1) basing performance data on observed behaviors, (2) using multiple observers and occasions, (3) recording data at the time of performance, and (4) allowing for a feasible system to receive feedback.PARTICIPANTS: Sixty-two third-year University of Ottawa students were assessed during their 8-week internal medicine inpatient experience.MEASUREMENTS AND MAIN RESULTS: Four performance rating forms (Admission Rating Form, Ward Rating Form, Multidisciplinary Team Rating Form, and Patient’s Rating Form) were introduced to document student performance. Voluntary participation rates were variable (12%–64%) with patients excluded from the analysis because of low response rate (12%). The mean number of evaluations per student per rotation (19) exceeded the number of evaluations needed to achieve sufficient reliability. Reliability coefficients were high for the Ward Form (.86) and the Admission Form (.73) but not for the Multidisciplinary Team (.22) Form. There was an examiner effect (rater leniency), but this was small relative to real differences between students. Correlations between the Ward Form and the Admission Form were high (.47), while those with the Multidisciplinary Team Form were lower (.37 and .26, respectively). The CWS approach ITE was considered to be content valid by expert judges.CONCLUSIONS: The collection of ongoing performance data was reasonably feasible, reliable, and valid.

Homelessness and health

Homelessness is evolving into a national and international crisis. Within developed countries, homeless rates are now believed to approximate 1% of most urban populations.[1][1] In the 2001 Canadian census, 14 145 people were counted as living in a shelter. Of note, 1490 (0r 10%) people were under

Content specificity and oral certification examinations

This study reports on the generalizability of different skills assessed in the oral certification examinations in Internal Medicine of the Royal College of Physicians and Surgeons of Canada. Assessments from the 1992 examination were examined prospectively to determine (i) inter‐rater reliability, (ii) correlation from morning to afternoon sessions, and (iii) overall test reliability. While inter‐rater reliability was acceptable and in the range reported from previous studies, the generalizability across sessions was very low, ranging from 0.30 to 0.47, presumably reflecting content specificity. As a consequence, the overall test reliability was low, ranging from 0.57 to 0.69. Collapsing the overall scores into three decision categories (pass, borderline, fail) lowered the test reliability still further. Strategies to resolve this problem are suggested.

evaluation of a multicenter ethics objective structured clinical examination

The purpose of this study was to evaluate a six-station ethics objective structured clinical examination (OSCE) on a volunteer sample of 66 medical students and 33 residents from three Ontario medical schools. The internal consistency reliability was 0.46 and the median interrater reliability was 0.675 (range 0.30 to 0.89). The residents’ scores were higher than those of the medical students (F=2.24, 0.046). Also, the scores differed among the three schools (F=3.19, p=0.0004). The ethics OSCE has adequate interrater reliability and construct validity, but low internal consistency reliability. There are differences among the schools that may assist in ethics curriculum evaluation and development

Learning to account for the social determinants of health affecting homeless persons.

Context  Intersecting social determinants of health constrain access to care and treatment adherence among homeless populations. Because clinicians seldom receive training in the social determinants of health, they may be unprepared to account for or address these factors when developing treatment strategies for homeless individuals.

100 years and counting

As the first issue of CMAJ was being mailed in January 1911, editors at The Globe in Toronto — it would not be known as The Globe and Mail until 1936 — had just published their first editorial of the year. “To be a Canadian in 1911,” they wrote, “is to be given a chance which is matched

Prevalence of methicillin-resistant Staphylococcus aureus in a Canadian inner-city shelter

BACKGROUND Methicillin-resistant Staphylococcus aureus (MRSA) colonization is increasingly of concern in community settings. However, despite a recent outbreak in Calgary, Alberta, data on the prevalence of MRSA in Canadian communities are lacking. Globally, few studies have been performed in high-risk groups such as inner-city populations. METHODS A cross-sectional study of the prevalence of MRSA among residents and staff at three Ottawa, Ontario, shelters was conducted. All participants completed a questionnaire, and provided nasal swabs as well as one of rectal, anal or groin swabs. RESULTS Among 84 participants, the prevalence of MRSA colonization was 2.4%. Among the resident subgroup, the prevalence was 4.5%, while no MRSA isolates were found among 40 staff participants. All isolates were USA100 (CMRSA-2) subtypes. CONCLUSIONS The prevalence of MRSA colonization among residents is higher than baseline population rates, but is consistent with other inner-city populations. Although community outbreaks of USA300 and USA400 strains are increasingly reported, movement of nosocomial strains (ie, USA100 [CMRSA-2]) into communities remains an important avenue in the spread of MRSA and underscores the importance of nosocomial MRSA control.