André Jacques

Poorly performing physicians: does the Script Concordance Test detect bad clinical reasoning?

Introduction Evaluation of poorly performing physicians is a worldwide concern for licensing bodies. The Collège des Médecins du Québec currently assesses the clinical competence of physicians previously identified with potential clinical competence difficulties through a day‐long procedure called the Structured Oral Interview (SOI). Two peer physicians produce a qualitative report. In view of remediation activities and the potential for legal consequences, more information on the clinical reasoning process (CRP) and quantitative data on the quality of that process is needed. This study examines the Script Concordance Test (SCT), a tool that provides a standardized and objective measure of a specific dimension of CRP, clinical data interpretation (CDI), to determine whether it could be useful in that endeavor. Methods Over a 2‐year period, 20 family physicians took, in addition to the SOI, a 1‐hour paper‐and‐pencil SCT. Three evaluators, blind as to the purpose of the experiment, retrospectively reviewed SOI reports and were asked to estimate clinical reasoning quality. Subjects were classified into 2 groups (below and above median of the score distribution) for the 2 assessment methods. Agreement between classifications is estimated with the use of the Kappa coefficient. Results Intraclass correlation for SOI was 0.89. Cronbach alpha coefficient for the SCT was 0.90. Agreement between methods was found for 13 participants (Kappa: 0.30, P = 0.18), but 7 out of 20 participants were classified differently in both methods. All participants but 1 had SCT scores below 2 SD of panel mean, thus indicating serious deficiencies in CDI. Discussion The finding that the majority of the referred group did so poorly on CDI tasks has great interest for assessment as well as for remediation. In remediation of prescribing skills, adding SCT to SOI is useful for assessment of cognitive reasoning in poorly performing physicians. The structured oral interview should be improved with more precise reporting by those who assess the clinical reasoning process of examinees, and caution is recommended in interpreting SCT scores; they reflect only a part of the reasoning process.

An innovative approach to remedial continuing medical education, 1992-2002.

The authors describe the process of remedial retraining programs organized and planned for Quebec physicians by the Collège des médecins du Québec (CMQ) and report the outcomes of these efforts from April 1992 to March 2002. The CMQ (the Quebec medical licensing authority) developed a process to identify physicians who had shortcomings in their clinical performance, determine their educational needs, propose, in collaboration with the four medical schools in the province, personalized retraining programs (clinical training programs, tutorials, focused readings, workshops, and refresher courses), and subsequently evaluate the impact of these retraining programs. During the ten-year period reported, 305 physicians (216 family physicians and 89 specialists) were referred to the Practice Enhancement Division of the CMQ for personalized remedial retraining. The vast majority of these physicians were men (81%). The following difficulties were identified: therapeutic knowledge (37%), diagnostic knowledge (32%), record-keeping (14%), technical skills (10%), clinical judgment (5%), and communication skills (2%). A total of 329 personalized retraining programs were completed: 273 clinical training programs, 41 tutorials, and 15 focused readings. A reevaluation of all these physicians showed that 70% of the retraining programs had succeeded, 15% were partially successful and only 13% had failed. The remaining 2% involved missing data or withdrawal of physicians. The authors conclude that the collaborative CME process described has important and effective original features.

Improving the quality of surgical pathology reports for breast cancer: a centralized audit with feedback.

CONTEXT Good communication of pathologic characteristics of a malignancy is crucial to therapy choices and accurate prognostication. The information must be easily retrieved from a surgical pathology report. OBJECTIVES To evaluate, first in 1999, the quality of surgical pathology reports for segmental breast resections for cancer in Quebec hospitals. Subsequently, to reevaluate, in 2003, the same indicators to determine if the first surveillance, with feedback, was associated with an improvement in the quality of the reports. DESIGN All Quebec hospitals performing the preset number of 20 or more segmental breast resections for cancer in 1999 and 2003 participated. A committee of pathologists, after review of the literature, chose 7 diagnostic elements deemed vital to a surgical pathology report for conservative breast cancer surgery. Medical archivists in each institution were instructed on how to retrieve the data. The main outcome measure was the presence or absence of the diagnostic information clearly presented on the surgical pathology report. RESULTS Fifty-one hospitals participated in 1999 and 50 in 2003. Overall, conformity improved from 85.0% in 1999 for the first evaluation to 92.5% in 2003 for the second evaluation (P < .001). Six of the 7 indicators showed an improvement in the level of conformity between the first and second evaluations. Conformity was weakest for recording the distance between the tumor and the resection margin (68.2%) and vascular/lymphatic invasion (61.4%) in 1999. CONCLUSIONS Surveillance of quality of surgical pathology reports, with feedback, is significantly associated with an improvement in the quality of reports.

When to recommend compulsory versus optional Cme programs? A study to establish criteria

When should remedial continuing medical education (CME) be compulsory for family physicians? When should it be optional? Should it be structured or not? In 1993-1994, the authors addressed this need for criteria by conducting a study that used reports on 14 physicians who had undergone a structured oral interview (SOI) at the College of Physicians of Quebec. (The SOI is a day-long encounter during which two specially trained physician-interviewers present a physician with standardized clinical cases that focus on ten specific aspects of a family physicians competence.) The 14 SOI reports were reviewed by 12 external physician-judges in an attempt to see how consistently they could link the ten aspects of competence, as shown in the reports, to five particular types of recommended remedial CME programs (the strictest being “compulsory program with suspended license” and the most lenient being “simple suggestions for improvement”). There was substantial agreement among the judges when choosing between compulsory and optional programs (kappa = 0.63, p < .05). The main criteria used when recommending an optional program were overall strengths and the quality of clinical reasoning. The same two criteria were also used for recommending a compulsory program, but the judges also considered three additional factors: the physicians ability to recognize his or her limits and how he or she handled referrals and prescribed medications. Many of the criteria used by the judges were based on unique information that came out of observations and interactions during the SOIs, such as quality of argumentation, sustaining a train of thought, sense for the case as a whole, or awareness of ones limits. Finally, the external judges corroborated the decisions previously made by the College of Physicians of Quebec concerning the appropriate CME programs for the 14 physicians.

Determining the Number of Patient Charts Necessary for a Reliable Assessment of Practicing Family Physicians’ Performance

In many countries, peer assessment programs based on the examination of patient charts are becoming a standard to assess physician’s clinical performance. Although data on validity of the process are acceptable, reliability issues need some improvement. This article addresses the rarely studied aspect of optimal number of patient charts for an acceptable reliable assessment. Fifteen patient charts for each of a group of 20 practicing physicians were independently reviewed by 4 professional peer assessors. Generalizability (G) and decision (D) studies were applied to the data. It appears that as few as 10 patient charts are sufficient for any assessor to obtain a G coefficient of 0.80. Results of the current study suggest the possibility of getting generalizable assessments by peer reviewer with minimal information. These results are not in accordance with the concept of case specificity in which it is claimed that performance on a case is a poor predictor of performance on a different case.

Expecting understanding, understanding expectations: Continuing medical education and the doctor-patient relationship

&NA; This paper describes the development and evaluation of a continuing medical education program on the doctor‐patient relationship. The Collège des Médecins du Québec and the Quebec chapter of the College of Family Physicians of Canada joined forces to meet the needs expressed by physicians in prior research. The course content focused on the structure of the encounter, the concepts of “offer and demand”, and “the patient‐centered interview”. The format was adapted to the audience: a 90‐minute workshop using two videotape vignettes, small group discussions, and a 15‐minute didactic presentation. The evaluations consisted of a self‐administered questionnaire at the end of the workshop and a mailed questionnaire 6 to 15 months later. The workshops were so popular that the total number of workshops (30) planned for over a 2‐year period were completed in less than 9 months. The great majority of the participants indicated in the immediate evaluations that the workshop had met their expectations, that they felt that they had met the objectives, and that they could see practical applications. The second evaluation corroborated the findings obtained in the initial evaluation and the opinion of the respondents continued to be positive in regard to the relevance and quality of the workshops. The authors are very pleased by the success of the workshops, which were conducted 74 times until December 1995. Two other workshops have been designed and are in the process of implementation. One workshop addresses issues of intimacy in the doctor‐patient relationship and the other focuses on difficult doctor‐patient relationships.