Ernest N. Skakun

Performance on ABA-ASA in-training examination predicts success for RCPSC certification

Purpose: Most Canadian University Departments of Anesthesia require residents to take the American Board of Anesthesiology-American Society of Anesthesiologists (ABA-ASA) in-training examination (ITE). The result is expressed as a percentile relative to all examinees at similar levels of training. Its value as a predictor of performance in the Royal College of Physicians and Surgeons of Canada (RCPSC) certification examinations is not known.Methods: All English speaking Canadian residency programs in Anesthesia were surveyed. Results of the ABA-ASA ITE of former residents who had completed RCPSC certification were collected as percentile scores according to level of training. Level of training was based on the number of months of anesthesia training and classified according to American residency program nomenclature. The ABA-ASA ITE scores were correlated with success on the RCPSC written and oral examinations. The probability of success on the RCPSC examinations was determined by calculating the cutoff score with the best sensitivity and specificity as determined by Receiver Operating Characteristic (ROC) curves for each year in which the examination was taken and for both the written and oral examinations.Results: Nine residency programs provided information on 165 residents. A weak positive correlation was found between scores on each year of the ABA-ASA exam. Scores >50th percentile for any year were highly predictive of success in the written component (>60th percentile for the oral component). Scores<20th percentile were predictive of failure on both the written and oral components of the RCPSC examination.Conclusion: The ABA-ASA ITE is a useful tool in predicting performance on the RCPSC examination.RésuméObjectif: La plupart des départements d’anesthésie des universités canadiennes exigent de leurs résidents qu’ils se présentent à l’examen américain des résidents en cours de formation de l7rsAmerican Board of Anesthesiology-American Society of Anesthesiologists (ABA-ASA) in-training examination (ITE). Le résultat est exprimé en percentile relatif au nombre total de candidats au même niveau de formation. Sa valeur prédictive de performance à l’examen de certification du Collège royal des médecins et chirurgiens du Canada (CRMCC) n’est pas connue.Méthode: Tous les programmes canadiens anglais de résidence en anesthésie ont été étudiés. Les résultats de l’ABA-ASA ITE des anciens résidents qui se sont présentés à l’examen du CRMCC ont été recueillis et cotés selon le niveau de formation. Le niveau de formation a été fondé sur le nombre de mois de formation en anesthésie et classé selon la nomenclature du programme américan de résidence. La corrélation des scores d’ABA-ASA ITE avec les examens écrits et oraux du CRMCC a été un succès. La probabilité de réussite des examens du CRMCC a été déterminée par le calcul de la cote minimale ou maximale du meilleur taux de sensibilité et de spécificité établis au moyen de l’analyse par les courbes ROC pour chaque année où l’examen a été fait et pour les examens oraux et écrits.Résultats: Neuf programmes de résidence ont fourni des renseignements sur 165 résidents. Une faible corrélation positive a été découverte entre les scores de chaque année d’examen de l’ABA-ASA. Les scores >50e percentile, quelle que soit l’année, ont été fortement prédictifs de succès pour la composante écrite de l’examen et les scores >60e percentile, pour la composante orale. Les scores <20e percentile ont été prédictifs d’échecs aux examens oraux et écrits du CRMCC.Conclusion: L’ABA-ASA ITE est utile pour prédire la performance à l’examen du CRMCC.

The inter-rater and intra-rater reliability of a new Canadian oral examination format in anesthesia is fair to good

PurposeIn response to the Royal College’s request to improve the validity and reliability of oral examinations, the Examination Board in anesthesia proposed a structured oral examination format. Prior to its introduction, we studied this format in two residency programs to determine reliability of the examiners.MethodsTwenty faculty and 26 residents from two Canadian residency programs participated (Sites A and B). Pairs of examiners scored five or six residents examined consecutively on two standardized questions using global rating scales with anchored performance criteria. Residents’ performances were scored independently during the examination (Time 1) and later from a videotaped recording (Time 2). Correlations between scores of the pairs of examiners and between scores of each examiner were determined.ResultsCorrelations demonstrating inter-rater agreement between examiners at Site A ranged from -.324 to .915 (mean .506) at Time 1. At Time 2, correlations ranged from .64 to .887 (mean .791). At Site B correlations ranged from .279 to .989 (mean .707) at Time 1 and at Time 2 correlations ranged from - .271 to .924 (mean .477).Correlations demonstrating intra-rater agreement of examiners at Site A ranged from .054 to .983 (mean .723) and at Site B correlations ranged from -.055 to .974 (mean .662).Correlations > 0.4 were seen in 80% of the scores and > 0.7 in 50% indicating fair to good intra-rater and interrater reliability using this format.ConclusionsDespite the limitations of our study our results compare favourably with those previously reported in anesthesia. We recommend the adoption of this format to the Royal College of Physicians and Surgeons of Canada Examination Board.RésuméObjectifC’est à la demande du Collège royal, d’améliorer la validité et la fiabilité des examens oraux, que le Bureau des examinateurs en anesthésie a proposé un modèle d’examen oral structuré. Avant sa mise en application, nous l’avons testé dans deux programmes de résidence afin de déterminer la fiabilité des examinateurs.MéthodeVingt facultés et 26 résidents de deux programmes canadiens ont participé à l’étude (Sites A et B). Des paires d’examinateurs ont utilisé une échelle de notation globale comportant des critères de rendement définis pour évaluer cinq ou six résidents appelés à répondre consécutivement à deux questions normalisées. Les résultats des résidents ont été cotés séparément pendant l’examen (Temps 1) puis, à partir d’un enregistrement vidéo (Temps 2). Les corrélations entre les scores des paires d’examinateurs et entre les scores de chaque examinateur ont été établies.RésultatsLes corrélations démontrant une concordance interexaminateurs au Site A sont de −0,324 à 0,915 (moyenne de 0,506) au Temps 1. Au Temps 2, de 0,64 à 0,887 (moyenne de 0,791). Au Site B, elles sont de 0,279 à 0,989 (moyenne 0,707) au Temps 1,et au Temps 2 de −0,271 à 0,924 (moyenne de 0,477). Les corrélations sur la l’accord intra-examinateurs au Site A sont de 0,054 à 0,983 (moyenne de 0,723) et au Site B de −0,055 à 0,974 (moyenne de 0,662). Les corrélations étaient > 0,4 dans 80 % des scores et > 0,7 dans 50 %; la fiabilité intra-examinateurs et interexaminateurs ainsi indiquée est de moyenne à bonne avec ce modèle.ConclusionMalgré les limites de notre étude, nos résultats se comparent favorablement avec ceux qui ont déjà été signalés en anesthésie. Nous recommandons l’adoption du modèle par le Bureau des examinateurs du Collège royal des médecins et chirurgiens du Canada.

The use of computerized patient management problems in a certifying examination

The 1974 paediatiic certification process of the Royal College of Physicians and Surgeons of Canada included four CPMPs, in addition to the conventional MCQs and orals. The CPMPs were successfully administered to 160 candidates in eight cities across Canada on the same day. The examination was judged by the candidates to be a better test of clinical skill than MCQs. The CPMP examination did not penalize French‐speaking candidates, was economically feasible, and a good security risk. Correlations between MCQ, orals and CPMPs, indicated that each examination measured some aspects of paediatric competence not tested by the others.

Strategy choices in multiple-choice items

No abstract available.

Setting Standards for Multiple-Choice Items in Clinical Reasoning

Standard setting is a critical component in licensing decisions. In this article, it is argued that Grosss modification of the Nedelsky procedure is a valid approach to setting standards when clinical reasoning is measured using multiple-choice questions. By defining minimum competence in terms of how candidates think about problems, the Nedelsky approach more closely reflects the construct than do competing procedures. An example application taken from the Medical Council of Canadas Qualifying Examination is used to show that the procedure leads to credible standards, that the assumptions on which the process is based are reasonable, and that the standards are consistent over time.

A preliminary investigation of three types of multiple choice questions

The present study reports the results of an evaluation of three types of multiple choice questions—the five choice completion, multiple completion, and assertion‐reason. Fifty‐four questions, eighteen of each type and measuring the candidate on the same scientific principle and classified as either factual or comprehension, were developed and included in the General Surgery certifying examination of the Royal College of Physicians and Surgeons of Canada. In addition to using descriptive statistics, the multitrait‐multi‐method technique was used to investigate whether the item types measured different aspects of examinee capabilities. Results indicated that performance on the five choice completion and the multiple completion type questions was roughly the same, whereas performance on the assertion‐reason type was lower. The results of the multitrait‐multimethod validation revealed that the three item types were unable to discriminate between the two traits of factual and comprehension.

The research productivity of Canadian physicians: how the timing of obtaining a PhD has an influence.

Purpose To determine whether the sequence of training to obtain MD and PhD degrees is associated with different career paths for physicians who have their PhD before medical school and those who obtain it after their MD, and to explore the factors that encourage or dissuade Canadian dual-degree physicians in pursuing a research career. Method In 2003, questionnaires from the University of Alberta, Edmonton, Canada, were sent to all 734 Canadian physicians having MDs and PhDs, identified through the Canadian Medical Directory. Data collected were gender; year and country of MD; sequence of obtaining degrees; portion of time on clinical, research, teaching, and administrative duties; number of publications and currently held grant amounts; and perceived incentives and disincentives to research careers. Two focus groups were held with a subset of physicians to further explore themes. Results The response rate was 64%. On the basis of the timing of the PhD relative to the MD, physicians were designated early PhDs (26%), concurrent PhDs (12%), or late PhDs (62%). Late PhDs spent more time in research and less time on clinical practice than the other two groups and spent more time teaching and had published more papers than the early PhDs. Grant amounts were highest for late PhDs. Lack of time and resources were the major disincentives to research, and noteworthy incentives were the opportunity for intellectual challenge and creativity, and previous research experience. Conclusions Physicians who obtain a PhD after an MD have a more research-focused career than those who enter medical school with a PhD.

A PRELIMINARY INVESTIGATION OF COMPUTERIZED PATIENT MANAGEMENT PROBLEMS IN RELATION TO OTHER EXAMINATIONS

In order to provide a better assessment of the performance of medical students, new testing devices have been introduced. One of these devices labelled as the computerized patient management problem has been used by the Royal College of Physicians and Surgeons of Canada as an additional examination in the final certifying process of pediatricians. The present paper uses factor analysis to determine whether computerized patient management problems span the test space spanned by multiple choice examinations and a rating scale. From the analysis it would appear that the computerized problems overlap with the multiple choice examinations to a certain degree and bear no relationship to the rating scale.

Paediatric candidates' attitudes to computerized patient management problems in a certifying examination

In September 1974 candidates who had taken a computerized patient management problem examination (CPMP), as part of the certification process in paediatrics required by the Royal College of Physicians and Surgeons of Canada, completed a questionnaire designed to elicit their reactions to CPMPs. The results indicated that respondents were favourable to CPMPs, that there was little distraction caused by the equipment, logic or semantics of the problems, and that CPMPs were an acceptable examination technique. It was recommended that, in the future, CPMP examinees be allowed practice time to familiarize themselves with the equipment, that pre‐examination instructions should be clarified, and that problems judged inadequate by candidates be revised.

The Effect of Health Reform on Surgical Clerks' Experience and Performance

A recent health reform initiative has been the regionalisation of health care delivery in the province, including the conversion of two of four active treatment hospitals in the City of Edmonton to community health centres. The regionalisation occurred midway through the academic year, when 76 of the 116 students of the Class of 1996 had already completed the surgical clerkship. Following regionalisation, 40 students were assigned to the remaining two hospitals to complete their rotations. The purpose of this study was to determine the impact of regionalisation on students’ clinical experience and performance. Results revealed that students who completed their surgical clerkship following regionalisation completed fewer elective and emergency admissions and received less personal feedback and fewer chart notes from their preceptors than those who undertook the surgical clerkship before regionalisation. Performance on the ward and OSCE was similar for both groups, but students’ scores on the MCQ declined ‘postregionalisation.’ These health reforms affected the quantity of clinical experience and personal feedback afforded surgical clerks, and grades declined on one of two objective measures of performance.