John L. Falcone

The American Board of Surgery Certifying Examination: a retrospective study of the decreasing pass rates and performance for first-time examinees.

BACKGROUND There has been a noticeable decrease in the pass rate for the American Board of Surgery Certifying Examination during the last 5 years. We hypothesize that this decline is statistically significant, and we wish to determine whether the pass rates had any geographic patterns of distribution. METHODS In this retrospective cohort trial, publically available electronic data sets of pass rates on the American Board of Surgery Certifying Examination were evaluated from the American Board of Surgery website. χ(2) tests were used to determine whether there was any association between the pass rates and the academic year. A descriptive geographic evaluation of program-specific pass rates for first-time examinees was also performed. RESULTS From 2006 to 2010, there has been a 7% decrease in the pass rate for the American Board of Surgery Certifying Examination. A χ(2) test shows that there is a statistically significant association with the pass percentage on the American Board of Surgery Certifying Examination and the year (p < 0.0001). Subgroup analysis demonstrated a difference in pass rate between 2006 and 2007 (p = 0.02). Geographic analysis showed the pass rates for first-time examinees were the highest in Rhode Island (100%) and the lowest in Puerto Rico (63%) from 2005 to 2010. Three of the 5 highest-performing states are on the Pacific Coast, and 4 of the 9 lowest-performing states are in the southern United States. There are differences between these 2 groups of states (p < 0.001). CONCLUSIONS There was a significant decrease in the pass rate for the American Board of Surgery Certifying Examination from 2006 to 2010. There also were some geographic patterns relating to first-time examinee performance from 2005 to 2010.

Practice-Based Learning and Improvement: A Two-Year Experience with the Reporting of Morbidity and Mortality Cases by General Surgery Residents

BACKGROUND The Accreditation Council for Graduate Medical Education (ACGME) core competency of practice-based learning and improvement can be assessed with surgical Morbidity and Mortality Conference (MMC). We aim to describe the MMC reporting patterns of general surgery residents, describe the adverse event rate for patients and compare that with existing published rates, and describe the nature of our institutional adverse events. We hypothesize that reporting patterns and incidence rates will remain constant over time. METHODS In this retrospective cohort study, archived MMC case lists were evaluated from January 1, 2009 to December 31, 2010. The reporting patterns of the residents, the adverse event ratios, and the specific categories of adverse events were described over the academic years. χ(2) and Fishers exact tests were used to compare across academic years, using an α = 0.05. RESULTS There were 85 surgical MMC case lists evaluated. Services achieved a reporting rate above 80% (p < 0.001). The most consistent reporting was done by postgraduate year (PGY) 5 level chief residents for all services (p > 0.05). Out of 11,368 patients evaluated from complete MMC submissions, 289 patients had an adverse event reported (2.5%). This was lower than published reporting rates for patient adverse event rates (p < 0.001). Adverse event rates were consistent for residents at the postgraduate year 2, 4, and 5 levels for all services (p > 0.05). Over 2 years, 522 adverse events were reported for 461 patients. A majority of adverse events were from death (24.1%), hematologic and/or vascular events (16.7%), and gastrointestinal system events (16.1%). CONCLUSIONS Surgery resident MMC reporting patterns and adverse event rates are generally stable over time. This study shows which adverse event cases are important for chief residents to report.

Compliance on the American Board of Pediatrics Certifying Examination and the Importance of Location and Size on Pass Rates

Objective. To describe residency program compliance to a 60% pass rate and 80% eligibility standard outlined by the Accreditation Council for Graduate Medical Education (ACGME) on the American Board of Pediatrics Certifying Examination. The hypothesis is that larger programs will have higher pass rates. Methods. Pediatric residency programs were retrospectively evaluated from 2008 to 2010 regarding the ACGME standards. Simple linear regression was performed to see if program pass rates were dependent on program size. Results. A total of 162/163 (83.4%) programs had first-time examinee pass rates of 60%. A total of 179/193 (92.7%) programs satisfied the 80% eligibility standard. The Northeast performed lower than Midwest, Southern, and Western states (P < .001). The West performed higher than the Northeast, Midwest, and Southern states (p < 0.05). Simple linear regression showed that performance depends on program size (P < .001). Conclusions. A majority of programs satisfy the minimum ACGME standards. Program performance is associated with program size and location. These findings may alter application patterns to pediatric residency programs.

Using Elements from an Acute Abdominal Pain Objective Structured Clinical Examination (OSCE) Leads to More Standardized Grading in the Surgical Clerkship for Third-Year Medical Students

BACKGROUND There is poor reliability in the Likert-based assessments of patient interaction and general knowledge base for medical students in the surgical clerkship. The Objective Structured Clinical Examination (OSCE) can be used to assess these competencies. OBJECTIVE We hypothesize that using OSCE performance to replace the current Likert-based patient interaction and general knowledge base assessments will not affect the pass/fail rate for third-year medical students in the surgical clerkship. METHODS In this retrospective study, third-year medical student clerkship data from a three-station acute abdominal pain OSCE were collected from the 2009-2010 academic year. New patient interaction and general knowledge base assessments were derived from the performance data and substituted for original assessments to generate new clerkship scores and ordinal grades. Two-sided nonparametric statistics were used for comparative analyses, using an α = 0.05. RESULTS Seventy third-year medical students (50.0% female) were evaluated. A sign test showed a difference in the original (4.45/5) and the new (4.20/5) median patient interaction scores (p < 0.01). A sign test did not show a difference in the original (4.00/5) and the new (4.11/5) median general knowledge base scores (p = 0.28). Nine clerkship grades changed between these different grading schemes (p = 0.045), with an overall agreement of 87.1% and a kappa statistic of 0.81. There were no differences in the pass/fail rate (p > 0.99). CONCLUSIONS We conclude that there are no differences in pass/fail rate, but there is a more standardized distribution of patient interaction assessments and utilization of the full spectrum of possible passing grades. We recommend that the current patient interaction assessment for third-year medical students in the surgical clerkship be replaced with that found through trained standardized patients in this three-station acute abdominal pain OSCE.

Surgical Morbidity and Mortality Conference using teleconferencing allows for increased faculty participation and moderation from satellite campuses and saves costs.

INTRODUCTION Surgical faculty participation in Morbidity and Mortality Conference (MMC) satisfies criteria for Continuing Medical Education (CME) credit. We hypothesize that using teleconferencing between the main campus to 2 satellite campuses will quantitatively increase faculty attendance and participation as a moderator at surgical MMC. We also want to perform a cost-benefit analysis of teleconferencing. METHODS In this retrospective descriptive study, faculty attendance at MMC at the main campus and 2 satellite campuses was observed from January 1, 2010 through December 31, 2010. Groups were compared with nonparametric statistics, using an α = 0.05. We performed an annual cost-benefit analysis of teleconferencing with consultation of an economist/financial advisor. The explicit and implicit costs of teleconferencing were compared with the opportunity cost-benefit of travel prevention. RESULTS In 2010, there were 45 MMC activities. A total of 236 Continuing Medical Education credit hours were reported, with 186 credit hours at the main campus and 68 credit hours at the satellite campuses. A Mann-Whitney U test showed an increase in the median total attendance (5 per conference) with the addition of the satellite campus attendance (2 per conference) (p < 0.001). There were no differences between the number of moderators at the main campus and the satellite campuses (p > 0.99). Cost benefits per faculty member was

Performance on the American Board of Family Medicine Certification Examination by Country of Medical Training

96.70 per conference at 1 satellite campus and

Validity and Interrater Reliability of a Regional Mock Oral Board Examination

193.60 per conference at the second satellite campus. A total of 73.1 hours of travel time was prevented, with a total annual net benefit of

Differential Diagnosis in a 3-Station Acute Abdominal Pain Objective Structured Clinical Examination (OSCE): A Needs Assessment in Third-Year Medical Student Performance and Summative Evaluation in the Surgical Clerkship

7624. CONCLUSIONS Teleconferencing allows for increased faculty attendance at MMC and allows for faculty to stay at their respective hospitals for patient care. Teleconferencing also results in significant cost savings. We recommend that institutions with similar resources consider teleconferencing as a way to increase faculty member attendance at surgical MMC and to save hospital costs.

Pass Rates on the American Board of Family Medicine Certification Exam by Residency Location and Size

Background: Performance on the American Board of Family Medicine (ABFM) Certification and Recertification Examinations by country of medical school training has not been examined. Based on internal medicine patterns, we hypothesize that examinees trained in the United States and Canada would outperform examinees trained in other countries. Methods: In this retrospective cohort study from 2004 to 2011, data on the ABFM examinations were obtained from the ABFM. Fisher exact and χ2 tests were performed across years based on the country of examinee training. Simple linear regression was performed to evaluate pass rates over time. All statistics were performed using an α = 0.05. Results: The overall pass rate over the study period was 84.4% (74,821 of 88,680). The pass rate for US medical graduates (USMGs) was 88.3% (60,328 of 68,332). The pass rate for Canadian medical graduates (CMGs) was 93.8% (872 of 930). The pass rate for non-Canadian foreign medical gradates (NC-FMGs) was 70.1% (13,621 of 19,418). CMGs had a higher pass rate than USMGs (P < .001) and NC-FMGs (P < .001). Simple linear regression showed significant decreasing trends over time for all examinees (P = .02), for USMGs (P = .02), and for CMGs (P = .02). Conclusions: USMGs and CMGs outperform NC-FMGs on the ABFM certification and recertification examinations. These findings may alter acceptance patterns for Family Medicine residency programs.

Home-Field Advantage: The Role of Selection Bias in the General Surgery National Residency Matching Program

BACKGROUND There have been decreasing pass rates recently on the American Board of Surgery Certifying Examination (ABSCE). General surgery residents from the University of Pittsburgh Medical Center, the West Penn Allegheny Health System, the Conemaugh Health System, and Mercy Hospital participate in a mock oral board examination, which is similar to the ABSCE. The aims of the study are to compare examinee performance on the mock oral boards with the ABSCE and to evaluate the interrater reliability of examiner pairs. METHODS In this retrospective study from 2003 to 2010, outcomes on the mock oral boards and the first attempt of the ABSCE for chief residents were compared for the 4 regional residency programs. Interrater reliability for examiner pairs was evaluated with agreement and kappa statistics. Nonparametric statistics were performed, with α = 0.05. RESULTS A total of 32 of 38 (84.2%) chief residents passed the mock oral boards. The median score for each of the 3 rooms was 6 (clear pass). A total of 37 of 38 (97.4%) residents passed the ABSCE. The sensitivity of the mock oral boards was 83.8%, with a positive predictive value of 96.9%, and an accuracy of 81.6%. A total of 25 of 47 (53.2%) examiner pairs were from the same residency institution, whereas 22 of 47 (46.8%) were from different institutions. The median agreement was 100% (interquartile range (IQR) [100% - 100%]). The median kappa statistic was 1.00 (IQR [0.38-1.00]). The Mann-Whitney U tests showed no difference in agreement or kappa for examiner pairs from the same or from different institutions (p> 0.05). CONCLUSIONS The mock oral boards have substantial sensitivity and positive predictive value in relation to the ABSCE. There are also very high levels of interrater agreement and interrater reliability. This regional mock oral board examination is valuable for ABSCE preparation.