Mary R. Robeson

Do global rating forms enable program directors to assess the ACGME competencies

Purpose. In 1999 the Accreditation Council for Graduate Medical Education (ACGME) mandated that GME programs require their residents to be proficient in six general competencies. The purpose of this study was to ascertain whether an existing global rating form could be modified to assess these competencies. Method. A rating form covering 23 skills described in the ACGME competencies was developed. The directors of 92 specialty and subspecialty programs at Thomas Jefferson University Hospital and the Albert Einstein Medical Center in Philadelphia were asked to rate residents at the end of the 2001–02 and 2002–03 academic years. Results. Ratings for 1,295 of 1,367 (95%) residents were available. Residents were awarded the highest mean ratings on items tied to professionalism, compassion, and empathy. The lowest mean ratings were assigned for items related to consideration of costs in care and management of resources. Factor analysis indicated that the program directors viewed overall competence in two dimensions of medical knowledge and interpersonal skills. This factor structure was stable for groups of specialties, and residents’ gender and training level. Mean ratings in each dimension were progressively higher for residents at advanced levels of training. Conclusion. Global rating forms, the tool that program directors use most frequently to document residents’ competence, may not be adequate to assess the six general competencies. The results are consistent with earlier published research indicating that physicians view competence in just two broad dimensions, which questions the premise of the six ACGME competencies. Further research is needed to validate and measure six distinct dimensions of clinical competence.

The virtues of extended matching and uncued tests as alternatives to multiple choice questions.

The objectives of this study were to compare the reliability and validity of written test formats that are widely used in medical education (multiple choice, uncued, extended matching, and true/false) and evaluate the effects of uncued examinations on long-term retention of medical knowledge. Uncued tests were introduced into a traditional course in general and systemic pathology (six interim tests). In the following year, students were given eight tests written in the four formats, each being used twice. The academic achievement of students in these 2 years was compared with that of students in 2 previous years, in which multiple choice tests were used. Measures of academic achievement included performance on a final comprehensive examination and the United States Medical Licensing Examination (USMLE). Student performance on uncued tests was consistent over time (i.e., there was no learning curve). Mean scores ranged from 77% to 84%, and coefficient alpha reliability estimates on 100-item tests were excellent (0.79 to 0.90). Extended matching tests were also reliable, with a mean coefficient alpha of 0.90. There was no significant relationship between test format and student performance on subsequent comprehensive examinations. Our results indicate that extended matching and uncued tests have considerable advantages over multiple choice and true/false examinations. They are more reliable, better able to discriminate the well-prepared from the marginal student, and well suited for tested core knowledge. Contrary to our expectation, extended matching questions with 20 choices presented to the student were as statistically reliable and valid as uncued queries with several hundred choices.

Beta-blocker initiation and adherence after hospitalization for acute myocardial infarction:

Aims: We sought to: (1) estimate the proportion of patients who initiated beta-blocker therapy after acute myocardial infarction (AMI) in Regione Emilia-Romagna (RER); (2) examine predictors of post-AMI beta-blocker initiation; and (3) assess adherence to such therapy. Methods and Results: Using healthcare claims data covering all of RER, we identified a cohort of 24,367 patients with a hospitalization for AMI between 2004 and 2007, who were discharged from the hospital alive and without contraindications to beta-blocker therapy. We estimated the proportion of eligible patients with at least one prescription for a beta-blocker following discharge and performed a multivariable logistic regression analysis to identify independent predictors of post-AMI beta-blocker initiation. We computed the proportion of days covered (PCD) as a measure of medication adherence at 6 and 12 months post-discharge. Following discharge, 16,383 (67%) cohort members initiated beta-blocker therapy. Independent predictors of beta-blocker initiation included age and receipt of invasive procedures during hospitalization, such as coronary artery bypass graft surgery (odds ratio [OR], 2.37; 95% confidence interval [CI], 2.00–2.81), percutaneous transluminal coronary angioplasty (OR, 1.42; 95% CI, 1.31–1.54), and cardiac catheterization (OR, 1.21; 95% CI, 1.11–1.32). Among initiators, adherence to beta-blocker treatment at 6 and 12 months was low and decreased in each study year. Conclusion: Overall, use of and adherence to post-AMI beta-blocker therapy was suboptimal in RER between 2004 and 2007. Older patients and those with indicators of frailty were less likely to initiate therapy. The proportion of patients adherent at 6 and 12 months decreased over time.

Gender Comparisons Prior to, during, and after Medical School Using Two Decades of Longitudinal Data at Jefferson Medical College.

Similarities and differences prior to, during, and after medical school between 3,541 men and 1,121 woman graduates of Jefferson Medical College were investigated. Gender comparisons were made on examination scores, admissions interview ratings, competence ratings in residency, specialty choice, board certification, income estimates, and academic appointments. Results indicated that prior to medical school, women scored higher on verbal tests, whereas men outscored women on quantitative and science tests. During medical school, men performed better than women in the basic science examinations, but not in the clinical science examinations. Men and women had similar postgraduate competence ratings except that women were rated higher than men in the socioeconomic aspects of patient care. Women had lower board certification rates, expected less income, and had a higher proportion offaculty appointments than did men. Gender differences in specialty choices, faculty appointments, and estimated income could have important implications for health care manpower

Predicting risk of hospitalisation or death: a retrospective population-based analysis

Objectives Develop predictive models using an administrative healthcare database that provide information for Patient-Centred Medical Homes to proactively identify patients at risk of hospitalisation for conditions that may be impacted through improved patient care. Design Retrospective healthcare utilisation analysis with multivariate logistic regression models. Data A population-based longitudinal database of residents served by the Emilia-Romagna, Italy, health service in the years 2004–2012 including demographic information and utilisation of health services by 3 726 380 people aged ≥18 years. Outcome measures Models designed to predict risk of hospitalisation or death in 2012 for problems that are potentially avoidable were developed and evaluated using the area under the receiver operating curve C-statistic, in terms of their sensitivity, specificity and positive predictive value, and for calibration to assess performance across levels of predicted risk. Results Among the 3 726 380 adult residents of Emilia-Romagna at the end of 2011, 449 163 (12.1%) were hospitalised in 2012; 4.2% were hospitalised for the selected conditions or died in 2012 (3.6% hospitalised, 1.3% died). The C-statistic for predicting 2012 outcomes was 0.856. The model was well calibrated across categories of predicted risk. For those patients in the highest predicted risk decile group, the average predicted risk was 23.9% and the actual prevalence of hospitalisation or death was 24.2%. Conclusions We have developed a population-based model using a longitudinal administrative database that identifies the risk of hospitalisation for residents of the Emilia-Romagna region with a level of performance as high as, or higher than, similar models. The results of this model, along with profiles of patients identified as high risk are being provided to the physicians and other healthcare professionals associated with the Patient Centred Medical Homes to aid in planning for care management and interventions that may reduce their patients’ likelihood of a preventable, high-cost hospitalisation.

USMLE Step 2 performance and test administration date in the fourth year of medical school.

Purpose. To determine whether the time interval between completing the third-year curriculum and test administration affects a students USMLE Step 2 score. Method. Scores for 846 students in the classes of 2000–2004 were grouped in ten time periods depending on test date. A linear regression model to predict performance on Step 2 using gender, Step 1, and grades in medicine, pediatrics and obstetrics–gynecology was developed based on the class of 1999. Analysis of covariance was used to test the effect of time on scores, adjusting for predicted performance. Results. Step 2 scores decreased significantly (p < .001) across time. Students’ mean scores were four points higher than predicted in the early months and five to eight points lower near the end of the senior year. Conclusions. Students who scheduled Step 2 early in the senior year achieved higher scores, on average, than those who waited until later in the year.

A statewide system to track medical students' careers: the Pennsylvania model.

In 1994 the Commonwealth of Pennsylvania announced a statewide Generalist Physician Initiative (GPI) modeled after The Robert Wood Johnson Foundations GPI. Three-year grants totaling more than

Sooner or Later? Usmle Step 1 Performance and Test Administration Date at the End of the Second Year

9 million were awarded to seven of Pennsylvanias medical schools, including two that had already received GPI grants from the foundation. Stimulated by these initiatives, the states six allopathic and two osteopathic medical schools decided to work together to develop a collaborative longitudinal tracking system to follow the careers of all their students from matriculation into their professional careers. This statewide data system, which includes information for more than 18,000 students and graduates beginning with the entering class of 1982, can be used to evaluate the impact of the Pennsylvania GPI, and it also yielded a local longitudinal tracking system for each medical school. This paper outlines the concept of the system, its technical implementation, and the corresponding implications for other medical schools considering the development of similar outcomes assessment systems.

Who is most likely to require a higher level of care: predicting risk of hospitalization.

Young physicians must pass the United States Medical Licensing Examination (USMLE) for state licensure and specialty board certification. During 2000–2001, nearly all U.S. medical schools (115) required that students attempt Step 1 at some point during their MD programs. Students at schools with curricula consisting of two years of basic science education followed by two years of clinical rotations are often required to complete Step 1 of this comprehensive examination immediately after the end of the second year of medical school to document their basic science knowledge before proceeding into the clinical sciences. The consequences of performance on the USMLE extend beyond medical school. Step 1 has become an important factor in screening and selecting residency candidates through the electronic application process. This cumulative examination has become even more of a high-stakes test among those vying for the most competitive residency programs. For example, the Web site for the San Francisco Matching Program (^http://www.sfmatch.org/&) posts in the dean section the mean Step 1 scores for students applying to graduate medical education programs in otolaryngology, ophthalmology, and neurological surgery. The mean scores for these competitive specialties are significantly higher than the national mean scores. Before the examination became available on computer in 1999, Step 1 was administered twice annually as a paper-and-pencil test. Previously, the majority of examinees chose to take the test in May, but now the computer-based version can be scheduled throughout the year. Students can schedule the examination at their own personal convenience; however, they are subject to the confines of their medical schools’ curricula and the faculty’s policies for promotion and graduation. Little is known about how this change to flexible test scheduling affects student performances. This information can have an impact on the advice that advisors must provide to medical students. In this study, we analyzed the relationships between students’ Step 1 scores and the time intervals that had passed after they had completed the second-year medical school curriculum.

Who is a generalist? An analysis of whether physicians trained as generalists practice as generalists.

Health-care services are increasingly focused on care of individuals with chronic illness. Predicting those who are at greatest risk is critical in the design of case or disease management programs. The goal of this project is to develop models that can be used to identify people who are at high risk for hospitalization, and who would potentially benefit from participating in chronic disease management programs.