Thomas R. O'Neill

Maintenance of certification and its association with the clinical knowledge of family physicians.

Purpose To examine the relationship between maintenance of certification (MOC) and the clinical knowledge demonstrated by family physicians as they move further away from formal training. Method Performances of 10,801 examinees—2,440 seeking initial certification; 8,361 seeking MOC—on the summer 2009 American Board of Family Medicine (ABFM) certification examination were compared across 30 cohorts that represented recent residency program graduates and already-certified family physician diplomates with varying years of clinical experience. Experience was defined as the time in years since the year of initial certification. This study employed a natural-groups, cross-sectional design; however, it was used to draw longitudinal inferences. Results Family physicians who maintained certification performed better than recent graduates. They increased their examination scores by almost 17 points each successive time that they took the exam, with scores reaching their highest point 28 to 31 years after initial certification. Multiple comparison analyses confirmed that the trend was significant; however, subanalyses revealed that this trend remained significant only for U.S. medical graduates (USMGs) but not international medical graduates. Those family physicians that did not maintain their certification performed significantly worse than recent graduates. Conclusions The findings suggest that ABFM diplomates who are USMGs and maintain their certification perform better on the ABFM certification examination with additional years of experience until approximately 30 years after residency training.

Prevalence of Burnout in Board Certified Family Physicians

Physician burnout has become a critical issue in a rapidly changing health care environment and is reported to be increasing. However, little is known about the prevalence of this problem among board-certified family physicians. Using an abbreviated burnout survey, we found a lower prevalence of this problem than has been previously reported.

Family Physicians' Scope of Practice and American Board of Family Medicine Recertification Examination Performance

Purpose: Previous research indicated that rural family physicians were more likely to pass the American Board of Family Medicine (ABFM) Maintenance of Certification for Family Physicians (MC-FP) examination. One possible explanation is that rural family physicians may have a broader scope of practice. Method: This was a cross-sectional study of family physicians taking the ABFM MC-FP examination in 2013. Examination results were linked with the Scope of Practice for Primary Care (SP4PC) scale. Linear and logistic regression models, with and without SP4PC score, determined associations between scope of practice and examination results. Results: Among 10,978 examinees, rural physicians had a higher passing rate (90.7% vs 86.8%, P < .05) and higher SP4PC score (16.1 vs 14.3 P < .05) compared with urban physicians. Regression models without SP4PC score confirmed that urban physicians were less likely to pass (OR = 0.73; 95% CI, 0.62–0.87) and scored lower, −15.6 points, compared with rural physicians. Including SP4PC score completely attenuated the relationship between practice location and passing (OR = 0.86; 95% CI, 0.73–1.02) and decreased the relationship between score and practice location (−5.8 points). Each point increase on the SP4PC score was associated with 9% higher odds of passing (OR = 1.09; 95% CI, 1.07–1.11) and 4.9 more points. Conclusion: A broader scope of practice rather than rural or urban practice location, was associated with increased likelihood of passing the MC-FP examination. If higher board scores are associated with providing higher quality of care, then maintaining a broad scope of practice may enable the delivery of higher quality primary care.

Performance on the American Board of Family Medicine (ABFM) Certification Examination: Are Superior Test-Taking Skills Alone Sufficient to Pass?

Introduction: Certification examinations used by American specialty boards have been the sine qua non for demonstrating the knowledge sufficient for attainment of board certification in the United States for more than 75 years. Some people contend that the examination is predominantly a test of superior test-taking skills rather than of family medicine decision-making ability. In an effort to explore the validity of this assertion, we administered the American Board of Family Medicine (ABFM) Certification to examinees who had demonstrated proficiency in taking standardized tests but had limited medical knowledge. Methods: Four nonphysician experts in the field of measurement and testing were administered one version of the 2009 ABFM certification examination. Scaled scores were calculated for each examinee, and psychometric analyses were performed on the examinees responses to examination items and compared with the performance of physicians who took the same examination. Results: The minimum passing threshold for the examination was a scaled score of 390, corresponding to 57.7% to 61.0% of questions answered correctly, depending on the version of the examination. The 4 nonphysician examinees performed poorly, with scaled scores that ranged from 20 to 160 (mean, 87.5; SD, 57.4). The number of questions answered correctly ranged from 24.0% to 35.1% (mean, 29.2%; SD, 0.05%). Rasch analyses of the examination items revealed that the nonphysician examinees were more likely to use guessing strategies in an effort to answer questions correctly. Distracter analysis suggest near-complete randomness in the nonphysician responses. Conclusions: Though all 4 nonphysician examinees performed better than would have been predicted by chance alone, none performed well enough to even fall within 8 SE below the passing thresholds; their performance was far below that of almost all physicians who completed the examination. Given that the nonphysicians relied heavily on the identifying cues in the phrasing of items and the manner in which response options were presented, the results affirm the notion that the ABFM certification examination is not primarily a measure of generic test-taking ability but measures information critical to the estimation of a family physicians knowledge sufficient for certification. Item analysis confirmed that items were well written, provided minimal cueing, and required medical knowledge to answer correctly.

Examining the Functioning and Reliability of the Family Medicine Milestones

BACKGROUND The Family Medicine (FM) Milestones are a framework designed to assess development of residents in key dimensions of physician competency. Residency programs use the milestones in semiannual reviews of resident performance from entry toward graduation. OBJECTIVE To examine the functioning and reliability of the FM Milestones and to determine whether they measure the amount of a latent trait (eg, knowledge or ability) possessed by a resident or simply indicate where a resident falls along the training sequence. METHODS This study utilized the Rasch Partial Credit model to examine academic year 2014-2015 ratings for 10 563 residents from 476 residency programs (postgraduate year [PGY] 1 = 3639; PGY-2 = 3562; PGY-3 = 3351; PGY-4 = 11). RESULTS Reliability was exceptionally high at 0.99. Mean scores were 3.2 (SD = 1.3) for PGY-1; 5.0 (SD = 1.3) for PGY-2; 6.7 (SD = 1.2) for PGY-3; and 7.4 (SD = 1.0) for PGY-4. Keyform analysis showed a rating on 1 item was likely to be similar for all other items. CONCLUSIONS Our findings suggest that FM Milestones seem to largely function as intended. Lack of spread in item difficulty and lack of variation in category probabilities show that FM Milestones do not measure the amount of a latent trait possessed by a resident, but rather describe where a resident falls along the training sequence. High reliability indicates residents are being rated in a stable manner as they progress through residency, and individual residents deviating from this rating structure warrant consideration by program leaders.

Response: Re: Performance on the Maintenance of Certification for Family Physicians (MC-FP) Examination: Comparison of Initial Certifiers with Experienced Physicians

The perception that state-of-the-art clinical knowledge declines as a physician moves further away from formal training is prevalent. This perception is reinforced by a significant body of research. As a result, seasoned family physicians may have concerns that the American Board of Family Medicine’s Maintenance of Certification for Family Physicians (MC-FP) examination may be biased against them. However, recent research has found that family physicians maintaining their certification performed better than recent graduates, with scores reaching their highest point approximately 30 years after their initial certification. The belief that the examination is biased against veteran physicians or merely fails to recognize their years of additional experience may be partially reinforced by only considering the MC-FP examination’s passing rates without regard to the distribution of scores. To illustrate, the April 2013 MC-FP results for US medical graduates who did not fail their most recent previous attempt, are described (Table 1, Figure 1, and Figure 2). Note that the minimum passing standard was 390 in 2013. Examinees meeting these criteria account for 73% of the examinees testing during this administration. The results seem contradictory in that initial certifiers pass (92.8%) at a higher rate than those maintaining their certification (89.5%), but this group has a higher mean score by approximately 18 points (Table 1). If only the pass rate is considered, one would conclude that the initial certifiers perform better on the examination and that the examination might be biased against those attempting to maintain certi-

The ABFM Begins to Use Differential Item Functioning

The American Board of Family Medicine (ABFM) believes that it is important to have evidence to show that the pass-fail decisions related to its examinations are based upon accurately determining the minimum knowledge necessary to be a board certified family physician, and furthermore, that these decisions are unbiased against any particular subset of the population. Accordingly, as part of the ABFM’s commitment to continuously improve the Maintenance of Certification for Family Physicians (MC-FP) process, the ABFM has started using differential item functioning (DIF) procedures to detect potentially biased items on its examinations. Although gender information has been collected for some time from examination applicants, we began collecting ethnicity data for applicants taking the MC-FP exam this past spring so that we could begin to conduct these analyses. DIF procedures are based upon the idea that a test item is biased if individuals from different subpopulations, who are of equal ability, do not have the same probability of answering it correctly.1,2 Although pass rates are an indicator of whether a particular subpopulation is performing at a level comparable to the other subpopulations, it is silent with regard to whether the meaning of the scores is stable across subpopulations. These differences could be due to bias in the items that would effectively destabilize the construct.3 By this we mean that the items, when ordered by their difficulty, form a linear construct of less to more. If some items are more difficult or less difficult relative to the other items for a specific subpopulation, then the construct represented by the test is degraded to the extent that the items are disordered for that subpopulation. On the other hand, the hierarchical construct represented by the test could be very stable and the difference in pass rates could be due to differences of socioeconomic status and the potential associated inequities inherent in the educational system. DIF analysis permits us to disentangle item level bias from differences in ability among subpopulations. The process of calibrating test questions with regard to their difficulty, both for samples from a subpopulation and from the overall population, is probabilistic. Therefore, this type of DIF study is best used as a screening tool to find biased items. It does not prove that the items are biased. The ABFM DIF process can be viewed as having 3 stages: (1) flagging potentially biased items, (2) examining the flagged questions’ content for sources of bias, and (3) determining their final disposition.

Frequency and Criticality of Diagnoses in Family Medicine Practices: From the National Ambulatory Medical Care Survey (NAMCS)

Background: Family medicine is a specialty of breadth, providing comprehensive health care for the individual and the family that integrates the broad scope of clinical, social, and behavioral sciences. As such, the scope of practice (SOP) for family medicine is extensive; however, over time many family physicians narrow their SOP. We sought to provide a nationally representative description of the most common and the most critical diagnoses that family physicians see in their practice. Methods: Data were extracted from the 2012 National Ambulatory Medical Care Survey (NAMCS) to select all ICD-9 codes reported by family physicians. A panel of family physicians then reviewed 1893 ICD-9 codes to place each code into an American Board of Family Medicine Family Medicine Certification Examination test plan specifications (TPS) category and provide a rating for an Index of Harm (IoH). Results: An analysis of all 1893 ICD-9 codes seen by family physicians in the 2012 NAMCS found that 198 ICD-9 codes could not be assigned a TPS category, leaving 1695 ICD-9 codes in the dataset. Top 10 lists of ICD-9 codes by TPS category were created for both frequency and IoH. Conclusions: This study provides a nationally representative description of the most common diagnoses that family physicians are seeing in their practice and the criticality of these diagnoses. These results provide insight into the domain of the specialty of family medicine. Medical educators may use these results to better tailor education and training to practice.

Performance of Graduating Residents on the American Board of Family Medicine Certification Examination 2009–2016

In response to growing concern about the declining performance on the American Board of Family Medicine Certification Examination, several strategies were employed to assist program directors with preparing their residents to take the examination. The effect of these efforts seems to have resulted in significant improvement in performance.

The Predictive Validity of the National Board of Osteopathic Medical Examiners' COMLEX-USA Examinations With Regard to Outcomes on American Board of Family Medicine Examinations.

Purpose To examine the predictive validity of the National Board of Osteopathic Medical Examiners’ Comprehensive Osteopathic Medical Licensing Examination of the United States of America (COMLEX-USA) series with regard to the American Board of Family Medicine’s (ABFM’s) In-Training Examination (ITE) and Maintenance of Certification for Family Physicians (MC-FP) Examination. Method A repeated-measures design was employed, using test scores across seven levels of training for 1,023 DOs who took the MC-FP for the first time between April 2012 and November 2014 and for whom the ABFM had ITE scores for each of their residency years. Pearson and disattenuated correlations were calculated; Fisher r to z transformation was performed; and sensitivity, specificity, and positive and negative predictive values for the COMLEX-USA Level 2–Cognitive Evaluation (CE) with regard to the MC-FP were computed. Results The Pearson and disattenuated correlations ranged from 0.55 to 0.69 and from 0.61 to 0.80, respectively. For MC-FP scores, only the correlation increase from the COMLEX-USA Level 2-CE to Level 3 was statistically significant (for Pearson correlations: z = 2.41, P = .008; for disattenuated correlations: z = 3.16, P < .001). The sensitivity, specificity, and positive and negative predictive values of the COMLEX-USA Level 2-CE with the MC-FP were 0.90, 0.39, 0.96, and 0.19, respectively. Conclusions Evidence was found that the COMLEX-USA can assist family medicine residency program directors in predicting later resident performance on the ABFM’s ITE and MC-FP, which is becoming increasingly important as graduate medical education accreditation moves toward a single aligned model.