Weifeng Weng

The comprehensive care project: measuring physician performance in ambulatory practice.

OBJECTIVE To investigate the feasibility, reliability, and validity of comprehensively assessing physician-level performance in ambulatory practice. DATA SOURCES/STUDY SETTING Ambulatory-based general internists in 13 states participated in the assessment. STUDY DESIGN We assessed physician-level performance, adjusted for patient factors, on 46 individual measures, an overall composite measure, and composite measures for chronic, acute, and preventive care. Between- versus within-physician variation was quantified by intraclass correlation coefficients (ICC). External validity was assessed by correlating performance on a certification exam. DATA COLLECTION/EXTRACTION METHODS Medical records for 236 physicians were audited for seven chronic and four acute care conditions, and six age- and gender-appropriate preventive services. PRINCIPAL FINDINGS Performance on the individual and composite measures varied substantially within (range 5-86 percent compliance on 46 measures) and between physicians (ICC range 0.12-0.88). Reliabilities for the composite measures were robust: 0.88 for chronic care and 0.87 for preventive services. Higher certification exam scores were associated with better performance on the overall (r = 0.19; p<.01), chronic care (r = 0.14, p = .04), and preventive services composites (r = 0.17, p = .01). CONCLUSIONS Our results suggest that reliable and valid comprehensive assessment of the quality of chronic and preventive care can be achieved by creating composite measures and by sampling feasible numbers of patients for each condition.

Setting a Fair Performance Standard for Physicians’ Quality of Patient Care

BackgroundAssessing physicians’ clinical performance using statistically sound, evidence-based measures is challenging. Little research has focused on methodological approaches to setting performance standards to which physicians are being held accountable.ObjectiveDetermine if a rigorous approach for setting an objective, credible standard of minimally-acceptable performance could be used for practicing physicians caring for diabetic patients.DesignRetrospective cohort study.ParticipantsNine hundred and fifty-seven physicians from the United States with time-limited certification in internal medicine or a subspecialty.Main MeasuresThe ABIM Diabetes Practice Improvement Module was used to collect data on ten clinical and two patient experience measures. A panel of eight internists/subspecialists representing essential perspectives of clinical practice applied an adaptation of the Angoff method to judge how physicians who provide minimally-acceptable care would perform on individual measures to establish performance thresholds. Panelists then rated each measure’s relative importance and the Dunn–Rankin method was applied to establish scoring weights for the composite measure. Physician characteristics were used to support the standard-setting outcome.Key ResultsPhysicians abstracted 20,131 patient charts and 18,974 patient surveys were completed. The panel established reasonable performance thresholds and importance weights, yielding a standard of 48.51 (out of 100 possible points) on the composite measure with high classification accuracy (0.98). The 38 (4%) outlier physicians who did not meet the standard had lower ratings of overall clinical competence and professional behavior/attitude from former residency program directors (p = 0.01 and p = 0.006, respectively), lower Internal Medicine certification and maintenance of certification examination scores (p = 0.005 and p < 0.001, respectively), and primarily worked as solo practitioners (p = 0.02).ConclusionsThe standard-setting method yielded a credible, defensible performance standard for diabetes care based on informed judgment that resulted in a reasonable, reproducible outcome. Our method represents one approach to identifying outlier physicians for intervention to protect patients.

An Assessment of Patient‐Based and Practice Infrastructure–Based Measures of the Patient‐Centered Medical Home: Do We Need to Ask the Patient?

OBJECTIVE To examine the importance of patient-based measures and practice infrastructure measures of the patient-centered medical home (PCMH). DATA SOURCES A total of 3,671 patient surveys of 202 physicians completing the American Board of Internal Medicine (ABIM) 2006 Comprehensive Care Practice Improvement Module and 14,457 patient chart reviews from 592 physicians completing ABIMs 2007 Diabetes and Hypertension Practice Improvement Module. METHODOLOGY We estimated the association of patient-centered care and practice infrastructure measures with patient rating of physician quality. We then estimated the association of practice infrastructure and patient rating of care quality with blood pressure (BP) control. RESULTS Patient-centered care measures dominated practice infrastructure as predictors of patient rating of physician quality. Having all patient-centered care measures in place versus none was associated with an absolute 75.2 percent increase in the likelihood of receiving a top rating. Both patient rating of care quality and practice infrastructure predicted BP control. Receiving a rating of excellent on care quality from all patients was associated with an absolute 4.2 percent improvement in BP control. For reaching the maximum practice-infrastructure score, this figure was 4.5 percent. CONCLUSION Assessment of physician practices for PCMH qualification should consider both patient based patient-centered care measures and practice infrastructure measures.

A three-part model for measuring diabetes care in physician practice.

Background To assess the psychometric properties of the three components of the Diabetes Practice Improvement Module, to compare reliabilities of composites to individual measures, and to identify associations among practice-based and patient-based measures. Method Data include practice systems surveys of 626 physicians, 13,965 chart audits, and 12,927 patient surveys. Quality composites were identified using factor analysis. Means with reliabilities (intraclass correlation coefficient [ICC] and Cronbach’s α) are reported. Associations among patient-based quality measures and practice measures with case-mix adjustments were estimated via hierarchical models. Results Composite ICCs range from 0.11 to 0.54, and single items range from 0.05 to 0.49. Staff communication, efficiency, care access, and patient knowledge correlate with patient satisfaction (P < .001). Clinical outcomes are associated with clinical processes (e.g., annual foot exam) and appropriate treatment (P < .001). Patient adjusters (e.g., overall health or factors limiting self-care) are important for the models; physician characteristics used (e.g., age, practice size) seem less important. Conclusions Composites require smaller patient sample sizes and result in more reliable measures than do individual items. Additionally, the data show meaningful relationships between composites; physician-directed components (i.e., clinical processes and treatments) are related to clinical outcomes, and patients are clearly more satisfied with care if it is easily accessible and if communication about care is good.

Measuring physicians' performance in clinical practice: reliability, classification accuracy, and validity.

Much research has been devoted to addressing challenges in achieving reliable assessments of physicians’ clinical performance but less work has focused on whether valid and accurate classification decisions are feasible. This study used 957 physicians certified in internal medicine (IM) or a subspecialty, who completed the American Board of Internal Medicine (ABIM) Diabetes Practice Improvement Module (PIM). Ten clinical and two patient-experience measures were aggregated into a composite measure. The composite measure score was highly reliable (r = .91) and classification accuracy was high across the entire score scale (>0.90), which indicated that it is possible to differentiate high-performing and low-performing physicians. Physicians certified in endocrinology and those who scored higher on their IM certification examination had higher composite scores, providing some validity evidence. In summary, it is feasible to create a psychometrically robust composite measure of physicians’ clinical performance, specifically for the quality of care they provide to patients with diabetes.

Gaps In Quality Of Diabetes Care In Internal Medicine Residency Clinics Suggest The Need For Better Ambulatory Care Training

To ensure that medical residents will be prepared to deliver consistently high-quality care, they should be trained in settings that provide such care. Residents in internal medicine, particularly, need to learn good care habits in order to meet the needs of patients with diabetes and other common chronic and high-impact illnesses. To assess the strength of such training, we compared the quality of medical care provided in sixty-seven US internal medicine residency ambulatory clinics with the quality of care provided by 703 practicing general internists. We found significant quality gaps in process, intermediate outcome, and patient-experience measures. These inadequacies in ambulatory training for internal medicine residents must be addressed by policy makers and educators-for example, by accelerating the movement toward new residency curricula that emphasize competency-based training.