Cheryl L. Damberg

What Is the Concordance Between the Medical Record and Patient Self-Report as Data Sources for Ambulatory Care?

Background:The validity of quality of care assessments relies upon data quality, yet little is known about the relative completeness and validity of data sources for evaluating the quality of care. Objectives:We evaluated concordance between ambulatory medical record and patient survey data. Levels of concordance, variations by type of item, sources of disagreement between data sources, and implications for quality of care assessment efforts are discussed. Design and Subjects:This was an observational study that included 1270 patients sampled from 39 West Coast medical organizations with at least 1 of the following: diabetes, ischemic heart disease, asthma or chronic obstructive pulmonary disease, or low back pain. Measures:Items from both data sources were grouped into 4 conceptual domains: diagnosis, clinical services delivered, counseling and referral, and medication use. We present total agreement, kappa, sensitivity, and specificity at the item and domain-levels and for all items combined. Results:We found good concordance between survey and medical records overall, but there was substantial variation within and across domains. The worst concordance was in the counseling and referrals domain, the best in the medication use domain. Patients were able to report with good sensitivity on memorable items. Conclusions:Quality ratings are likely to vary in differing directions, depending on the data source used. The most appropriate data source for analyses of components of and overall quality of care must be considered in light of study objectives and resources. We recommend data collection from multiple sources to most accurately portray the patient and provider experience of medical care.

Pay for performance in the hospital setting: what is the state of the evidence?

More than 40 private sector hospital pay-for-performance (P4P) programs now exist, and Congress is considering initiating a Medicare hospital P4P program. Given the growing interest in hospital P4P, this systematic review of the literature examines the current state of knowledge about the effect of P4P on clinical process measures, patient outcomes and experience, safety, and resource utilization. Little formal evaluation of hospital P4P has occurred, and most of the 8 published studies have methodological flaws. The most rigorous studies focus on clinical process measures and demonstrate that hospitals participating in the Centers for Medicare and Medicaid Services-Premier Hospital Quality Incentive Demonstration, a P4P program, had a 2- to 4-percentage point greater improvement than the improvement observed in control hospitals. There is a need for more systematic evaluation of hospital P4P to understand its effect and whether the benefits of investing in P4P outweigh the associated costs. (Am J Med Qual 2009;24:19-28)

Taking Stock Of Pay-For-Performance: A Candid Assessment From The Front Lines

Pay-for-performance (P4P) has been widely adopted, but it remains unclear how providers are responding and whether results are meeting expectations. Physician organizations involved in the California Integrated Healthcare Associations (IHA) P4P program reported having increased physician-level performance feedback and accountability, speeded up information technology adoption, and sharpened their organizational focus and support for improvement in response to P4P; however, after three years of investment, these changes had not translated into breakthrough quality improvements. Continued monitoring is required to determine whether early investments made by physician organizations provide a basis for greater improvements in the future.

Administrative versus clinical data for coronary artery bypass graft surgery report cards: the view from California.

Objective:The objective of this study was to compare the performance of a risk model for isolated coronary artery bypass graft (CABG) surgery based on administrative data with that of a clinical risk model in predicting mortality and identifying hospital performance outliers. Methods:Clinical data records from the California CABG Mortality Reporting Program for 38,230 isolated CABG patients undergoing surgery in 2000–2001 were linked to records in the California patient discharge data (PDD) abstract. Risk factors based on administrative data that mirrored clinical risk factors were developed using the condition present at admission indicator in the PDD to separate preoperative acute conditions from complications of surgery. Using logistic regression, risk model performance across data sources was compared along with hospital risk-adjusted mortality ranks and quality ratings. Results:The administrative data showed lower prevalence of risk factors when compared with the clinical data. The clinical risk model had somewhat better discrimination (C = 0.824) than the administrative model (C = 0.799). The clinical model yielded 17 outliers and the administrative model 16 with agreement on 12 hospitals’ status. Performance of the administrative risk model was minimally affected by removal of information from prior admissions and removal of risk factors not confirmed in the clinical record. Conclusions:Unique properties of the California administrative data, including the ability to distinguish acute preoperative risk factors from complications of surgery, permitted construction of an administrative risk model that predicts mortality on par with most published clinical models. Despite this, the administrative model identified slightly different hospital outliers, which may indicate somewhat biased assessments of hospital patient risk.

Episode-Based Performance Measurement And Payment: Making It A Reality

Proposals to use episodes of care as a basis for payment and performance measurement are largely conceptual at this stage, with little empirical work or experience in applied settings to guide their design. Based on analyses of Medicare data, we identified key issues that will need to be considered related to defining episodes and determining which provider is accountable for an episode. We suggest a number of applied studies and demonstrations that would facilitate more rapid movement of episode-based approaches from concept to implementation.

Contextual Issues Influencing Implementation and Outcomes Associated With an Integrated Approach to Managing Pain, Agitation, and Delirium in Adult ICUs

Objective:This pilot study was designed to identify which contextual factors facilitate/hinder the implementation of the awakening, breathing, coordination, delirium, and early mobility (ABCDE) bundle for guidance in future studies. Design:The sources of data for this study included document review, planned site visits (including interviews and observations), a brief online contextual factors survey, and self-reported process and outcome data. Patients:All patients in the four participating SF Bay Area ICUs were eligible to be included in this pilot study. Setting:This study took place in the four San Francisco Bay Area ICUs participating in the ICU Clinical Impact Interest Group, funded by the Gordon and Betty Moore Foundation from January 2012 through June 2013. Interventions:This was a pilot evaluation study to identify factors that facilitated/hindered the implementation of the ABCDE bundle, interventions designed to decrease the prevalence of ICU-acquired delirium and muscle weakness. The ABCDE bundle consists of spontaneous awakening trials, spontaneous breathing trials, coordination of awakening and breathing trials, choice of sedation, delirium screening and treatment, and early progressive mobility. Measurements:Process data related to bundle element compliance were collected at baseline and monthly during the intervention period. Outcome data (average ICU length of stay and average days on mechanical ventilation) were collected at baseline and quarterly during the intervention period. Hospital-specific results of the online contextual factors survey and information gathered through interviews and observations during site visits also contributed to the analysis. Main Results:Factors related to structural characteristics of the ICU, an organizational-wide patient safety culture, an ICU culture of quality improvement, implementation planning, training/support, and prompts/documentation are believed to have facilitated the rate and success of ABCDE bundle implementation. Excessive turnover (both in project and ICU leadership), staff morale issues, lack of respect among disciplines, knowledge deficits, and excessive use of registry staff are believed to have hindered implementation. Conclusions:Successful implementation of the elements of the ABCDE bundle can result in significant improvements in ICU patient care. The results of this study highlight specific structural and cultural elements of ICUs and hospitals that can positively and negatively influence the implementation of complex care bundles like the ABCDE bundle. Further research is needed to assess the influence of these contextual factors across a broader variety of ICUs and hospitals.

Paying for performance: implementing a statewide project in California.

The US health care system falls far short of providing care consistent with national standards of care and available knowledge.

Will financial incentives stimulate quality improvement? Reactions from frontline physicians.

Pay-for-performance is being applied at the physician level to stimulate improvements in quality of care and cost efficiency; however, little is known about how physicians will respond. We interviewed physicians exposed to a financial incentive program in California to identify possible barriers to the successful application of financial incentives by exploring physicians’ opinions of and experiences with pay-for-performance programs. Reasons physicians cited for quality deficiencies included insurance coverage limitations and lack of patient compliance, time, and proper physician oversight. Physicians believe that they play a significant role and have a moderate to high degree of control over quality of care and that it is important to self-monitor. Physicians expressed the need for accurate and timely data, peer comparisons, and more patient time, staff support, and consultations with colleagues to successfully monitor and deliver quality care. Many support increased pay for delivering high-quality care but question measurement accuracy, bonus payment financing, and health plan involvement.

Do Physician Organizations Located in Lower Socioeconomic Status Areas Score Lower on Pay-for-Performance Measures?

ABSTRACTBACKGROUNDPhysician organizations (POs)—independent practice associations and medical groups—located in lower socioeconomic status (SES) areas may score poorly in pay-for-performance (P4P) programs.OBJECTIVETo examine the association between PO location and P4P performance.DESIGNCross-sectional study; Integrated Healthcare Association’s (IHA’s) P4P Program, the largest non-governmental, multi-payer program for POs in the U.S.PARTICIPANTS160 POs participating in 2009.MAIN MEASURESWe measured PO SES using established methods that involved geo-coding 11,718 practice sites within 160 POs to their respective census tracts and weighting tract-specific SES according to the number of primary care physicians at each site. P4P performance was defined by IHA’s program and was a composite mainly representing clinical quality, but also including measures of patient experience, information technology and registry use.KEY RESULTSThe area-based PO SES measure ranged from −11 to +11 (mean 0, SD 5), and the IHA P4P performance score ranged from 23 to 86 (mean 69, SD 15). In bivariate analysis, there was a significant positive relationship between PO SES and P4P performance (p < 0.001). In multivariate analysis, a one standard deviation increase in PO SES was associated with a 44% increase (relative risk 1.44, 95%CI, 1.22-1.71) in the likelihood of a PO being ranked in the top two quintiles of performance (p < 0.001).CONCLUSIONSPhysician organizations’ performance scores in a major P4P program vary by the SES of the areas in which their practice sites are located. P4P programs that do not account for this are likely to pay higher bonuses to POs in higher SES areas, thus increasing the resource gap between these POs and POs in lower SES areas, which may increase disparities in the care they provide.

A Five-Point Checklist To Help Performance Reports Incentivize Improvement And Effectively Guide Patients

Public reports of provider performance on measures of the quality, costs, and outcomes of health care can spur improvement and help patients find the best providers. However, the likelihood that these benefits will materialize depends on the methods underlying each performance report. This paper presents a five-point methodological checklist to guide those who want to improve their performance reporting methods. The central goal is to help report makers minimize the frequency and severity of provider performance misclassification and avoid adverse unintended consequences of reporting. We believe that if those who produce the reports publicly explain how they address each checklist item, this increased transparency will encourage more rigorous methods and improve the chances that reports will lead to better, more efficient care.