Matt Sutton

Using natural experiments to evaluate population health interventions: new Medical Research Council guidance

Natural experimental studies are often recommended as a way of understanding the health impact of policies and other large scale interventions. Although they have certain advantages over planned experiments, and may be the only option when it is impossible to manipulate exposure to the intervention, natural experimental studies are more susceptible to bias. This paper introduces new guidance from the Medical Research Council to help researchers and users, funders and publishers of research evidence make the best use of natural experimental approaches to evaluating population health interventions. The guidance emphasises that natural experiments can provide convincing evidence of impact even when effects are small or take time to appear. However, a good understanding is needed of the process determining exposure to the intervention, and careful choice and combination of methods, testing of assumptions and transparent reporting is vital. More could be learnt from natural experiments in future as experience of promising but lesser used methods accumulates.

Reduced Mortality with Hospital Pay for Performance in England

BACKGROUND Pay-for-performance programs are being adopted internationally despite little evidence that they improve patient outcomes. In 2008, a program called Advancing Quality, based on the Hospital Quality Incentive Demonstration in the United States, was introduced in all National Health Service (NHS) hospitals in the northwest region of England (population, 6.8 million). METHODS We analyzed 30-day in-hospital mortality among 134,435 patients admitted for pneumonia, heart failure, or acute myocardial infarction to 24 hospitals covered by the pay-for-performance program. We used difference-in-differences regression analysis to compare mortality 18 months before and 18 months after the introduction of the program with mortality in two comparators: 722,139 patients admitted for the same three conditions to the 132 other hospitals in England and 241,009 patients admitted for six other conditions to both groups of hospitals. RESULTS Risk-adjusted, absolute mortality for the conditions included in the pay-for-performance program decreased significantly, with an absolute reduction of 1.3 percentage points (95% confidence interval [CI], 0.4 to 2.1; P=0.006) and a relative reduction of 6%, equivalent to 890 fewer deaths (95% CI, 260 to 1500) during the 18-month period. The largest reduction, for pneumonia, was significant (1.9 percentage points; 95% CI, 0.9 to 3.0; P<0.001), with nonsignificant reductions for acute myocardial infarction (0.6 percentage points; 95% CI, -0.4 to 1.7; P=0.23) and heart failure (0.6 percentage points; 95% CI, -0.6 to 1.8; P=0.30). CONCLUSIONS The introduction of pay for performance in all NHS hospitals in one region of England was associated with a clinically significant reduction in mortality. As compared with a similar U.S. program, the U.K. program had larger bonuses and a greater investment by hospitals in quality-improvement activities. Further research is needed on how implementation of pay-for-performance programs influences their effects. (Funded by the NHS National Institute for Health Research.).

Has payment by results affected the way that English hospitals provide care? Difference-in-differences analysis

Objective To examine whether the introduction of payment by results (a fixed tariff case mix based payment system) was associated with changes in key outcome variables measuring volume, cost, and quality of care between 2003/4 and 2005/6. Setting Acute care hospitals in England. Design Difference-in-differences analysis (using a control group created from trusts in England and providers in Scotland not implementing payment by results in the relevant years); retrospective analysis of patient level secondary data with fixed effects models. Data sources English hospital episode statistics and Scottish morbidity records for 2002/3 to 2005/6. Main outcome measures Changes in length of stay and proportion of day case admissions as a proxy for unit cost; growth in number of spells to measure increases in output; and changes in in-hospital mortality, 30 day post-surgical mortality, and emergency readmission after treatment for hip fracture as measures of impact on quality of care. Results Length of stay fell more quickly and the proportion of day cases increased more quickly where payment by results was implemented, suggesting a reduction in the unit costs of care associated with payment by results. Some evidence of an association between the introduction of payment by results and growth in acute hospital activity was found. Little measurable change occurred in the quality of care indicators used in this study that can be attributed to the introduction of payment by results. Conclusion Reductions in unit costs may have been achieved without detrimental impact on the quality of care, at least in as far as these are measured by the proxy variables used in this study.

Doctor Behaviour under a Pay for Performance Contract: Treating, Cheating and Case Finding?

The UK National Health Service introduced a pay for performance scheme for primary care providers in 2004/5. The scheme rewarded providers for the proportion of eligible patients who received appropriate treatment. Eligible patients were those who had been reported by the provider as having the relevant disease minus those they exception reported as not suitable for treatment. Using rich provider level data, we find that differences in reported disease rates between providers, and differences in exception rates both between and within providers, suggest gaming. Faced with ratio performance indicators, providers acted on denominators as well as numerators.

Deprivation and quality of primary care services: evidence for persistence of the inverse care law from the UK Quality and Outcomes Framework

Objective: To examine whether the quality of primary care measured by the 2004 contract varies with socioeconomic deprivation. Design: Retrospective analysis of publicly available data, comparing quality indicators used for payment that allow exclusion of patients (payment quality) and indicators based on the care delivered to all patients (delivered quality). Setting and participants: 1024 general practices in Scotland. Main outcome measures: Regression coefficients summarising the relationships between deprivation and payment and delivered quality. Results: Little systematic association is found between payment quality and deprivation but, for 17 of the 33 indicators examined, delivered quality falls with increasing deprivation. Absolute differences in delivered quality are small for most simpler process measures, such as recording of smoking status or blood pressure. Greater inequalities are seen for more complex process measures such as diagnostic procedures, some intermediate outcome measures such as glycaemic control in diabetes and measures of treatment such as influenza immunisation. Conclusions: The exclusions system succeeds in not penalising practices financially for the characteristics of the population they serve, but does not reward the additional work required in deprived areas and contributes to a continuation of the inverse care law. The contract data collected prevent examination of most complex process or treatment measures and this analysis is likely to underestimate the extent of continuing inequalities in care. Broader lessons cannot be drawn on the effect on inequalities of this new set of incentives until changes are made to the way contract data are collected and analysed.

Effect of a national primary care pay for performance scheme on emergency hospital admissions for ambulatory care sensitive conditions: controlled longitudinal study

Objective To estimate the impact of a national primary care pay for performance scheme, the Quality and Outcomes Framework in England, on emergency hospital admissions for ambulatory care sensitive conditions (ACSCs). Design Controlled longitudinal study. Setting English National Health Service between 1998/99 and 2010/11. Participants Populations registered with each of 6975 family practices in England. Main outcome measures Year specific differences between trend adjusted emergency hospital admission rates for incentivised ACSCs before and after the introduction of the Quality and Outcomes Framework scheme and two comparators: non-incentivised ACSCs and non-ACSCs. Results Incentivised ACSC admissions showed a relative reduction of 2.7% (95% confidence interval 1.6% to 3.8%) in the first year of the Quality and Outcomes Framework compared with ACSCs that were not incentivised. This increased to a relative reduction of 8.0% (6.9% to 9.1%) in 2010/11. Compared with conditions that are not regarded as being influenced by the quality of ambulatory care (non-ACSCs), incentivised ACSCs also showed a relative reduction in rates of emergency admissions of 2.8% (2.0% to 3.6%) in the first year increasing to 10.9% (10.1% to 11.7%) by 2010/11. Conclusions The introduction of a major national pay for performance scheme for primary care in England was associated with a decrease in emergency admissions for incentivised conditions compared with conditions that were not incentivised. Contemporaneous health service changes seem unlikely to have caused the sharp change in the trajectory of incentivised ACSC admissions immediately after the introduction of the Quality and Outcomes Framework. The decrease seems larger than would be expected from the changes in the process measures that were incentivised, suggesting that the pay for performance scheme may have had impacts on quality of care beyond the directly incentivised activities.

Record rewards: the effects of targeted quality incentives on the recording of risk factors by primary care providers

Financial incentives may increase performance on targeted activities and have unintended consequences for untargeted activities. An innovative pay-for-performance scheme was introduced for UK general practices in 2004. It incentivised particular quality indicators for targeted groups of patients. We estimate the intended and unintended consequences of this Quality and Outcomes Framework (QOF) using dynamic panel probit models estimated on individual patient records from 315 general practices over the period 2000/1-2005/6. We focus on annual rates of recording of blood pressure, smoking status, cholesterol, body mass index and alcohol consumption. The recording of each risk factor is designated as incentivised or unincentivised for each individual based on whether they have one of the diseases targeted by the QOF. The effect on incentivised factors was substantially larger on the targeted patient groups (+19.9 percentage points) than on the untargeted groups (+5.3 percentage points). There was no obvious evidence of effort diversion but there was evidence of substantial positive spillovers (+10.9 percentage points) onto unincentivised factors for the targeted groups. Moreover, provider responses were larger on those indicators for which more stringent standards were set and greater rewards offered. We conclude that the incentives induced providers to improve targeted quality and make investments in quality that extended beyond the scheme. We estimate that the average provider was paid pound20 500 for recording 410 additional items of information on the risk factors targeted by the financial incentives. Allowance for the positive spillovers reduces the estimated average reward from pound50 to pound25 per additional record.

Determinants of primary medical care quality measured under the new UK contract: cross sectional study

Abstract Objective To identify factors associated with the quality of primary medical care incentivised under the new UK general medical services contract. Design Cross sectional study. Setting NHS Ayrshire and Arran area, Scotland. Participants 60 general practices. Main outcome measures Quality scores reflecting the total points achieved on the 10 clinical domains and holistic care. Univariate and multivariate regression analyses were used to relate quality scores to measures of population characteristics, urban-rural location, general practitioner characteristics, clinical team size and composition, practice characteristics, and income from other sources. Results Deprivation was associated with higher scores. Quality scores increased with the size of the clinical team. Practices with higher income from other sources had lower quality scores. Practices that were accredited, had training status, or contained younger general practitioners had higher quality scores, but these effects were explained by other associated factors. 53% of the variation in quality scores was explained by a multivariate model, which included measures of deprivation, clinical team size and composition, and financial incentives. Conclusions Population characteristics showed little association with the quality of primary medical care incentivised under the UK general medical services contract. Larger clinical teams delivered higher quality clinical care, but the nurse-doctor composition of the clinical team did not influence quality. Practices that were more likely to respond to financial incentives because of previous behaviour or lower income from other sources recorded higher quality. If generalisable, the results suggest that initiatives to improve primary medical care quality should focus on the structure and resourcing of providers.

Income related inequalities in self assessed health in Britain: 1979–1995

Study objective: To measure and decompose income related inequalities in self assessed health in England, Scotland, and Wales, 1979–1995. Design: The relation between individual health and a non-linear transformation of equivalised income, allowing for sex, age, country, and year effects, was estimated by multiple regression. The share of health attributable to transformed income and the Gini coefficient for transformed income were calculated. Inequality in health was measured by the partial concentration index, which is the product of the Gini coefficient and the share of health attributable to transformed income. Participants and setting: Representative annual samples of the adult population living in private households in Great Britain 1979–1995. The total analysed sample was 299 968 people. Main results: Pro-rich health inequality was largest in Wales and smallest in England over the period because the effect of increased income on health was greatest in Wales and least in England. In all three countries, pro-rich health inequality increased throughout the period. In the early 1980s this was primarily attributable to increases in income inequality. Thereafter the increased share of health attributable to income was the principal cause. Conclusions: Reductions in pro-rich health inequality can be achieved by reducing income inequality, reducing the effect of income on health, or both.

Long-Term Effect of Hospital Pay for Performance on Mortality in England

BACKGROUND A pay-for-performance program based on the Hospital Quality Incentive Demonstration was introduced in all hospitals in the northwest region of England in 2008 and was associated with a short-term (18-month) reduction in mortality. We analyzed the long-term effects of this program, called Advancing Quality. METHODS We analyzed 30-day in-hospital mortality among 1,825,518 hospital admissions for eight conditions, three of which were covered by the financial-incentive program. The hospitals studied included the 24 hospitals in the northwest region that were participating in the program and 137 elsewhere in England that were not participating. We used difference-in-differences regression analysis to compare risk-adjusted mortality for an 18-month period before the program was introduced with subsequent mortality in the short term (the first 18 months of the program) and the longer term (the next 24 months). RESULTS Throughout the short-term and the long-term periods, the performance of hospitals in the incentive program continued to improve and mortality for the three conditions covered by the program continued to fall. However, the reduction in mortality among patients with these conditions was greater in the control hospitals (those not participating in the program) than in the hospitals that were participating in the program (by 0.7 percentage points; 95% confidence interval [CI], 0.3 to 1.2). By the end of the 42-month follow-up period, the reduced mortality in the participating hospitals was no longer significant (-0.1 percentage points; 95% CI, -0.6 to 0.3). From the short term to the longer term, the mortality for conditions not covered by the program fell more in the participating hospitals than in the control hospitals (by 1.2 percentage points; 95% CI, 0.4 to 2.0), raising the possibility of a positive spillover effect on care for conditions not covered by the program. CONCLUSIONS Short-term relative reductions in mortality for conditions linked to financial incentives in hospitals participating in a pay-for-performance program in England were not maintained.