Jenny Neuburger

Avoidability of hospital deaths and association with hospital-wide mortality ratios: retrospective case record review and regression analysis

Objectives To determine the proportion of avoidable deaths (due to acts of omission and commission) in acute hospital trusts in England and to determine the association with the trust’s hospital-wide standardised mortality ratio assessed using the two commonly used methods - the hospital standardised mortality ratio (HSMR) and the summary hospital level mortality indicator (SHMI). Design Retrospective case record review of deaths. Setting 34 English acute hospital trusts (10 in 2009 and 24 in 2012/13) randomly selected from across the spectrum of HSMR. Main outcome measures Avoidable death, defined as those with at least a 50% probability of avoidability in view of trained medical reviewers. Association of avoidable death proportion with the HSMR and the SHMI assessed using regression coefficients, to estimate the increase in avoidable death proportion for a one standard deviation increase in standardised mortality ratio. Participants 100 randomly selected hospital deaths from each trust. Results The proportion of avoidable deaths was 3.6% (95% confidence interval 3.0% to 4.3%). It was lower in 2012/13 (3.0%, 2.4% to 3.7%) than in 2009 (5.2%, 3.8% to 6.6%). This difference is subject to several factors, including reviewers’ greater awareness in 2012/13 of orders not to resuscitate, patients being perceived as sicker on admission, minor differences in review form questions, and cultural changes that might have discouraged reviewers from criticising other clinicians. There was a small but statistically non-significant association between HSMR and the proportion of avoidable deaths (regression coefficient 0.3, 95% confidence interval −0.2 to 0.7). The regression coefficient was similar for both time periods (0.1 and 0.3). This implies that a difference in HSMR of between 105 and 115 would be associated with an increase of only 0.3% (95% confidence interval −0.2% to 0.7%) in the proportion of avoidable deaths. A similar weak non-significant association was observed for SHMI (regression coefficient 0.3, 95% confidence interval −0.3 to 1.0). Conclusions The small proportion of deaths judged to be avoidable means that any metric based on mortality is unlikely to reflect the quality of a hospital. The lack of association between the proportion of avoidable deaths and hospital-wide SMRs partly reflects methodological shortcomings in both metrics. Instead, reviews of individual deaths should focus on identifying ways of improving the quality of care, whereas the use of standardised mortality ratios should be restricted to assessing the quality of care for conditions with high case fatality for which good quality clinical data exist.

The Impact of a National Clinician-led Audit Initiative on Care and Mortality after Hip Fracture in England: An External Evaluation using Time Trends in Non-audit Data

Background:Hip fracture is the most common serious injury of older people. The UK National Hip Fracture Database (NHFD) was launched in 2007 as a national collaborative, clinician-led audit initiative to improve the quality of hip fracture care, but has not yet been externally evaluated. Methods:We used routinely collected data on 471,590 older people (aged 60 years and older) admitted with a hip fracture to National Health Service (NHS) hospitals in England between 2003 and 2011. The main variables of interest were the use of early surgery (on day of admission, or day after) and mortality at 30 days from admission. We compared time trends in the periods 2003–2007 and 2007–2011 (before and after the launch of the NHFD), using Poisson regression models to adjust for demographic changes. Findings:The number of hospitals participating in the NHFD increased from 11 in 2007 to 175 in 2011. From 2007 to 2011, the rate of early surgery increased from 54.5% to 71.3%, whereas the rate had remained stable over the period 2003–2007. Thirty-day mortality fell from 10.9% to 8.5%, compared with a small reduction from 11.5% to 10.9% previously. The annual relative reduction in adjusted 30-day mortality was 1.8% per year in the period 2003–2007, compared with 7.6% per year over 2007–2011 (P<0.001 for the difference). Interpretation:The launch of a national clinician-led audit initiative was associated with substantial improvements in care and survival of older people with hip fracture in England.

Trends in the use of bilateral mastectomy in England from 2002 to 2011: retrospective analysis of hospital episode statistics

Objectives For healthy women at high risk of developing breast cancer, a bilateral mastectomy can reduce future risk. For women who already have unilateral breast cancer, removing the contralateral healthy breast is more difficult to justify. We examined trends in the number of women who had a bilateral mastectomy in England between 2002 and 2011. Design Retrospective cohort study using the Hospital Episode Statistics database. Setting NHS hospital trusts in England. Participants Women aged between 18 and 80 years who had a bilateral mastectomy (or a contralateral mastectomy within 24 months of unilateral mastectomy) with or without a diagnosis of breast cancer. Main outcome measures Number and incidence of women without breast cancer who had a bilateral mastectomy; number and proportion who had a bilateral mastectomy as their first breast cancer operation, and the proportion of those undergoing bilateral mastectomy who had immediate breast reconstruction. Results Among women without breast cancer, the number who had a bilateral mastectomy increased from 71 in 2002 to 255 in 2011 (annual incidence rate ratio 1.16, 95% CI 1.13 to 1.18). In women with breast cancer, the number rose from 529 to 931, an increase from 2% to 3.1% of first operations (OR for annual increase 1.07, 95% CI 1.05 to 1.08). Across both groups, rates of immediate breast reconstruction roughly doubled and reached 90% among women without breast cancer in 2011. Conclusions The number of women who had a bilateral mastectomy nearly doubled over the last decade, and more than tripled among women without breast cancer. This coincided with an increase in the use of immediate breast reconstruction.

Socioeconomic differences in patient-reported outcomes after a hip or knee replacement in the English National Health Service

BACKGROUND We investigated socioeconomic differences in patient-reported outcomes after a hip or knee replacement and the contribution of health differences beforehand. METHODS Our sample included 121 983 patients in England who had an operation in 2009-2011. Socioeconomic status was measured with quintiles of the ranking of areas by the English Index of Multiple Deprivation. Outcomes at 6 months were the Oxford hip or knee score (OHS or OKS) that measure pain and disability on a scale from 0 (worst) to 48 (best), and the percentage reporting no improvement in problems. Adjustment was made for age, sex, ethnicity, comorbidity, general health, revision surgery, primary diagnosis, preoperative OHS or OKS and having longstanding problems. RESULTS Comparing the most- with the least-deprived group, the mean OHS was 5.0 points lower and the OKS 5.4 lower. Adjusted differences, reflecting the differences in improvement in the condition, were 2.8 [95% confidence interval (CI): 2.5-3.0] on OHS and 2.4 (95% CI: 2.2-2.7) on OKS. Adjusted odds ratios for reporting no improvement were 1.4 (1.2-1.6) for the hip and 1.4 (1.3-1.5) for the knee. CONCLUSIONS On average, patients living in socioeconomically deprived areas had worse outcomes after surgery, partly related to preoperative differences in health and disease severity and partly to less postoperative improvement.

Impact on hospital performance of introducing routine patient reported outcome measures in surgery.

Objectives To determine the impact of introducing patient reported outcome measures (PROMs) on the selection of patients for surgery and on outcome for four elective operations (hip replacement, knee replacement, varicose vein surgery and groin hernia repair). Methods Patient-level data from the National PROMs programme for England from April 2009 to March 2012 were analysed to determine the extent of change in mean preoperative scores and mean adjusted postoperative scores using disease-specific and generic PROMs assessed using standardized effect sizes (SESs). Variation between providers was determined by intra-class correlation coefficients for each time period. Consistency in outlier ratings was tested using weighted Kappa statistics. Results There was little apparent impact. Preoperative severity increased slightly for two procedures only: varicose vein surgery (SES disease-specific PROM 0.10; generic PROM −0.07) and to a lesser extent for hip replacement (SES disease-specific PROM −0.03). There was little inter-provider variation and it did not change significantly over time. There were also slight improvements in outcomes for hip and knee replacement (SES for disease-specific and generic PROMs 0.03) though not for hernia repair and a slight worsening for varicose vein surgery. The extent of variation in performance between providers was unchanged. The proportion of providers deemed to be outliers did not change over time. There was only moderate consistency in those providers deemed to be outliers for hip and knee replacement (Kappa 0.31–0.47) and it was even weaker for the other two procedures. Although 35% of providers of hip replacement were outliers in at least one year, only 6% were consistently outliers. Such inconsistency may be partly due to regression to the mean. Conclusions The minimal impact that the routine use and feedback of PROMs had on provider behaviour during the initial years suggests that more attention needs to be paid to how results are communicated and to the provision of advice as to what action may be taken.

Clinicians' and patients' views of metrics of change derived from patient reported outcome measures (PROMs) for comparing providers' performance of surgery

BackgroundPatient reported outcome measures (PROMs) are increasingly being used to compare the performance of health care providers. Our objectives were to determine the relative frequency of use of different metrics that can be derived from PROMs, explore clinicians’ and patients’ views of the options available, and make recommendations.MethodsFirst a rapid review of the literature on metrics derived from two generic (EQ-5D and EQ-VAS) and three disease-specific (Oxford Hip Score; Oxford Knee Score; Aberdeen Varicose Vein Questionnaire) PROMs was conducted. Next, the findings of the literature review were mapped onto our typology of metrics to determine their relative frequency of use, Finally, seven group meetings with surgical clinicians (n = 107) and six focus groups with patients (n = 45) were held which were audio-taped, transcribed and analysed thematically.ResultsOnly nine studies (9.3% of included papers) used metrics for comparing providers. These were derived from using either the follow-up PROM score (n = 3) or the change in score as an outcome (n = 5), both adjusted for pre-intervention score. There were no recorded uses of the proportion reaching a specified (‘good’) threshold and only two studies used the proportion reaching a minimally important difference (MID).Surgical clinicians wanted multiple outcomes, with most support expressed for the mean change in score, perceiving it to be more interpretable; there was also some support for the MID. For patients it was apparent that rather than the science behind these measures, the most important aspects were the use of language that would make the metrics personally meaningful and linking the metric to a familiar scale.ConclusionsFor clinicians the recommended metrics are the mean change in score and the proportion achieving a MID, both adjusted for pre-intervention score. Both need to be clearly described and explained. For patients we recommend the proportion achieving a MID or proportion achieving a significant improvement in hip function, both adjusted for pre-intervention score.

Using Patient-reported Outcomes (PROs) to Compare the Providers of Surgery: Does the Choice of Measure Matter?

Background:Patient-reported outcomes (PROs) are being used to compare health care providers with little knowledge of how the choice of measure affects such comparisons. Objectives:To assess how much difference the choice of PRO makes to a provider’s adjusted outcome and whether the choice affects a provider’s rating. Research Design:PROs collected in England from patients undergoing: hip replacement (243 providers; 52,692 patients); knee replacement (244; 60,118); varicose vein surgery (100; 11,163); and groin hernia repair (201; 31,714). Four case-mix–adjusted outcomes (mean postoperative disease-specific and generic PRO; proportion achieving a minimally important difference in disease-specific PRO; proportion reporting improvement on single transitional item). We calculated the associations between measures and for each measure, the proportion of providers rated as statistically above or below average and the level of agreement in ratings. Results:For major surgery, disease-specific PROs were strongly correlated with the generic PRO (hip 0.90; knee 0.88), they rated high proportions of providers as above or below average (hip 25.1%; knee 19.3%) and there was agreement in ratings with the generic PRO. Even so, for a large proportion of providers (hip 30%; knee 16%) their rating depended on the choice of measure. For minor surgery, correlations between measures were mostly weak. The single transitional item identified the most outliers (varicose vein 20%, hernia 10%). Conclusions:Choice of outcome measure can determine a provider’s rating. Measure selection depends on whether the priority is to avoid missing “poor” providers or avoid mislabeling average providers as “poor.”

Sociodemographic differences in the severity and duration of disease amongst patients undergoing hip or knee replacement surgery

BACKGROUND Differences in the use of hip and knee replacement by sex, age, ethnicity or socioeconomic status may lead to differences in disease severity between those who have surgery. METHODS Analyses used data collected from 117,736 patients in 2009-10 via the Patient Reported Outcome Measures (PROMs) programme in England. Adjusted differences were estimated in the Oxford Hip Score (OHS) or the Oxford Knee Score (OKS), both expressed on a scale from 0 to 48, and the proportion with longstanding problems (>5 years), expressed as odds ratios (ORs). RESULTS Women had more severe pain and disability than men on average (difference OHS 2.3 and OKS 3.3), but less often longstanding problems. Compared with white patients, average severity was higher in South Asian patients (difference OHS 2.7 and OKS 3.0) and in black patients (difference OHS 0.9 and OKS 1.6), who also more often had longstanding problems (OR 1.40 for hip and 1.54 for knee). Patients from deprived areas had more severe disease (difference OHS 3.6 and OKS 3.3 between least and most deprived quintile). CONCLUSIONS There is evidence that non-white and deprived patients tend to have hip and knee replacement surgery at a later stage in the course of their disease.

Funnel plots for comparing provider performance based on patient-reported outcome measures

Background Patient-reported outcome measures (PROMs) often produce skewed distributions of individual scores after a healthcare intervention. For health performance indicators derived from skewed distributions, funnel plots designed with symmetric control limits may increase the risk of false alarms about poor performance. Aim To investigate the accuracy of funnel plots with symmetric control limits when comparing provider performance based on PROMs. Methods The authors used a database containing condition-specific PROMs for 17 453 hip replacements and 7656 varicose vein procedures performed by providers in the English NHS. The mean postoperative PROM score, adjusted for patient characteristics, was used as the measure of performance. To compare performance, symmetric 99.8% control limits were calculated on funnel plots, 3 SDs away from the overall mean on either side. These were compared to control limits derived directly from percentiles of simulated (bootstrap) distributions of mean scores. Results The simulated control limits on funnel plots for both procedures were asymmetric. The empirical probability of falling outside the symmetric 99.8% ‘poor performance’ control limit was inflated from the stipulated rate of 0.1% to 0.2–0.3% for provider sample sizes of up to 150 procedures. The authors observed that, out of 237 providers of hip replacement, eight had adjusted mean scores that exceeded the symmetric ‘poor performance’ limit compared with only five that exceeded the corresponding simulated limit. In other words, three (1.3%) were differently classified. For varicose vein surgery, five out of 160 providers exceeded the symmetric limit and four exceeded the simulated limit, that is, 1 (0.6%) was differently classified. Conclusions When designing funnel plots for comparisons of provider performance based on highly skewed data, the use of simulated control limits should be considered.

How is feedback from national clinical audits used? Views from English National Health Service trust audit leads

Objective To explore how the output of national clinical audits in England is used by professionals and whether and how their impact could be enhanced. Methods A mixed-methods study with the primary recipients of four national clinical audits of cancer care of 607 local audit leads, 274 (45%) completed a questionnaire and 32 participated in an interview. Our questions focused on how the audits were used and whether barriers existed to using the audits for local service improvement. We described variation in questionnaire responses between the audits using chi-squared tests. Results are reported as percentages with their 95% confidence intervals. Qualitative data were analysed using Framework analysis. Results More than 90% of survey respondents believed that the audit findings were relevant to their clinical work, and interviewees described how they used the audits for a range of purposes. Forty-two percent of survey respondents said they had changed their clinical practice, and 56% had implemented service improvements in response to the audits. The degree of change differed between the four audits, evident in both the questionnaire and the interview data. In the interviews, two recurring barriers emerged: (1) the importance of data quality, which, in turn, influenced the perceived relevance and validity of the audit data and therefore the ability to make changes based on it and (2) the need for clear presentation of benchmarked local performance data. The perceived authority and credibility of the professional bodies supporting the audits was a key factor underpinning the use of the audit findings. Conclusion National cancer audit and feedback is used to improve services, but their impact could be enhanced by improving the data quality and relevance of feedback.