Rowena Jacobs

Alternative methods to examine hospital efficiency: Data Envelopment Analysis and Stochastic Frontier Analysis

There has been increasing interest in the ability of different methods to rank efficient hospitals over their inefficient counterparts. The UK Department of Health has used three cost indices to benchmark NHS hospitals (Trusts). This study uses the same dataset and compares the efficiency rankings from the cost indices with those obtained using Data Envelopment Analysis (DEA) and Stochastic Frontier Analysis (SFA). The paper concludes that the methods each have particular strengths and weaknesses and potentially measure different aspects of efficiency. Several specifications should be used to develop ranges of inefficiency to act as signalling devices rather than point estimates. It is argued that differences in efficiency scores across different methods may be due to random “noise” and reflect data deficiencies. The conclusions concur with previous findings that there are not truly large efficiency differences between Trusts and savings from bringing up poorer performers would in fact be quite modest.

Measuring Efficiency in Health Care: Analytic Techniques and Health Policy

With the healthcare sector accounting for a sizeable proportion of national expenditures, the pursuit of efficiency has become a central objective of policymakers within most health systems. However, the analysis and measurement of efficiency is a complex undertaking, not least due to the multiple objectives of health care organizations and the many gaps in information systems. In response to this complexity, research in organizational efficiency analysis has flourished. This 2006 book examines some of the most important techniques currently available to measure the efficiency of systems and organizations, including data envelopment analysis and stochastic frontier analysis, and also presents some promising new methodological approaches. Such techniques offer the prospect of many new and fruitful insights into health care performance. Nevertheless, they also pose many practical and methodological challenges. This is an important critical assessment of the strengths and limitations of efficiency analysis applied to health and health care.

Exploring the relationship between senior management team culture and hospital performance

The purpose of this study was to explore relationships between senior management team culture and organizational performance in English hospital organizations (NHS trusts [National Health Service]). We used an established culture-rating instrument, the Competing Values Framework, to assess senior management team culture. Organizational performance was assessed using a wide variety of routinely collected measures. Data were gathered from all English NHS acute hospital trusts, a total of 197 organizations. Multivariate econometric analyses were used to explore the associations between measures of culture and measures of performance using regressions, ANOVA, multinomial logit, and ordered probit. Organizational culture varied across hospital organizations, and at least some of this variation was associated in consistent and predictable ways with a variety of organizational characteristics and measures of performance. The findings provide particular support for a contingent relationship between culture and performance.

The relationship between organizational culture and performance in acute hospitals.

This paper examines the relationship between senior management team culture and organizational performance in English acute hospitals (NHS Trusts) over three time periods between 2001/2002 and 2007/2008. We use a validated culture rating instrument, the Competing Values Framework, to measure senior management team culture. Organizational performance is assessed using a wide range of routinely collected indicators. We examine the associations between organizational culture and performance using ordered probit and multinomial logit models. We find that organizational culture varies across hospitals and over time, and this variation is at least in part associated in consistent and predictable ways with a variety of organizational characteristics and routine measures of performance. Moreover, hospitals are moving towards more competitive culture archetypes which mirror the current policy context, though with a stronger blend of cultures. The study provides evidence for a relationship between culture and performance in hospital settings.

How robust are hospital ranks based on composite performance measures

Background:Composite indices of healthcare performance are an aggregation of underlying individual performance measures. They are increasingly being used to rank healthcare organizations. Although composite indicators present the “big picture” in a way that is easy to interpret, misleading conclusions may be drawn if attention is not paid to key methodological issues in their construction. Objectives:We examine variability in performance measures in the context of the construction and use of composite measures. We illustrate how variability in the underlying data and the resulting composite may undermine the robustness of performance measures in health care. We also illustrate how variation in the methodological rules applied to aggregate the individual indicators can have an important impact on composite scores. Methods:We use data for 117 English acute hospitals to illustrate the generic methodological issues. The variance in performance measures is partitioned into “controllable” and “uncontrollable” elements. We create a composite index from the underlying performance indicators and use Monte Carlo simulations to examine the robustness of the composite. Results:Random variation beyond the control of organizations gives rise to considerable uncertainty in hospital scores. Composites are also sensitive to changes made to the weighting system and to the aggregation rules. Some hospitals can jump almost half of the league table as a result of subtle changes. Conclusions:Great care is warranted in interpreting the results of composite performance measures. Suggestions for their future development are made.

How Do Performance Indicators Add Up? An Examination of Composite Indicators in Public Services

Composite indicators are an aggregation of underlying performance indicators into a single index and have been used widely in the public sector to create league tables. This article investigates the degree to which composite measures are an appropriate metric for measuring performance. The authors illustrate the degree of uncertainty in the construction of composites and how rankings are sensitive to the way in which the performance indicators are aggregated. The article highlights the issues which need to be considered in the development and use of composite indicators for performance management purposes.

Impact of crisis resolution and home treatment teams on psychiatric admissions in England

BACKGROUND In 2000/01 crisis resolution and home treatment (CRHT) teams were introduced in England and have been associated in previous studies with reductions in in-patient admissions. AIMS To examine whether the implementation of CRHT teams has been associated with reductions in admissions. METHOD We used data from a previous national study for 229 primary care trusts (PCTs) between 1998/99 and 2003/04. We used a robust policy evaluation methodology to simultaneously examine temporal changes (PCTs before versus after the introduction of CRHT teams) and cross-sectional changes (PCTs with and without CRHT teams). RESULTS Controlling for various confounding factors, using different control groups and estimation methods, we find no significant differences in admissions between PCTs with and without CRHT teams. CONCLUSIONS Contrary to previous studies, we find no evidence that the CRHT policy per se has made any difference to admissions and suggest a need for more research on the policy as a whole.

A New Approach To Measuring Health System Output and Productivity

This paper considers methods to measure output and productivity in the delivery of health services, with an application to NHS hospital sector. It first develops a theoretical framework for measuring quality adjusted outputs and then considers how this might be implemented given available data. Measures of input use are discussed and productivity growth estimates are presented for the period 1998/9-2003/4. The paper concludes that available data are unlikely fully to capture quality improvements.

Is patient choice an effective mechanism to reduce waiting times

In many countries, patient choice is a routine part of the normal healthcare system. However, many choice initiatives in secondary care are part of policies aimed at reducing waiting times. This article provides evidence on the effectiveness of patient choice as a mechanism to reduce waiting times within a metropolitan area. The London Patient Choice Project was a large-scale pilot offering patients on hospital waiting lists a choice of alternative hospitals with shorter waiting times. A total of 22 500 patients were offered choice and 15 000 accepted. The acceptance rate of 66% was very high by international standards. In this article we address two questions. First, did the introduction of choice significantly reduce waiting times in London relative to the rest of the country where there was no choice? Second, how were the waiting times of London patients not offered choice affected by the choice regime? We examine the evidence on these issues for one specialty, orthopaedics. A difference-in-difference analysis is used to compare waiting times for hospitals within London before and after the introduction of choice. Although there was a small but significant reduction in waiting times in London relative to other areas where there was no patient choice, the main effect of the choice regime was to produce convergence of mean waiting times within London. Convergence was achieved by bringing down waiting times at the hospitals with high waiting times to the levels that prevailed in hospitals with low waiting times. This represented a clear improvement in equity of access, an important objective of the English National Health Service.

Relative performance evaluation of the English acute hospital sector

Relative performance evaluation has been suggested as a means to overcome information asymmetry between regulators and organizations when assessing efficiency. By comparing similar organizations the relationship between costs and effort can be better isolated. The English Department of Health (DoH) has undertaken relative performance evaluation in comparing the unit costs of acute hospitals using ordinary least squares (OLS) methods. After adjusting for exogenous influences in costs, residual unexplained cost differences are deemed to represent inefficiency. This paper questions the official interpretation of the OLS residuals. The OLS model is re-estimated to calculate confidence intervals around the residuals and as a stochastic cost frontier (SCF). It is concluded that English acute hospitals exhibit less in efficiency than is implied by official estimates.