Jon Nicholl

No surgical innovation without evaluation: the IDEAL recommendations

Surgery and other invasive therapies are complex interventions, the assessment of which is challenged by factors that depend on operator, team, and setting, such as learning curves, quality variations, and perception of equipoise. We propose recommendations for the assessment of surgery based on a five-stage description of the surgical development process. We also encourage the widespread use of prospective databases and registries. Reports of new techniques should be registered as a professional duty, anonymously if necessary when outcomes are adverse. Case series studies should be replaced by prospective development studies for early technical modifications and by prospective research databases for later pre-trial evaluation. Protocols for these studies should be registered publicly. Statistical process control techniques can be useful in both early and late assessment. Randomised trials should be used whenever possible to investigate efficacy, but adequate pre-trial data are essential to allow power calculations, clarify the definition and indications of the intervention, and develop quality measures. Difficulties in doing randomised clinical trials should be addressed by measures to evaluate learning curves and alleviate equipoise problems. Alternative prospective designs, such as interrupted time series studies, should be used when randomised trials are not feasible. Established procedures should be monitored with prospective databases to analyse outcome variations and to identify late and rare events. Achievement of improved design, conduct, and reporting of surgical research will need concerted action by editors, funders of health care and research, regulatory bodies, and professional societies.

Noninvasive Ventilation in Acute Cardiogenic Pulmonary Edema

BACKGROUND Noninvasive ventilation (continuous positive airway pressure [CPAP] or noninvasive intermittent positive-pressure ventilation [NIPPV]) appears to be of benefit in the immediate treatment of patients with acute cardiogenic pulmonary edema and may reduce mortality. We conducted a study to determine whether noninvasive ventilation reduces mortality and whether there are important differences in outcome associated with the method of treatment (CPAP or NIPPV). METHODS In a multicenter, open, prospective, randomized, controlled trial, patients were assigned to standard oxygen therapy, CPAP (5 to 15 cm of water), or NIPPV (inspiratory pressure, 8 to 20 cm of water; expiratory pressure, 4 to 10 cm of water). The primary end point for the comparison between noninvasive ventilation and standard oxygen therapy was death within 7 days after the initiation of treatment, and the primary end point for the comparison between NIPPV and CPAP was death or intubation within 7 days. RESULTS A total of 1069 patients (mean [+/-SD] age, 77.7+/-9.7 years; female sex, 56.9%) were assigned to standard oxygen therapy (367 patients), CPAP (346 patients), or NIPPV (356 patients). There was no significant difference in 7-day mortality between patients receiving standard oxygen therapy (9.8%) and those undergoing noninvasive ventilation (9.5%, P=0.87). There was no significant difference in the combined end point of death or intubation within 7 days between the two groups of patients undergoing noninvasive ventilation (11.7% for CPAP and 11.1% for NIPPV, P=0.81). As compared with standard oxygen therapy, noninvasive ventilation was associated with greater mean improvements at 1 hour after the beginning of treatment in patient-reported dyspnea (treatment difference, 0.7 on a visual-analogue scale ranging from 1 to 10; 95% confidence interval [CI], 0.2 to 1.3; P=0.008), heart rate (treatment difference, 4 beats per minute; 95% CI, 1 to 6; P=0.004), acidosis (treatment difference, pH 0.03; 95% CI, 0.02 to 0.04; P<0.001), and hypercapnia (treatment difference, 0.7 kPa [5.2 mm Hg]; 95% CI, 0.4 to 0.9; P<0.001). There were no treatment-related adverse events. CONCLUSIONS In patients with acute cardiogenic pulmonary edema, noninvasive ventilation induces a more rapid improvement in respiratory distress and metabolic disturbance than does standard oxygen therapy but has no effect on short-term mortality. (Current Controlled Trials number, ISRCTN07448447.)

Three techniques for integrating data in mixed methods studies

Techniques designed to combine the results of qualitative and quantitative studies can provide researchers with more knowledge than separate analysis

Evaluation and stages of surgical innovations

Surgical innovation is an important part of surgical practice. Its assessment is complex because of idiosyncrasies related to surgical practice, but necessary so that introduction and adoption of surgical innovations can derive from evidence-based principles rather than trial and error. A regulatory framework is also desirable to protect patients against the potential harms of any novel procedure. In this first of three Series papers on surgical innovation and evaluation, we propose a five-stage paradigm to describe the development of innovative surgical procedures.

Using the SF-36 and Euroqol on an elderly population.

An important methodological issue in measuring health-related quality of life is whether instruments such as SF-36 and EQ can be used on an elderly population. This paper aims to test the completion, reliability and validity of the SF-36 and Euroqol on an elderly female population, and to compare them with the OPCS Disability Survey. Three hundred and eighty women aged 75 and over participated in a randomized controlled trial of the use of clodronate provided the sample. As part of the trial, patients were asked to complete the UK SF-36 and Euroqol, and the OPCS disability survey instrument administered by interview in a hospital clinic at baseline. A random subsample of respondents were retested six months later. The SF-36 achieved poorer levels of completion by dimension (68.1%–88.9%) than the OPCS (99.2%) and Euroqol (84%–93.5%) instruments. There were no major floor effects in the distribution of scores, except for the role dimensions of SF-36. Correlation between test-retest were significant for all instruments, but lower for the role dimensions and social functioning of SF-36, and these dimensions also had 95% Cls for the mean differences in excess of 10 points. There was substantial agreement between the three instruments, and evidence for their construct validity against age and recent use of health services. The sensitivities of the instruments were tested through hypothetical changes in health status. There was some evidence of greater sensitivity to lower levels of morbidity in the SF-36. Where brevity is required and the health changes are expected to be substantial, then EQ may be sufficient. For greater sensitivity SF-36 seems to have an advantage, however lower completion rates and problems with consistency suggest it requires adaptation. One solution would be to use interviewer administration. Another would be to change the SF-36 to make it more suitable for use in elderly people, although this may reduce its usefulness as a generic instrument.

The quality of mixed methods studies in health services research.

Objectives To assess the quality of mixed methods studies in health services research (HSR). Methods We identified 118 mixed methods studies funded by the Department of Health in England between 1994 and 2004, and obtained proposals and/or final reports for 75. We applied a set of quality questions to both the proposal and report of each study, addressing the success of the study, the mixed methods design, the individual qualitative and quantitative components, the integration between methods and the inferences drawn from completed studies. Results Most studies were completed successfully. Researchers mainly ignored the mixed methods design and described only the separate components of a study. There was a lack of justification for, and transparency of, the mixed methods design in both proposals and reports, and this had implications for making judgements about the quality of individual components in the context of the design used. There was also a lack of transparency of the individual methods in terms of clear exposition of data collection and analysis, and this was more a problem for the qualitative than the quantitative component: 42% (19/45) versus 18% (8/45) of proposals (p 5 0.011). Judgements about integration could rarely be made due to the absence of an attempt at integration of data and findings from different components within a study. Conclusions The HSR community could improve mixed methods studies by giving more consideration to describing and justifying the design, being transparent about the qualitative component, and attempting to integrate data and findings from the individual components.

Randomised controlled trial and economic evaluation of a chest pain observation unit compared with routine care

Abstract Objectives To measure the effectiveness and cost effectiveness of providing care in a chest pain observation unit compared with routine care for patients with acute, undifferentiated chest pain. Design Cluster randomised controlled trial, with 442 days randomised to the chest pain observation unit or routine care, and cost effectiveness analysis from a health service costing perspective. Setting The emergency department at the Northern General Hospital, Sheffield, United Kingdom. Participants 972 patients with acute, undifferentiated chest pain (479 attending on days when care was delivered in the chest pain observation unit, 493 on days of routine care) followed up until six months after initial attendance. Main outcome measures The proportion of participants admitted to hospital, the proportion with acute coronary syndrome sent home inappropriately, major adverse cardiac events over six months, health utility, hospital reattendance and readmission, and costs per patient to the health service. Results Use of a chest pain observation unit reduced the proportion of patients admitted from 54% to 37% (difference 17%, odds ratio 0.50, 95% confidence interval 0.39 to 0.65, P < 0.001) and the proportion discharged with acute coronary syndrome from 14% to 6% (8%, −7% to 23%, P = 0.264). Rates of cardiac event were unchanged. Care in the chest pain observation unit was associated with improved health utility during follow up (0.0137 quality adjusted life years gained, 95% confidence interval 0.0030 to 0.0254, P = 0.022) and a saving of £78 per patient (−£56 to £210, P = 0.252). Conclusions Care in a chest pain observation unit can improve outcomes and may reduce costs to the health service. It seems to be more effective and more cost effective than routine care.

Why, and how, mixed methods research is undertaken in health services research in England: a mixed methods study

BackgroundRecently, there has been a surge of international interest in combining qualitative and quantitative methods in a single study – often called mixed methods research. It is timely to consider why and how mixed methods research is used in health services research (HSR).MethodsDocumentary analysis of proposals and reports of 75 mixed methods studies funded by a research commissioner of HSR in England between 1994 and 2004. Face-to-face semi-structured interviews with 20 researchers sampled from these studies.Results18% (119/647) of HSR studies were classified as mixed methods research. In the documentation, comprehensiveness was the main driver for using mixed methods research, with researchers wanting to address a wider range of questions than quantitative methods alone would allow. Interviewees elaborated on this, identifying the need for qualitative research to engage with the complexity of health, health care interventions, and the environment in which studies took place. Motivations for adopting a mixed methods approach were not always based on the intrinsic value of mixed methods research for addressing the research question; they could be strategic, for example, to obtain funding. Mixed methods research was used in the context of evaluation, including randomised and non-randomised designs; survey and fieldwork exploratory studies; and instrument development. Studies drew on a limited number of methods – particularly surveys and individual interviews – but used methods in a wide range of roles.ConclusionMixed methods research is common in HSR in the UK. Its use is driven by pragmatism rather than principle, motivated by the perceived deficit of quantitative methods alone to address the complexity of research in health care, as well as other more strategic gains. Methods are combined in a range of contexts, yet the emerging methodological contributions from HSR to the field of mixed methods research are currently limited to the single context of combining qualitative methods and randomised controlled trials. Health services researchers could further contribute to the development of mixed methods research in the contexts of instrument development, survey and fieldwork, and non-randomised evaluations.

Rethinking pragmatic randomised controlled trials: introducing the “cohort multiple randomised controlled trial” design

Pragmatic trials are important for informing routine clinical practice, but current designs have shortcomings. Clare Relton and colleagues outline the new “cohort multiple randomised controlled trial” design, which could help address the problems associated with existing approaches

An epistemology of patient safety research: a framework for study design and interpretation. Part 2. Study design

This is the second in a four-part series of articles detailing the epistemology of patient safety research. This article concentrates on issues of study design. It first considers the range of designs that may be used in the evaluation of patient safety interventions, highlighting the circumstances in which each is appropriate. The paper then provides details about an innovative study design, the stepped wedge, which may be particularly appropriate in the context of patient safety interventions, since these are expected to do more good than harm. The unit of allocation in patient safety research is also considered, since many interventions need to be delivered at cluster or service level. The paper also discusses the need to ensure the masking of patients, caregivers, observers and analysts wherever possible to minimise information biases and the Hawthorne effect. The difficulties associated with masking in patient safety research are described and suggestions given on how these can be ameliorated. The paper finally considers the role of study design in increasing confidence in the generalisability of study results over time and place. The extent to which findings can be generalised over time and place should be considered as part of an evaluation, for example by undertaking qualitative or quantitative measures of fidelity, attitudes or subgroup effects.