Nick Black

The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions.

OBJECTIVE: To test the feasibility of creating a valid and reliable checklist with the following features: appropriate for assessing both randomised and non-randomised studies; provision of both an overall score for study quality and a profile of scores not only for the quality of reporting, internal validity (bias and confounding) and power, but also for external validity. DESIGN: A pilot version was first developed, based on epidemiological principles, reviews, and existing checklists for randomised studies. Face and content validity were assessed by three experienced reviewers and reliability was determined using two raters assessing 10 randomised and 10 non-randomised studies. Using different raters, the checklist was revised and tested for internal consistency (Kuder-Richardson 20), test-retest and inter-rater reliability (Spearman correlation coefficient and sign rank test; kappa statistics), criterion validity, and respondent burden. MAIN RESULTS: The performance of the checklist improved considerably after revision of a pilot version. The Quality Index had high internal consistency (KR-20: 0.89) as did the subscales apart from external validity (KR-20: 0.54). Test-retest (r 0.88) and inter-rater (r 0.75) reliability of the Quality Index were good. Reliability of the subscales varied from good (bias) to poor (external validity). The Quality Index correlated highly with an existing, established instrument for assessing randomised studies (r 0.90). There was little difference between its performance with non-randomised and with randomised studies. Raters took about 20 minutes to assess each paper (range 10 to 45 minutes). CONCLUSIONS: This study has shown that it is feasible to develop a checklist that can be used to assess the methodological quality not only of randomised controlled trials but also non-randomised studies. It has also shown that it is possible to produce a checklist that provides a profile of the paper, alerting reviewers to its particular methodological strengths and weaknesses. Further work is required to improve the checklist and the training of raters in the assessment of external validity.

Evidence based policy: proceed with care.

# Evidence based policy: proceed with care {#article-title-2} The emergence of evidence based medicine in the early 1990s led to some clinicians challenging managers and policymakers to be equally evidence based in their policymaking. This demand was shared by some health policy analysts: “At a time when ministers are arguing that medicine should be evidence-based, is it not reasonable to suggest that this should also apply to health policy? If doctors are expected to base their decisions on the findings of research surely politicians should do the same … the case for evidence-based policymaking is difficult to refute.”1 The need to be seen to be making evidence based decisions has permeated all areas of British public policy. The government has proclaimed the need for evidence based policing, and the 1998 strategic defence review introduced evidence based defence.2 In the health sector, the concept of evidence based policy has gained ground, and a journal has been launched devoted to this challenge ( Journal of Evidence Based Health Policy and Management). Despite some groups using evidence based policy as a fig leaf, it seems difficult to argue with the idea that scientific research should drive policy. However, before accepting the argument we need to understand the implied model of policymaking. #### Summary points Evidence based policy is being encouraged in all areas of public service, including health care Research currently has little direct influence on health services policy or governance policies The implicit assumption of a linear relation between research evidence and policy needs to be replaced with a more interactive model Researchers need a better understanding of the policy process, funding bodies must change their conception of how research influences policy, and policy makers should become more involved in the conceptualisation and conduct of research Until then, researchers should be cautious about uncritically accepting the notion of evidence based policy In …

Patient reported outcome measures could help transform healthcare

Routine use of patient reported outcome measures (PROMs) has the potential to help transform healthcare, says Nick Black. Not only can PROMs help patients and clinicians make better decisions, but they can also enable comparisons of providers’ performances to stimulate improvements in services

Epidemiology of severe sepsis occurring in the first 24 hrs in intensive care units in england, Wales, and Northern Ireland

ObjectiveTo investigate the numbers, clinical characteristics, resource use, and outcomes of admissions who met precise clinical and physiologic criteria for severe sepsis (as defined in the PROWESS trial) in the first 24 hrs in the intensive care unit. DesignObservational cohort study, with retrospective analysis of prospectively collected data. SettingNinety-one adult general intensive care units in England, Wales, and Northern Ireland between 1995 and 2000. PatientsPatients were 56,673 adult admissions. InterventionsNone. Measurements and Main ResultsWe found that 27.1% of adult intensive care unit admissions met severe sepsis criteria in the first 24 hrs in the intensive care unit. Most were nonsurgical (67%), and the most common organ system dysfunctions were seen in the cardiovascular (88%) and respiratory (81%) systems. Modeling the data for England and Wales for 1997 suggested that 51 (95% confidence interval, 46–58) per 100,000 population per year were admitted to intensive care units and met severe sepsis criteria in the first 24 hrs.Of the intensive care unit admissions who met severe sepsis criteria in the first 24 hrs, 35% died before intensive care unit discharge and 47% died during their hospital stay. Hospital mortality rate ranged from 17% in the 16–19 age group to 64% in those >85 yrs. In England and Wales in 1997, an estimated 24 (95% confidence interval, 21–28) per 100,000 population per year died after intensive care unit admissions with severe sepsis in the first 24 hrs.For intensive care unit admissions who met severe sepsis criteria in the first 24 hrs, median intensive care unit length of stay was 3.56 days (interquartile range, 1.50–9.32) and median hospital length of stay was 18 days (interquartile range, 8–36 days). These admissions used 45% of the intensive care unit and 33% of the hospital bed days used by all intensive care unit admissions. ConclusionsSevere sepsis is common and presents a major challenge for clinicians, managers, and healthcare policymakers. Intensive care unit admissions meeting severe sepsis criteria have a high mortality rate and high resource use.

Threats to Applicability of Randomised Trials: Exclusions and Selective Participation

Background: Although the randomised controlled trial (RCT) is regarded as the ‘gold standard’ in terms of evaluating the effectiveness of interventions, it is susceptible to challenges to its external validity if those participating are unrepresentative of the reference population for whom the intervention in question is intended. In the past, reporting on numbers and types of potential subjects that have been excluded by design, and centres, clinicians or patients that have elected not to participate, has generally been poor, and the threat to inference posed by possible selection bias is unclear. Methods: A systematic review was undertaken, based largely on MEDLINE and EMBASE with follow-up of cited references, to assess the extent, nature and importance of excluding potential subjects or the unwillingness of particular centres, clinicians or patients to participate. Results: RCTs vary widely in the extent to which potential future recipients of treatment are included. The reasons cited for excluding certain categories of patient may be medical or scientific. Medical reasons include a high risk of adverse effects and the belief that benefit will be relatively small or absent (or has already been established) in the groups in question. Scientific reasons include more precise estimates of treatment effect because of a relatively homogeneous sample and the reduction of potential bias by excluding those individuals most likely to be lost to follow-up. Many RCTs have blanket exclusions, such as the elderly, women and ethnic minorities, but reasons for these exclusions are seldom given. Evaluative research is undertaken predominantly in university or teaching centres. Non-randomised studies are more likely than RCTs to include non-teaching centres. The effect of patient non-participation appears to depend on whether the RCT is concerned with treatment of an existing condition or with disease prevention. Participants in treatment trials tend to be more severely ill than those who do not participate. In contrast, those who participate in prevention trials are more likely to have adopted a healthy lifestyle than those who decline. Most evaluative studies fail to document adequately the characteristics of those who, while eligible, do not participate. However, subjects included in RCTs (i.e. eligible and participating) tend to have a different prognosis than patients identified from clinical databases. Conclusions: Narrow inclusion criteria may offer benefits such as increased precision and reduced loss to follow-up, but there are important disadvantages, such as uncertainty about extrapolation of results, which may result in denial of effective treatment to groups who might benefit, and delay in obtaining definitive results because of reduced recruitment rate. Selective participation by teaching centres and sicker patients in treatment RCTs may exaggerate the measured treatment effect. Prevention trials, on the other hand, may underestimate effects as participants have less capacity to benefit.

Consensus Development Methods: A Review of Best Practice in Creating Clinical Guidelines

Background: Although there is debate about the appropriate place of guidelines in clinical practice, guidelines can be seen as one way of assisting clinicians in decision-making. Given the likely diversity of opinion that any group of people may display when considering a topic, methods are needed for organising subjective judgements. Three principal methods (Delphi, nominal group technique, consensus development conference) exist which share the common objective of synthesising judgements when a state of uncertainty exists. Objectives: To identify the factors that shape and influence the clinical guidelines that emerge from consensus development methods and to make recommendations about best practice in the use of such methods. Methods: Five electronic databases were searched: Medline (1966-1996), PsychLIT (1974-1996), Social Science Citation Index (1990-1996), ABI Inform and Sociofile. From the searches and reference lists of articles a total of 177 empirical and review articles were selected for review. Results: The output from consensus development methods may be affected by: the way the task is set (choice of cues, recognition of contextual cues, the focus of the task, the comprehensiveness of the scenarios); the selection of participants (choice of individuals, degree of homogeneity of the group, their background, their number); the selection and presentation of scientific information (format, extent to which its quality and content is assessed); the way any interaction is structured (number of rating rounds, ensuring equitable participation, physical environment for meetings); and the method of synthesising individual judgements (definition of agreement, rules governing outliers, method of mathematical aggregation). Conclusions: Although a considerable amount of research has been carried out, many aspects have not been investigated sufficiently. For the time being at least, advice on those aspects has, therefore, to be based on the users own commonsense and the experience of those who have used or participated in these methods. Even in the long term, some aspects will not be amenable to scientific study. Meanwhile, adherence to best practice will enhance the validity, reliability and impact of the clinical guidelines produced.

Indications for and outcome of total abdominal hysterectomy for benign disease : a prospective cohort study

Objective To describe the indications for total abdominal hysterectomy for women with nonmalignant disease and to determine the immediate (initial ten days) and medium term outcome.

Use of consensus development to establish national research priorities in critical care

Abstract Objectives: To test the feasibility of using a nominal group technique to establish clinical and health services research priorities in critical care and to test the representativeness of the groups views. Design: Generation of topics by means of a national survey; a nominal group technique to establish the level of consensus; a survey to test the representativeness of the results. Setting: United Kingdom and Republic of Ireland. Subjects: Nominal group composed of 10 doctors (8 consultants, 2 trainees) and 2 nurses. Main outcome measure: Level of support (median) and level of agreement (mean absolute deviation from the median) derived from a 9 point Likert scale. Results: Of the 325 intensive care units approached, 187 (58%) responded, providing about 1000 suggestions for research. Of the 106 most frequently suggested topics considered by the nominal group, 37 attracted strong support, 48 moderate support and 21 weak support. There was more agreement after the group had met—overall mean of the mean absolute deviations from the median fell from 1.41 to 1.26. The groups views represented the views of the wider community of critical care staff (r=0.73, P<0.01). There was no significant difference in the views of staff from teaching or from non-teaching hospitals. Of the 37 topics that attracted the strongest support, 24 were concerned with organisational aspects of critical care and only 13 with technology assessment or clinical research. Conclusions: A nominal group technique is feasible and reliable for determining research priorities among clinicians. This approach is more democratic and transparent than the traditional methods used by research funding bodies. The results suggest that clinicians perceive research into the best ways of delivering and organising services as a high priority.

Developing clinical guidelines: a challenge to current methods

Current methods for producing clinical guidelines are cumbersome and not always reliable. Could a more streamlined approach improve coverage and make decisions more transparent?

Impact of surgery for stress incontinence on morbidity: cohort study.

Abstract Objectives: To describe the impact of surgery for stress incontinence on the severity of symptoms, other mental and physical symptoms, and overall health. To describe the incidence of postoperative complications. Design: Prospective cohort study; questionnaires completed by patients before and 3, 6, and 12 months after surgery. Questionnaires completed by surgeons both before and after surgery. Setting: 18 hospitals in the North Thames region. Subjects: 442 women treated surgically for stress incontinence between January 1993 and June 1994. 367 women returned the 3 month questionnaire; 364 returned the 6 month questionnaire; and 359 returned the 12 month questionnaire. 49 surgeons provided perioperative information on 285 of the 442 women and postoperative information on 278. Main outcome measures: Stress incontinence symptom severity index, other urinary symptoms, bowel function, mental health, complications, global measures. Results: Most women (288; 87%) reported an improvement in the severity of their stress incontinence, though only 92 (28%) were cured (continent). These improvements persisted for at least 12 months. The likelihood of improvement was similar regardless of whether urodynamic pressure studies had been conducted before surgery. Following surgery, women were less likely to suffer from urinary frequency, nocturia, postvoid fullness, dysuria, and urgency. While mental health improved for 194 (71%), a quarter of women reported deterioration. Only 37 (10%) were satisfied with postoperative pain control. A third experienced one or more complications while in hospital, most commonly difficulty urinating. This problem affected 1 in 11 women after discharge. A year after surgery two thirds of women reported feeling better (251; 72%), that the outcome met or exceeded their expectations (230; 66%), and that they would recommend the operation to a friend in a similar situation (239; 68%). Surgeons tended to be more optimistic about the effects of surgery; they were satisfied with the outcome in 176 (85%) cases and would again treat 245 (94%) of the women as they had done previously. Conclusions: Although surgery reduces the severity of stress incontinence it is not as effective as current textbooks suggest. Women considering surgery should be provided with more accurate information on the likelihood of an improvement in symptoms and the occurrence of complications, including postoperative pain. Urgency and urge incontinence should not be considered contraindications to surgery. The need for urodynamic assessment before surgery should be reappraised. Key messages Although surgery improves stress incontinence in most women (87%), only 28% are continent one year later The need for preoperative urodynamic testing should be reappraised Urgency and urge incontinence should not be considered contraindications to surgery Women considering surgery should receive more accurate information on the probability of an improvement in symptoms and possible complications There is a need for a rigorous, pragmatic, randomised trial of surgery for stress incontinence