Beverley Shea

External Validation of a Measurement Tool to Assess Systematic Reviews (AMSTAR)

Background Thousands of systematic reviews have been conducted in all areas of health care. However, the methodological quality of these reviews is variable and should routinely be appraised. AMSTAR is a measurement tool to assess systematic reviews. Methodology AMSTAR was used to appraise 42 reviews focusing on therapies to treat gastro-esophageal reflux disease, peptic ulcer disease, and other acid-related diseases. Two assessors applied the AMSTAR to each review. Two other assessors, plus a clinician and/or methodologist applied a global assessment to each review independently. Conclusions The sample of 42 reviews covered a wide range of methodological quality. The overall scores on AMSTAR ranged from 0 to 10 (out of a maximum of 11) with a mean of 4.6 (95% CI: 3.7 to 5.6) and median 4.0 (range 2.0 to 6.0). The inter-observer agreement of the individual items ranged from moderate to almost perfect agreement. Nine items scored a kappa of >0.75 (95% CI: 0.55 to 0.96). The reliability of the total AMSTAR score was excellent: kappa 0.84 (95% CI: 0.67 to 1.00) and Pearsons R 0.96 (95% CI: 0.92 to 0.98). The overall scores for the global assessment ranged from 2 to 7 (out of a maximum score of 7) with a mean of 4.43 (95% CI: 3.6 to 5.3) and median 4.0 (range 2.25 to 5.75). The agreement was lower with a kappa of 0.63 (95% CI: 0.40 to 0.88). Construct validity was shown by AMSTAR convergence with the results of the global assessment: Pearsons R 0.72 (95% CI: 0.53 to 0.84). For the AMSTAR total score, the limits of agreement were −0.19±1.38. This translates to a minimum detectable difference between reviews of 0.64 ‘AMSTAR points’. Further validation of AMSTAR is needed to assess its validity, reliability and perceived utility by appraisers and end users of reviews across a broader range of systematic reviews.

Laboratory Evaluation in the Diagnosis of Lyme Disease

1. Introduction 1.1 Lyme disease is the most common tick-borne disease in North America. From 1982 through 1994, more than 70 000 cases were reported in North America; most of these cases were in the United States [1]. It is important that clinicians diagnose Lyme disease correctly because efficacious therapy is available and delayed or inadequate treatment can lead to many morbid sequelae. Lyme disease is a complex multisystem disease caused by the spirochete Borrelia burgdorferi [2]. It affects persons of all ages and both sexes. Since the disease was recognized in Connecticut in 1975 [3], endemic areas have been identified in several regions in North America. In more restricted areas in some northeastern and upper midwestern U.S. states, the disease has assumed the characteristics of an emerging epidemic [4-9]. The true incidence is almost certainly underestimated because of under-reporting [10, 11]. 1.2 Most patients develop a distinctive rash, erythema migrans, that is accompanied by such flu-like symptoms as fatigue, headache, mild stiff neck, joint and muscle aches, and fever [12]. Some weeks or months after the initial exposure, symptoms and signs of disseminated disease (particularly neurologic, cardiac, or articular disease) may develop in untreated patients [13, 14]. 1.3 Case definitions of Lyme disease have been developed in the United States for national disease surveillance purposes. A positive serologic test result was initially required for patients who had erythema migrans alone and had not been exposed to Lyme disease in endemic areas [15], but the 1990 criteria established in the Centers for Disease Control and Preventions (CDCs) U.S. Lyme disease national surveillance definition reduced this requirement to a recommendation (Table 1) [16]. These criteria were developed for an epidemiologic case definition intended for surveillance purposes only. However, previous national disease surveillance criteria have been used in clinical studies [17, 18], and such definitions do provide standardization. Standardization allows comparisons of clinical studies and permits the performance of meta-analysis to facilitate development of clinically useful guidelines. Table 1. Criteria for Confirmed Lyme Disease 1.4 Requests for laboratory testing for Lyme disease have increased rapidly. In Wisconsin, for example, it was reported that more than 60 000 tests were being done annually [19]; in New Jersey, 5000 tests were done in 1 week in 1989 [20]. According to market projections for the United States, 2.79 million rapid tests were to have been done for Lyme disease in 1995 [21]. Testing is often done in persons who have only nonspecific signs and symptoms of illness, such as headache, fatigue, myalgia, or arthralgia. Even in highly endemic areas, the pretest probability of Lyme disease in such patients is less than 0.20 (usually much lower). Thus, even when highly experienced laboratories are used, the probability of a false-positive test result is higher than that of a true-positive result. This problem is compounded by the lack of standardized serologic tests for Lyme disease. Comparisons of the test results from different laboratories have shown poor reliability and accuracy; up to 21% of standardized positive samples are missed, and up to 7% of samples from persons with no known exposure are incorrectly identified as positive [22]. 1.5 This background paper provides a quantitative and qualitative evaluation of the predictive value of the laboratory diagnosis of Lyme disease. This evaluation forms the basis for guidelines on clinical diagnosis. Practitioners have been confused by the lack of consensus on diagnostic criteria for Lyme disease. The causes of this controversy arise from a combination of factors: the use of different tests in different laboratories, the use of different criteria to set positive and negative cutoff values for the same tests, different degrees of quality control in different laboratories, and differences in the community prevalence of Lyme disease. 1.6 We address each of these factors and make recommendations for the diagnostic workup of patients suspected of having Lyme disease. 2. Methods 2.1 Data Sources Relevant articles from the medical literature were identified by searching the MEDLINE database for English-language articles or articles with English-language abstracts published from 1982 (when the spirochetal cause of Lyme disease was established [23, 24]) to 1996. The keywords used were Lyme disease, Borrelia burgdorferi, diagnosis, ELISA, Western blot, immunofluorescence assay, polymerase chain reaction, urinary antigen detection, and culture. The computerized literature search was complemented by citations from authorities in the field. 2.2 Study Selection All identified articles were reviewed by using a modification of the methodologic criteria for evaluating diagnostic tests developed by Irwig and colleagues [25]. The included studies had to provide the following material: a clear statement on the test of interest, a description of the study characteristics that used a design that permitted the calculation of sensitivity and specificity, reproducible information on the sampling and clinical details of patients with the disease of interest and on controls (that is, data on the presence or absence of the criteria for Lyme disease described in the U.S. Lyme disease national surveillance case definition) (Table 1), and reproducible information on the reference standard (that is, cases diagnosed by experts who were blinded to the results of the diagnostic tests being evaluated). Because there are systematic differences in the strains of B. burgdorferi in different parts of the world, studies were excluded if they described results in patients outside of North America. When the same cohort of patients was described in more than one report, the results for individual patients were included only once. 2.3 Data Extraction Sensitivity, specificity, and likelihood ratios were calculated by using established methods [26]; a random-effects model was used to combine the proportions from the eligible studies [27]. 2.4 Estimates of Prevalence and Incidence Levels of the endemicity of Lyme disease in the United States can be estimated by using the annual incidence of Lyme disease reported to the CDC [1] (Figure 1). Figure 1. Rates of Lyme disease cases in the United States in 1993 as reported by states to the Centers for Disease Control and Prevention. 2.5 Epidemiologic studies of Lyme disease in communities in the eastern United States provide important information on the emergence of the disease in populations newly at risk, as well as some estimates of incidence and prevalence [29]. In two clusters of cases in New Jersey, risk was related to residence in new suburban housing developments and to occupational exposures among outdoor workers at a military reservation [6, 7]. A study on Fire Island, a barrier island off the southern coast of Long Island, New York, reported a seasonal incidence of 1% to 3% and a cumulative prevalence of 7.5% among residents of this summer vacation site [5]. A longitudinal study of a community of about 160 persons on Great Island, Massachusetts, found a slow build-up of incidence to a peak of 3 cases per 100 persons per year and a total cumulative prevalence of 16% over a 20-year period [9]. Two population-based studies in highly endemic suburban communities in Westchester, New York, reported seasonal attack rates of 2.6% and 3% and cumulative prevalences of 8.8% and 17%, respectively [30, 31]. On the basis of these data, we considered four categories of endemicity: low (incidence estimate, 0.01%), moderate (incidence estimate, 0.1%), high (incidence estimate, 1%), and very high (incidence estimate, 3%). 2.6 Likelihood Ratios and Treatment Thresholds of Tests Three of the authors constructed scenarios that describe three hypothetical patients. One had diffuse nonspecific muscle pain (scenario A), one had a rash resembling erythema migrans (scenario B), and one had episodic oligoarticular arthritis (scenario C) (Table 2). These models were used to compute the change in the probability of disease using likelihood ratios (likelihood ratio for positive test result = sensitivity [100 specificity]; likelihood ratio for negative test result = [100 sensitivity] specificity) [26] resulting from the use of enzyme-linked immunosorbent assay (ELISA) and Western blotting. Decision analysis was used to assess the relative cost-effectiveness of the management options in these clinical situations when the clinician must decide whether to perform laboratory testing for Lyme disease [32]. Incremental cost-effectiveness ratios were calculated as costs per quality-adjusted life-year for each scenario. This cost-effectiveness study is described in detail in a forthcoming paper [33]. Table 2. Hypothetical Patient Scenarios 3. Data Synthesis 3.1 Microbial Isolation Cultural isolation of B. burgdorferi is the best diagnostic evidence of Lyme disease. Borrelia burgdorferi grows well in Barbour, Stoenner, Kelly (BSK) medium, but it is difficult to obtain isolates from clinical specimens other than biopsy samples from erythema migrans lesions. 3.2 Thirty-four papers were identified by the literature search. None met the criteria formal analysis, but some case reports were worth noting. In the presence of erythema migrans, material has been collected from cutaneous lesions with various techniques, including direct aspiration of involved skin, aspiration after saline instillation, and skin biopsy. Wormser and colleagues [34] reported success rates of 29% with saline-lavage needle aspiration and 60% with 2-mm punch biopsies of the advancing edge of suspected primary erythema migrans lesions. Berger and colleagues [35] reported a success rate of more than 80% with biopsy specimens obtained from the leading edge of erythema migrans lesions. 3.3 Culture from sites other than the erythem

ROBIS: A new tool to assess risk of bias in systematic reviews was developed

Objective To develop ROBIS, a new tool for assessing the risk of bias in systematic reviews (rather than in primary studies). Study Design and Setting We used four-stage approach to develop ROBIS: define the scope, review the evidence base, hold a face-to-face meeting, and refine the tool through piloting. Results ROBIS is currently aimed at four broad categories of reviews mainly within health care settings: interventions, diagnosis, prognosis, and etiology. The target audience of ROBIS is primarily guideline developers, authors of overviews of systematic reviews (“reviews of reviews”), and review authors who might want to assess or avoid risk of bias in their reviews. The tool is completed in three phases: (1) assess relevance (optional), (2) identify concerns with the review process, and (3) judge risk of bias. Phase 2 covers four domains through which bias may be introduced into a systematic review: study eligibility criteria; identification and selection of studies; data collection and study appraisal; and synthesis and findings. Phase 3 assesses the overall risk of bias in the interpretation of review findings and whether this considered limitations identified in any of the phase 2 domains. Signaling questions are included to help judge concerns with the review process (phase 2) and the overall risk of bias in the review (phase 3); these questions flag aspects of review design related to the potential for bias and aim to help assessors judge risk of bias in the review process, results, and conclusions. Conclusions ROBIS is the first rigorously developed tool designed specifically to assess the risk of bias in systematic reviews.

Efficacy of the transcutaneous electrical nerve stimulation for the treatment of chronic low back pain: a meta-analysis.

Background. Low back pain affects a large proportion of the population. Transcutaneous electrical nerve stimulation (TENS) was introduced more than 30 years ago as an alternative therapy to pharmacologic treatments for chronic pain. However, despite its widespread use, the efficacy of TENS is still controversial. Purpose. The aim of this meta-analysis was to determine the efficacy of TENS in the treatment of chronic low back pain. Methods. The authors searched MEDLINE, EMBASE, PEDro, and the Cochrane Controlled Trials Register up to June 1, 2000. Only randomized controlled clinical trials of TENS for the treatment of patients with a clinical diagnosis of chronic low back pain were included. Abstracts were excluded unless further data could be obtained from the authors. Two reviewers independently selected trials and extracted data using predetermined forms. Data Analysis. Heterogeneity was tested with Cochrane’s Q test. A fixed effects model was used throughout for continuous variables, except where heterogeneity existed, in which case, a random effects model was used. Results are presented as weighted mean differences with 95% confidence intervals, where the difference between the treated and control groups was weighted by the inverse of the variance. Standardized mean differences were calculated by dividing the difference between the treated and control by the baseline variance. Standardized mean differences were used when different scales were integrated to measure the same concept. Dichotomous outcomes were analyzed with odds ratios. Main Results. Five trials were included, with 170 subjects randomized to the placebo group receiving sham TENS and 251 subjects receiving active TENS (153 for conventional mode, 98 for acupuncture-like TENS). The schedule of treatments varied greatly between studies ranging from one treatment/day for 2 consecutive days, to three treatments/day for 4 weeks. There were no statistically significant differences between the active TENS group compared with the placebo TENS group for any outcome measures. Subgroup analysis performed onTENS application and methodologic quality did not demonstrate a significant statistical difference (P > 0.05). Remaining preplanned subgroup analysis was not conducted because of the small number of included trials and the variety of outcome measures reported. Conclusion. The results of the meta-analysis present no evidence to support the use or nonuse of TENS alone in the treatment of chronic low back pain. Considering the small number of studies responding to the criteria to be included in this meta-analysis, it is clear that more appropriately designed studies are needed before a final conclusion. Clinicians and researchers should consistently report the characteristics of the TENS device and the application techniques used. New trials on TENS should make use of standardized outcome measures. This meta-analysis lacked data on how TENS efficacy is affected by four important factors: type of applications, site of application, treatment duration of TENS, and optimal frequencies and intensities.

Evidence-Based Rheumatology

Compiled by Cochrane collaborators and members of OMERACT (Outcome Measures in Rheumatology), Evidence-based Rheumatology is an essential resource for evidence-based medicine as applied to the musculoskeletal disorders. The introductory section covers the principles of evidence-based medicine in rheumatology, followed by clinical chapters covering all the major disorders. Each chapter includes non-drug therapy, drug therapy, and consumer evidence-based summaries. Evidence-Based Series: Evidence-based Rheumatology, part of the acclaimed series BMJ Evidence-based medicine textbooks that have revolutionised clinical medicine literature, comes with a fully searchable CD-ROM of the whole text. The text is kept up to date online at www.evidbasedrheum.com. Note: CD-ROM/DVD and other supplementary materials are not included as part of eBook file.

A comparison of the quality of Cochrane reviews and systematic reviews published in paper-based journals.

This study set out to compare Cochrane reviews and reviews published in paper-based journals. Two assessment tools were used to collect the data, a 23-itemchecklist developed by Sacks and a nine-itemscale developed by Oxman. Cochrane reviews were found to be better at reporting some items and paper-based reviews at reporting others. The overall quality was found to be low. This represents a serious situation because clinicians, health policy makers, and consumers are often told that systematic reviews represent “the best available evidence.” In the period since this study, the Cochrane Collaboration has taken steps to improve the quality of its reviews through, for example, more thorough prepublication refereeing, developments in the training and support offered to reviewers, and improvements in the system for postpublication peer review. In addition, the use of evidence-based criteria (i.e., the QUOROM statement) for reporting systematic reviews may help further to improve their quality.

Do silicone breast implants cause rheumatologic disorders? A systematic review for a court-appointed national science panel.

OBJECTIVE To assist in evaluating expert testimony and scientific evidence presented in law suits brought against silicone breast implant manufacturers, a US District Court Order established a National Science Panel to assess whether existing studies provide scientific evidence of an association between silicone breast implants and systemic classic/accepted connective disease, atypical connective disease, and certain signs and symptoms identified by plaintiffs in the law suits. Local disorders potentially associated with these implants were not addressed in this review. Therefore, we performed a systematic review of published studies on the association between silicone breast implants and systemic connective tissue disorders. METHODS Data from relevant studies (human cohort, case-control, or cross-sectional studies with > or = 10 participants and appropriate controls) were identified through literature searches of Medline, Current Contents, HealthStar, Biological Abstracts, EMBase, Toxline, and Dissertation Abstracts. Two independent reviewers, using standard collection forms, extracted data from the included studies. Adjusted relative risks (RRs) in cohort studies and odds ratios (ORs) in case-control and cross-sectional studies were reported if provided; otherwise, unadjusted RRs and ORs were calculated. RESULTS Twenty-four studies meeting inclusion criteria were identified. No association was evident between breast implants and any established or atypical connective tissue disorder. There was discordance among studies in reports of arthralgias, lymphadenopathy, myalgias, sicca symptoms, skin changes, and stiffness. CONCLUSION The panel found no evidence to support expert testimony suggesting an association between silicone breast implants and connective diseases. Discordance for symptoms may reflect differences in symptoms included in various categories, the small number of cases, and the effect of having single subjects with > 1 symptom represented in analyses of each symptom reported. The process presented here is an early example of the use of independent scientific panels to help courts clarify scientific evidence in legal proceedings.

Transcutaneous Electrical Nerve Stimulation for the Treatment of Chronic Low Back Pain : A Systematic Review

Study Design. Systematic review. Objective. To determine the effectiveness of transcutaneous electrical nerve stimulation (TENS) in the management of chronic LBP. Summary of Background Data. Chronic low back pain (LBP) affects a significant proportion of the population. TENS was introduced more than 30 years ago as an adjunct to pharmacologic pain management. However, despite its widespread use, the usefulness of TENS in chronic LBP is still controversial. Methods. We searched MEDLINE, EMBASE, PEDro, and the Cochrane Central Register of Controlled Trials (Issue 2, 2005), up to April 1, 2005. Only randomized controlled clinical trials (RCTs) evaluating the effect of TENS on chronic LBP were included. Two reviewers independently selected trials and extracted data using predetermined forms. Heterogeneity was tested with Cochrane’s Q test. A fixed effect model was used throughout for calculating continuous variables, except where heterogeneity existed, in which case a random effects model was used. Results are presented as weighted mean differences with 95% confidence intervals (95% CI), where the difference between the treated and control groups was weighted by the inverse of the variance. Standardized mean differences were calculated by dividing the difference between the treated and control by the baseline variance. Standardized mean differences were used when different scales were used to measure the same concept. Dichotomous outcomes were analyzed with odds ratios. Results. Two RCTs (175 patients) were included. They differed with respect to study design, methodologic quality, inclusion and exclusion criteria, characteristics of TENS application, treatment schedule, cointerventions, and measured outcomes. In one RCT, TENS produced significantly greater pain relief than the placebo control. However, in the other RCT, no statistically significantdifferences between treatment and control groups were shown for multiple outcome measures. Preplanned subgroup analyses, intended to examine the impact of different stimulation parameters, sites of TENS application, treatment durations, and baseline patient characteristics were not possible because of the small number of included trials. Conclusions. Evidence for the efficacy of TENS as an isolated intervention in the management of chronic LBP is limited and inconsistent. Larger, multicenter, RCTs are needed to better resolve its role in this condition. Increased attention should be given to the risks and benefits of long-term use, which more appropriately addresses the realities of managing chronic low back pain.

Issues relating to study design and risk of bias when including non-randomized studies in systematic reviews on the effects of interventions

Non-randomized studies may provide valuable evidence on the effects of interventions. They are the main source of evidence on the intended effects of some types of interventions and often provide the only evidence about the effects of interventions on long-term outcomes, rare events or adverse effects. Therefore, systematic reviews on the effects of interventions may include various types of non-randomized studies. In this second paper in a series, we address how review authors might articulate the particular non-randomized study designs they will include and how they might evaluate, in general terms, the extent to which a particular non-randomized study is at risk of important biases. We offer guidance for describing and classifying different non-randomized designs based on specific features of the studies in place of using non-informative study design labels. We also suggest criteria to consider when deciding whether to include non-randomized studies. We conclude that a taxonomy of study designs based on study design features is needed. Review authors need new tools specifically to assess the risk of bias for some non-randomized designs that involve a different inferential logic compared with parallel group trials. Copyright

Leflunomide for the treatment of rheumatoid arthritis

Leflunomide treatment appears to offer an alternative to methotrexate and sulfasalazine and is a welcome addition to the therapeutic armamentarium for treating active RA. Leflunomide treatment for more than 12 months results in clinically meaningful improvements in disease-specific measures of physical function. The phase 3 trials have shown leflunomide to be as effective as methotrexate and sulfasalazine and an option for initial DMARD therapy. As with all new agents, the long-term safety and value of leflunomide will be determined by use in the clinic.