John W Williams

Grading quality of evidence and strength of recommendations.

Abstract Users of clinical practice guidelines and other recommendations need to know how much confidence they can place in the recommendations. Systematic and explicit methods of making judgments can reduce errors and improve communication. We have developed a system for grading the quality of evidence and the strength of recommendations that can be applied across a wide range of interventions and contexts. In this article we present a summary of our approach from the perspective of a guideline user. Judgments about the strength of a recommendation require consideration of the balance between benefits and harms, the quality of the evidence, translation of the evidence into specific circumstances, and the certainty of the baseline risk. It is also important to consider costs (resource utilisation) before making a recommendation. Inconsistencies among systems for grading the quality of evidence and the strength of recommendations reduce their potential to facilitate critical appraisal and improve communication of these judgments. Our system for guiding these complex judgments balances the need for simplicity with the need for full and transparent consideration of all important issues. Clinical guidelines are only as good as the evidence and judgments they are based on. The GRADE approach aims to make it easier for users to assess the judgments behind recommendations

GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)

In the GRADE approach, randomized trials start as high-quality evidence and observational studies as low-quality evidence, but both can be rated down if most of the relevant evidence comes from studies that suffer from a high risk of bias. Well-established limitations of randomized trials include failure to conceal allocation, failure to blind, loss to follow-up, and failure to appropriately consider the intention-to-treat principle. More recently recognized limitations include stopping early for apparent benefit and selective reporting of outcomes according to the results. Key limitations of observational studies include use of inappropriate controls and failure to adequately adjust for prognostic imbalance. Risk of bias may vary across outcomes (e.g., loss to follow-up may be far less for all-cause mortality than for quality of life), a consideration that many systematic reviews ignore. In deciding whether to rate down for risk of bias--whether for randomized trials or observational studies--authors should not take an approach that averages across studies. Rather, for any individual outcome, when there are some studies with a high risk, and some with a low risk of bias, they should consider including only the studies with a lower risk of bias.

GRADE guidelines: 5. Rating the quality of evidence—publication bias

In the GRADE approach, randomized trials start as high-quality evidence and observational studies as low-quality evidence, but both can be rated down if a body of evidence is associated with a high risk of publication bias. Even when individual studies included in best-evidence summaries have a low risk of bias, publication bias can result in substantial overestimates of effect. Authors should suspect publication bias when available evidence comes from a number of small studies, most of which have been commercially funded. A number of approaches based on examination of the pattern of data are available to help assess publication bias. The most popular of these is the funnel plot; all, however, have substantial limitations. Publication bias is likely frequent, and caution in the face of early results, particularly with small sample size and number of events, is warranted.

Systems for grading the quality of evidence and the strength of recommendations I: Critical appraisal of existing approaches The GRADE Working Group

BackgroundA number of approaches have been used to grade levels of evidence and the strength of recommendations. The use of many different approaches detracts from one of the main reasons for having explicit approaches: to concisely characterise and communicate this information so that it can easily be understood and thereby help people make well-informed decisions. Our objective was to critically appraise six prominent systems for grading levels of evidence and the strength of recommendations as a basis for agreeing on characteristics of a common, sensible approach to grading levels of evidence and the strength of recommendations.MethodsSix prominent systems for grading levels of evidence and strength of recommendations were selected and someone familiar with each system prepared a description of each of these. Twelve assessors independently evaluated each system based on twelve criteria to assess the sensibility of the different approaches. Systems used by 51 organisations were compared with these six approaches.ResultsThere was poor agreement about the sensibility of the six systems. Only one of the systems was suitable for all four types of questions we considered (effectiveness, harm, diagnosis and prognosis). None of the systems was considered usable for all of the target groups we considered (professionals, patients and policy makers). The raters found low reproducibility of judgements made using all six systems. Systems used by 51 organisations that sponsor clinical practice guidelines included a number of minor variations of the six systems that we critically appraised.ConclusionsAll of the currently used approaches to grading levels of evidence and the strength of recommendations have important shortcomings.

Case-Finding Instruments for Depression in Primary Care Settings

Depressive disorders are common, persistent, and recurring afflictions among primary care patients. They cause substantial suffering for patients and their families and are associated with a loss of personal productivity and a markedly increased risk for suicide. Further, the presence of depression puts persons with comorbid conditions, such as recent myocardial infarction, at increased risk for illness and death. Persons with depression spend more time with their physicians during office visits and use more health care than persons without depression [1]. The annual health care cost associated with depression in the United States is estimated to be

Effect of Clinical Decision-Support Systems: A Systematic Review

43.7 billion [2]. Underscoring the importance of identifying patients with depression is that the effectiveness of therapy, including antidepressants, psychotherapy, and counseling, has clearly been established [3]. Despite these issues, primary care providers fail to diagnose and treat as many as 35% to 50% of patients with depressive disorders [4, 5]. Obstacles to the appropriate recognition of depression include inadequate provider knowledge of diagnostic criteria; competing comorbid conditions and priorities among primary care patients; time limitations in busy office settings; concern about the implications of labeling; poor reimbursement mechanisms; and uncertainty about the value, accuracy, and efficiency of screening mechanisms for identifying patients with depression. We address the last of these obstacles and assess the feasibility and operating characteristics of several case-finding instruments that have been used to detect depressive disorders in primary care settings. Our ultimate goal is to familiarize providers with the advantages and disadvantages of these instruments so that they can make informed decisions about incorporating them into practice. Methods Data Acquisition We did a MEDLINE search of the English-language medical literature published from 1966 through February 1994. Search terms included depressive disorder or depression, diagnosis, and the specific names of each of 11 case-finding instruments cited in previous relevant reviews or bibliographies [6-9]. Other sources were references identified from pertinent articles and national experts in the field of depression. Experts included authors of papers that were selected for review and two members of the Agency for Health Care Policy and Research Guideline Panel on Depression. Of 906 articles identified through MEDLINE, 210 were deemed potentially relevant. These were reviewed to identify studies that met the following selection criteria. Study samples had to have been composed of primary care patients attending clinic. Patients were excluded if they had been selected because they had specific conditions (such as chronic pain or cancer) or because they had specific demographic characteristics (for example, they were immigrants in a particular ethnic group). Both a case-finding instrument and a diagnostic criterion standard had to have been administered. The criterion standard had to have had formal standardized diagnostic criteria for depression. Accepted criterion standards were the Diagnostic Schedule Manual-3 criteria (DSM-III or DSM-III-R) and the Research Diagnostic Criteria, or a close approximation of these. Standard interview procedures, such as the Diagnostic Interview Schedule or the Structured Clinical Interview for DSM-III, had to have been used to arrive at the diagnosis. Chart or physician diagnoses of depression that were made without specified formal interview procedures and diagnostic criteria were excluded. Nineteen studies involving nine case-finding instruments met the selection criteria: Fourteen were found during the MEDLINE search; 1 came from a relevant bibliography; and 4 were unpublished at the time of the search and came from experts [10-28]. Of the remaining articles screened, 92% were excluded because they did not involve primary care patients, 6% were excluded because they had used an inadequate criterion standard, and 2% were excluded because they involved selected populations [29, 30] or because they had tested modified and unvalidated versions of case-finding instruments [31]. Data Extraction Articles were abstracted by two independent reviewers. Determination of study quality was made on the basis of 1) whether the case-finding instrument was administered and interpreted independently of the criterion standard and 2) whether the proportion of persons receiving the criterion standard assessment was less than or more than 50% of those approached for criterion standard assessment. Quality assessment addressed methodologic issues relevant to the evaluation of diagnostic tests (such as independent assessment and selection bias) and did not necessarily reflect the ability of studies to address their original aims. There were no disagreements about quality assessments. Data Synthesis Established cut-points for case-finding instruments (Table 1) were used. Two-by-two tables were constructed that categorized numbers of screened-positive and screened-negative persons who did and did not meet criterion standard diagnosis for major depression and major depression or dysthymia. Kraemers method [32] was used to adjust for verification bias for studies that used two-stage assessment techniques; whereby the criterion standard was administered only to a random sample of persons who screened negative on case-finding instruments [33]. The authors of all but one study provided us with additional data and analyses when two-by-two tables could not be derived from abstraction of the published article. This one study [28] was dropped from further review because its authors could not be contacted and tables could not be derived from published information. Table 1. Characteristics of Case-Finding Instruments That Have Been Used to Detect Depression in Primary Care Settings* A scattergram (Figure 1) plotting true-positive against false-positive rates was constructed to visually evaluate variability among studies [34]. To provide a visual reference for the consistency of study results, we modeled a summary receiver-operating curve based on the logit transformations of the true-positive and false-positive rates. Figure 1. Plot of true-positive rate against false-positive rate for case-finding instruments to detect major depression. Average sensitivities and specificities, weighted by study size and corrected for two-stage assessment techniques when indicated, were computed both by case-finding instrument and by overall instruments [35]. The point estimates and 95% CIs were calculated using a linear random-effects model [36, 37]. Approximate 95% CIs were estimated using quadratic root formulae because most of the point estimates were near unity [37]. Differences in weighted average sensitivities and specificities between case-finding instruments were evaluated using the z statistic with the Scheffe multiple-comparison adjustment [37]. Stratified analyses were done to evaluate whether estimated sensitivities and specificities varied between high-quality studies and those with major selection bias or lack of independent assessment. Regression analysis was used to determine associations between reported study prevalences and sensitivity estimates [37]. Results Descriptions of Case-Finding Instruments Characteristics of the nine case-finding instruments that have been evaluated in primary care settings are presented in Table 1. All of the questionnaires are written either at the easy (3rd to 5th grade) or average (6th to 9th grade) reading level [38]. Almost all can be self-administered in less than 5 minutes. Except for the General Health Questionnaire, all include specific questions aimed at assessing depressed mood or whether a patient feels sad or blue. All include questions assessing anhedonia. Most are available in languages other than English, such as Spanish. The Beck Depression Inventory, the Center for Epidemiologic Studies Depression Screen, and the Zung Self-Assessment Depression Scale are three commonly used, traditional instruments that were developed specifically to identify depression. They include similar numbers of questions and use response formats that rely either on ranking symptom severity or on classifying frequency of symptoms. The time frames of questions are today for the Beck Depression Inventory, over the past week for the Center for Epidemiologic Studies Depression Screen, and recently for the Zung Self-Assessment Depression Scale. These three instruments have been used in numerous settings (including the community, the clinic, and the hospital) not only to identify depression but also to rate severity of depression and to monitor response to therapy. The General Health Questionnaire and the Hopkins Symptom Checklist are questionnaires that screen for general psychiatric illness; the Hopkins Checklist has a specific category for depression. Both of these instruments have several versions with different numbers of questions. The Medical Outcomes Study Depression Screen is a depression-specific screening instrument that was developed by combining two questions from the Diagnostic Interview Schedule [39] with six questions from the Center for Epidemiologic Studies Depression Screen. A logistic regression scoring method is used; this requires a calculator. The Primary Care Evaluation of Mental Disorders (PRIME-MD) and the Symptom Driven Diagnostic System-Primary Care instruments are recently developed, multidimensional questionnaires. Each has screening questions arranged in several categories (for example, mood or depression, anxiety, alcohol abuse, and somatization) that are used to trigger more extensive diagnostic interviewing sections for specific DSM-III-R diagnoses. The depression components of these two instruments include the fewest questions of all case-finding instruments that have been studied in primary care settings. Descriptions of Studies and Fi

Systematic Review: Factors Associated With Risk for and Possible Prevention of Cognitive Decline in Later Life

BACKGROUND Despite increasing emphasis on the role of clinical decision-support systems (CDSSs) for improving care and reducing costs, evidence to support widespread use is lacking. PURPOSE To evaluate the effect of CDSSs on clinical outcomes, health care processes, workload and efficiency, patient satisfaction, cost, and provider use and implementation. DATA SOURCES MEDLINE, CINAHL, PsycINFO, and Web of Science through January 2011. STUDY SELECTION Investigators independently screened reports to identify randomized trials published in English of electronic CDSSs that were implemented in clinical settings; used by providers to aid decision making at the point of care; and reported clinical, health care process, workload, relationship-centered, economic, or provider use outcomes. DATA EXTRACTION Investigators extracted data about study design, participant characteristics, interventions, outcomes, and quality. DATA SYNTHESIS 148 randomized, controlled trials were included. A total of 128 (86%) assessed health care process measures, 29 (20%) assessed clinical outcomes, and 22 (15%) measured costs. Both commercially and locally developed CDSSs improved health care process measures related to performing preventive services (n= 25; odds ratio [OR], 1.42 [95% CI, 1.27 to 1.58]), ordering clinical studies (n= 20; OR, 1.72 [CI, 1.47 to 2.00]), and prescribing therapies (n= 46; OR, 1.57 [CI, 1.35 to 1.82]). Few studies measured potential unintended consequences or adverse effects. LIMITATIONS Studies were heterogeneous in interventions, populations, settings, and outcomes. Publication bias and selective reporting cannot be excluded. CONCLUSION Both commercially and locally developed CDSSs are effective at improving health care process measures across diverse settings, but evidence for clinical, economic, workload, and efficiency outcomes remains sparse. This review expands knowledge in the field by demonstrating the benefits of CDSSs outside of experienced academic centers. PRIMARY FUNDING SOURCE Agency for Healthcare Research and Quality.

A systematic review of newer pharmacotherapies for depression in adults: evidence report summary.

BACKGROUND Many biological, behavioral, social, and environmental factors may contribute to the delay or prevention of cognitive decline. PURPOSE To summarize evidence about putative risk and protective factors for cognitive decline in older adults and the effects of interventions for preserving cognition. DATA SOURCES English-language publications in MEDLINE, HuGEpedia, AlzGene, and the Cochrane Database of Systematic Reviews from 1984 through 27 October 2009. STUDY SELECTION Observational studies with 300 or more participants and randomized, controlled trials (RCTs) with 50 or more adult participants who were 50 years or older, drawn from general populations, and followed for at least 1 year were included. Relevant, good-quality systematic reviews were also eligible. DATA EXTRACTION Information on study design, outcomes, and quality were extracted by one researcher and verified by another. An overall rating of the quality of evidence was assigned by using the GRADE (Grading of Recommendations Assessment, Development, and Evaluation) criteria. DATA SYNTHESIS 127 observational studies, 22 RCTs, and 16 systematic reviews were reviewed in the areas of nutritional factors; medical factors and medications; social, economic, or behavioral factors; toxic environmental exposures; and genetics. Few of the factors had sufficient evidence to support an association with cognitive decline. On the basis of observational studies, evidence that supported the benefits of selected nutritional factors or cognitive, physical, or other leisure activities was limited. Current tobacco use, the apolipoprotein E epsilon4 genotype, and certain medical conditions were associated with increased risk. One RCT found a small, sustained benefit from cognitive training (high quality of evidence) and a small RCT reported that physical exercise helps to maintain cognitive function. LIMITATIONS The categorization and definition of exposures were heterogeneous. Few studies were designed a priori to assess associations between specific exposures and cognitive decline. The review included only English-language studies, prioritized categorical outcomes, and excluded small studies. CONCLUSION Few potentially beneficial factors were identified from the evidence on risk or protective factors associated with cognitive decline, but the overall quality of the evidence was low. PRIMARY FUNDING SOURCE Agency for Healthcare Research and Quality and the National Institute on Aging, through the Office of Medical Applications of Research, National Institutes of Health.

Grading quality of evidence and strength of recommendations in clinical practice guidelines

Depressive disorders, including major depression and dysthymia, are serious disabling illnesses. Approximately one in five persons is affected by a mood disorder at some point (1, 2). The attendant economic costs to society and personal burden to patients and families are enormous. In the United States, the estimated costs of treating depression and the costs incurred by lost productivity exceeded

The Effectiveness of Depression Care Management on Diabetes-Related Outcomes in Older Patients

44 billion in 1990 (3). The personal burden of depression includes higher mortality and impairment in multiple areas of functioning. The World Health Organization estimates that major depression is now the fourth most important cause worldwide of loss in disability-adjusted life-years and will be the second most important cause by 2020 (4, 5). In the late 1980s, the U.S. Department of Health and Human Services sponsored the development of standard treatment guidelines for major depression (6, 7). Since publication of the guidelines, widely publicized emphasis on recognizing and treating depression and development of many new pharmacotherapies have contributed to explosive growth in antidepressant prescribing and increasing pharmacy costs for health plans. Newer antidepressants and readily available herbal remedies have led to wider but sometimes confusing choices for clinicians. The purpose of this paper is to help clinicians make informed choices about antidepressants and herbal therapies for the treatment of depression. Because previous reviews have conclusively demonstrated the efficacy of older antidepressants, this paper focuses on 29 newer antidepressants and 3 herbal remedies (6, 8-11). Older antidepressants and psychosocial therapies are considered only when they are compared directly with a newer antidepressant. Our goal was to summarize data on the efficacy of newer antidepressants and herbal treatments compared with placebo, older antidepressants, and each other for a broad spectrum of depressive disorders. 1.0 Methods English-language and non-English-language literature was identified by using the Cochrane Collaboration Depression, Anxiety and Neurosis Groups specialized registry of 8451 clinical trial articles and from references of pertinent meta-analyses and consultation with experts (1, 6-8, 10, 12-54). The specialized registry contained trials addressing depression identified from multiple sources, including electronic databases, such as MEDLINE, EMBASE, PsychLIT, LILACS, Psyndex, SIGLE, CINAHL, Biological Abstracts, and The Cochrane Library; hand searches of 69 psychiatry-related journals; and contacts with 30 pharmaceutical companies. Sources were searched from 1980 to January 1998 to capture literature relevant to newly released antidepressants. The terms depression, depressive disorder, or dysthymic disorder were combined with a list of 32 specific newer antidepressants and herbal treatments to yield 1277 relevant records. The newer antidepressants are selective serotonin reuptake inhibitors (SSRIs); serotonin and noradrenaline reuptake inhibitors; selective norepinephrine reuptake inhibitors; reversible inhibitors of monoamine oxidase; 5-hydroxy-tryptophan (5-HT2) receptor antagonists; 5-HT1a receptor agonists; -aminobutyric acid (GABA) mimetics; dopamine reuptake inhibitors and antagonists; and herbal remedies, such as hypericum (Table 1). Randomized, controlled trials that were at least 6 weeks in duration; compared a newer antidepressant with another antidepressant (newer or older), placebo, or psychosocial intervention; involved participants with depressive disorders; and had a clinical outcome were reviewed. Two or more independent reviewers identified 315 such trials. Table 1. Classification and Dosage Range of Antidepressants Two persons independently abstracted data from each trial. Data were synthesized descriptively, with attention to participant and diagnostic descriptors; study design, including randomization method and blinding; intervention characteristics; and clinical outcomes. When the studies were conceptually homogenous, quantitative analyses were done by using an empirical Bayes random-effects estimator method. Conceptual homogeneity required similar trial design, comparison of similar drug classes, diagnostic homogeneity, and adequate numbers of trials to justify pooling. Statistical heterogeneity was evaluated by using the chi-square test for homogeneity and Galbraith plots to identify outliers. When statistical heterogeneity was identified, outlier studies were reviewed to identify possible reasons for heterogeneity and studies were reanalyzed without the outliers. Primary outcomes were symptomatic response rate, total discontinuation rates (dropouts), and rates of discontinuation because of adverse events. Secondary outcomes were health-related quality-of-life, functional status, and suicide. Response rates were defined as a 50% or greater improvement in symptoms as assessed by a depression symptoms rating scale or a rating of much or very much improved as assessed by a global assessment method. Response rates were computed by using a modified intention-to-treat approach. This approach computes response rates as the number of patients who stay in treatment and get better divided by the total number of randomly assigned patients. The modified intention-to-treat analysis produces an estimate of treatment effect that is conservative because it assumes that all persons who drop out of the study early receive no benefit. A sensitivity analysis was based on an end point method. In this method, the denominator for the risk ratio was the number of participants who completed follow-up or whose last observation was carried forward. Funnel plots with the Beggs rank-order correlation test and the Egger regression approach were used to estimate the possibility of publication bias whenever a quantitative meta-analysis was performed (54). Publication bias is the tendency of published studies to have different results (usually positive findings) from studies rejected from publication or never submitted for review (usually negative findings). More detailed methods and updates through September 1998 are available in the report on which this manuscript is based (55, 56). This study was funded by the Agency for Healthcare Research and Quality, which specified certain aspects of the study, such as a technical advisory panel and the report format. 2.0 Data Synthesis 2.1 Literature on Newer Antidepressants Three hundred fifteen randomized trials evaluated newer pharmacotherapies for depression. Because some trials had multiple treatment arms, the 315 trials yielded 355 pairwise comparisons. More than 90% of the trials focused on major depression (Table 2). Nine studies focused on dysthymia, a chronic mood disorder characterized by depressed mood for at least 2 years accompanied by two or more vegetative or psychological symptoms. Three studies each examined mixed-anxiety depression and subsyndromal depression, a less symptomatic, acute depression that causes less impairment in social or occupational functioning than major depression. Forty-four trials involved participants with heterogeneous groups of depressive disorders. Most studies (n =206) compared newer and older antidepressants. Serotonin reuptake inhibitors have been the most widely tested; 60 comparisons have been made with placebo, 123 with older antidepressants, and 36 with an SSRI or other newer agent. Table 2. Treatment Trials of Newer Antidepressants and Herbal Remedies More than 90% of the included trials were of short duration (6 to 8 weeks). Among trials reporting visit frequency, patients were seen weekly (62%), every other week (23%), monthly (5%), or on a schedule that varied over time. Trial reporting was often incomplete. Fewer than one third of studies described study settings, few studies described the nature and content of clinical interactions between providers and patients, and fewer than 10% described ethnic background or socioeconomic status of the participants. Of studies that described the study setting, 77 were based in mental health specialty practices and 27 were exclusively in primary care settings. Most studies reported whether recruitment involved inpatient or outpatient settings, and most (160 studies) were based in outpatient practices. Secondary outcomes (health-related quality-of-life, functional status, and suicide) were reported too infrequently for analysis. More than 90% of the randomized trials used double-blinded methods, but fewer than 5% reported whether blinding was successful. Few studies described the method of randomization or allocation and concealment. Approximately 30% of studies had relatively low dropout rates ( 20%), and approximately 20% reported dropout rates exceeding 40%. Analysis of adverse events was complicated by variability in data collection, including voluntary reporting, generic questioning, and standardized scales that may differ and affect the reliability of the overall estimates. 2.2 Major Depression: Newer Antidepressants for Initial Treatment The lifetime risk for major depressive disorder ranges from 10% to 25% for women and 5% to 12% for men, with a point prevalence rate of 5% to 9% for women and 2% to 3% for men (2, 6, 57). It affects persons of all ages, ethnicities, and socioeconomic circumstances. Major depression is characterized by at least 2 weeks of depressed mood or loss of interest or pleasure in nearly all activities (58). The person must experience at least four additional symptoms drawn from a list of vegetative (for example, loss of appetite) and psychological (for example, difficulty concentrating or making decisions) symptoms. In addition, the symptoms must cause clinically significant distress or impairment in social, occupational, or other areas of functioning. In the trials that we reviewed, the average severity of depression was moderate to moderately severe, as measured by a standard symptom rating scale (mean score, 24 [range, 14 to 32], stand