Matthew J. Page

Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study.

Background Systematic reviews (SRs) can help decision makers interpret the deluge of published biomedical literature. However, a SR may be of limited use if the methods used to conduct the SR are flawed, and reporting of the SR is incomplete. To our knowledge, since 2004 there has been no cross-sectional study of the prevalence, focus, and completeness of reporting of SRs across different specialties. Therefore, the aim of our study was to investigate the epidemiological and reporting characteristics of a more recent cross-section of SRs. Methods and Findings We searched MEDLINE to identify potentially eligible SRs indexed during the month of February 2014. Citations were screened using prespecified eligibility criteria. Epidemiological and reporting characteristics of a random sample of 300 SRs were extracted by one reviewer, with a 10% sample extracted in duplicate. We compared characteristics of Cochrane versus non-Cochrane reviews, and the 2014 sample of SRs versus a 2004 sample of SRs. We identified 682 SRs, suggesting that more than 8,000 SRs are being indexed in MEDLINE annually, corresponding to a 3-fold increase over the last decade. The majority of SRs addressed a therapeutic question and were conducted by authors based in China, the UK, or the US; they included a median of 15 studies involving 2,072 participants. Meta-analysis was performed in 63% of SRs, mostly using standard pairwise methods. Study risk of bias/quality assessment was performed in 70% of SRs but was rarely incorporated into the analysis (16%). Few SRs (7%) searched sources of unpublished data, and the risk of publication bias was considered in less than half of SRs. Reporting quality was highly variable; at least a third of SRs did not report use of a SR protocol, eligibility criteria relating to publication status, years of coverage of the search, a full Boolean search logic for at least one database, methods for data extraction, methods for study risk of bias assessment, a primary outcome, an abstract conclusion that incorporated study limitations, or the funding source of the SR. Cochrane SRs, which accounted for 15% of the sample, had more complete reporting than all other types of SRs. Reporting has generally improved since 2004, but remains suboptimal for many characteristics. Conclusions An increasing number of SRs are being published, and many are poorly conducted and reported. Strategies are needed to help reduce this avoidable waste in research.

Evaluation of a Theory-Informed Implementation Intervention for the Management of Acute Low Back Pain in General Medical Practice: The IMPLEMENT Cluster Randomised Trial

Introduction This cluster randomised trial evaluated an intervention to decrease x-ray referrals and increase giving advice to stay active for people with acute low back pain (LBP) in general practice. Methods General practices were randomised to either access to a guideline for acute LBP (control) or facilitated interactive workshops (intervention). We measured behavioural predictors (e.g. knowledge, attitudes and intentions) and fear avoidance beliefs. We were unable to recruit sufficient patients to measure our original primary outcomes so we introduced other outcomes measured at the general practitioner (GP) level: behavioural simulation (clinical decision about vignettes) and rates of x-ray and CT-scan (medical administrative data). All those not involved in the delivery of the intervention were blinded to allocation. Results 47 practices (53 GPs) were randomised to the control and 45 practices (59 GPs) to the intervention. The number of GPs available for analysis at 12 months varied by outcome due to missing confounder information; a minimum of 38 GPs were available from the intervention group, and a minimum of 40 GPs from the control group. For the behavioural constructs, although effect estimates were small, the intervention group GPs had greater intention of practising consistent with the guideline for the clinical behaviour of x-ray referral. For behavioural simulation, intervention group GPs were more likely to adhere to guideline recommendations about x-ray (OR 1.76, 95%CI 1.01, 3.05) and more likely to give advice to stay active (OR 4.49, 95%CI 1.90 to 10.60). Imaging referral was not statistically significantly different between groups and the potential importance of effects was unclear; rate ratio 0.87 (95%CI 0.68, 1.10) for x-ray or CT-scan. Conclusions The intervention led to small changes in GP intention to practice in a manner that is consistent with an evidence-based guideline, but it did not result in statistically significant changes in actual behaviour. Trial Registration Australian New Zealand Clinical Trials Registry ACTRN012606000098538

Empirical evidence of study design biases in randomized trials: Systematic review of meta-epidemiological studies

Objective To synthesise evidence on the average bias and heterogeneity associated with reported methodological features of randomized trials. Design Systematic review of meta-epidemiological studies. Methods We retrieved eligible studies included in a recent AHRQ-EPC review on this topic (latest search September 2012), and searched Ovid MEDLINE and Ovid EMBASE for studies indexed from Jan 2012-May 2015. Data were extracted by one author and verified by another. We combined estimates of average bias (e.g. ratio of odds ratios (ROR) or difference in standardised mean differences (dSMD)) in meta-analyses using the random-effects model. Analyses were stratified by type of outcome (“mortality” versus “other objective” versus “subjective”). Direction of effect was standardised so that ROR < 1 and dSMD < 0 denotes a larger intervention effect estimate in trials with an inadequate or unclear (versus adequate) characteristic. Results We included 24 studies. The available evidence suggests that intervention effect estimates may be exaggerated in trials with inadequate/unclear (versus adequate) sequence generation (ROR 0.93, 95% CI 0.86 to 0.99; 7 studies) and allocation concealment (ROR 0.90, 95% CI 0.84 to 0.97; 7 studies). For these characteristics, the average bias appeared to be larger in trials of subjective outcomes compared with other objective outcomes. Also, intervention effects for subjective outcomes appear to be exaggerated in trials with lack of/unclear blinding of participants (versus blinding) (dSMD -0.37, 95% CI -0.77 to 0.04; 2 studies), lack of/unclear blinding of outcome assessors (ROR 0.64, 95% CI 0.43 to 0.96; 1 study) and lack of/unclear double blinding (ROR 0.77, 95% CI 0.61 to 0.93; 1 study). The influence of other characteristics (e.g. unblinded trial personnel, attrition) is unclear. Conclusions Certain characteristics of randomized trials may exaggerate intervention effect estimates. The average bias appears to be greatest in trials of subjective outcomes. More research on several characteristics, particularly attrition and selective reporting, is needed.

Recruitment difficulties in a primary care cluster randomised trial: investigating factors contributing to general practitioners' recruitment of patients

BackgroundRecruitment of patients by health professionals is reported as one of the most challenging steps when undertaking studies in primary care settings. Numerous investigations of the barriers to patient recruitment in trials which recruit patients to receive an intervention have been published. However, we are not aware of any studies that have reported on the recruitment barriers as perceived by health professionals to recruiting patients into cluster randomised trials where patients do not directly receive an intervention. This particular subtype of cluster trial is commonly termed a professional-cluster trial. The aim of this study was to investigate factors that contributed to general practitioners recruitment of patients in a professional-cluster trial which evaluated the effectiveness of an intervention to increase general practitioners adherence to a clinical practice guideline for acute low-back pain.MethodGeneral practitioners enrolled in the study were posted a questionnaire, consisting of quantitative items and an open-ended question, to assess possible reasons for poor patient recruitment. Descriptive statistics were used to summarise quantitative items and responses to the open-ended question were coded into categories.ResultsSeventy-nine general practitioners completed at least one item (79/94 = 84%), representing 68 practices (85% practice response rate), and 44 provided a response to the open-ended question. General practitioners recalled inviting a median of two patients with acute low-back pain to participate in the trial over a seven-month period; they reported that they intended to recruit patients, but forgot to approach patients to participate; and they did not perceive that patients had a strong interest or disinterest in participating. Additional open-ended comments were generally consistent with the quantitative data.ConclusionA number of barriers to the recruitment of patients with acute low-back pain by general practitioners in a professional-cluster trial were identified. These barriers were similar to those that have been identified in the literature surrounding the recruitment of patients in individual patient randomised trials. To advance the evidence base for patient recruitment strategies in primary care settings, trialists undertaking professional-cluster trials need to develop and evaluate patient recruitment strategies that minimise the efforts required by practice staff to recruit patients, while also meeting privacy and ethical responsibilities and minimising the risk of selection bias.Trial registrationAustralian New Zealand Clinical Trials Registry ACTRN012606000098538 (date registered 14/03/2006).

Core domain and outcome measurement sets for shoulder pain trials are needed: systematic review of physical therapy trials

OBJECTIVES To explore the outcome domains and measurement instruments reported in published randomized controlled trials of physical therapy interventions for shoulder pain (rotator cuff disease, adhesive capsulitis, or nonspecific shoulder pain). STUDY DESIGN AND SETTING We included trials comparing physical therapy to any other intervention for shoulder pain, indexed up to March 2015 in CENTRAL, MEDLINE, EMBASE, or CINAHL Plus. Two authors independently selected trials for inclusion and extracted information on the domains and measurement instruments reported. RESULTS We included 171 trials. Most trials measured pain (87%), function (72%), and range of movement (67%), whereas adverse events, global assessment of treatment success, strength, and health-related quality of life were measured in 18-27% of trials, and work disability and referral for surgery were measured in less than 5% of trials. Thirty-five different measurement instruments for pain and 29 for function were noted. Measurement of function increased markedly from 1973 to 2014. In rotator cuff disease trials, there was a more frequent measurement of pain and strength and a less frequent measurement of range of movement compared with adhesive capsulitis trials. CONCLUSIONS There was wide diversity in the domains and measurement instruments reported. Our results provide the foundation for the development of a core domain and outcome measurement set for use in future shoulder pain trials.

Alcohol use and substance use disorders in Gulf War, Afghanistan, and Iraq War veterans compared with nondeployed military personnel.

Although recent veterans have been found to be at increased risk of psychiatric disorders, limited research has focused on alcohol or substance use disorders. This systematic review and meta-analysis examined whether alcohol or substance use disorders were more common in Gulf War, Afghanistan, and Iraq War veterans compared with military comparison groups nondeployed to the corresponding conflict, including never deployed personnel. Literature was searched (1990-2014) in multiple electronic databases. Studies were assessed for eligibility and quality, including risk of bias. Eighteen studies (1997-2014) met inclusion criteria. Pooled analysis based on a random-effects model yielded a summary odds ratio of 1.33 (95% confidence interval (CI): 1.22, 1.46) for alcohol (7 studies) and 2.13 (95% CI: 0.96, 4.72) for substance use (3 studies) disorders among Gulf War veterans, as well as 1.36 (95% CI: 1.11, 1.66) for alcohol (7 studies) and 1.14 (95% CI: 1.04, 1.25) for substance use (4 studies) disorders among Iraq/Afghanistan veterans; meta-regressions found no statistically significant association between theater of war and alcohol use or substance use disorders. Our findings indicate that Gulf and Iraq/Afghanistan war veterans are at higher alcohol use disorder risk than nondeployed veterans, but further studies with increased power are needed to assess substance use disorder risk in Gulf War veteran populations.

The pharmacological and non-pharmacological treatment of attention deficit hyperactivity disorder in children and adolescents: A systematic review with network meta-analyses of randomised trials.

Background Attention deficit hyperactivity disorder (ADHD) is one of the most commonly diagnosed psychiatric disorders in childhood. A wide variety of treatments have been used for the management of ADHD. We aimed to compare the efficacy and safety of pharmacological, psychological and complementary and alternative medicine interventions for the treatment of ADHD in children and adolescents. Methods and findings We performed a systematic review with network meta-analyses. Randomised controlled trials (≥ 3 weeks follow-up) were identified from published and unpublished sources through searches in PubMed and the Cochrane Library (up to April 7, 2016). Interventions of interest were pharmacological (stimulants, non-stimulants, antidepressants, antipsychotics, and other unlicensed drugs), psychological (behavioural, cognitive training and neurofeedback) and complementary and alternative medicine (dietary therapy, fatty acids, amino acids, minerals, herbal therapy, homeopathy, and physical activity). The primary outcomes were efficacy (treatment response) and acceptability (all-cause discontinuation). Secondary outcomes included discontinuation due to adverse events (tolerability), as well as serious adverse events and specific adverse events. Random-effects Bayesian network meta-analyses were conducted to obtain estimates as odds ratios (ORs) with 95% credibility intervals. We analysed interventions by class and individually. 190 randomised trials (52 different interventions grouped in 32 therapeutic classes) that enrolled 26114 participants with ADHD were included in complex networks. At the class level, behavioural therapy (alone or in combination with stimulants), stimulants, and non-stimulant seemed significantly more efficacious than placebo. Behavioural therapy in combination with stimulants seemed superior to stimulants or non-stimulants. Stimulants seemed superior to behavioural therapy, cognitive training and non-stimulants. Behavioural therapy, stimulants and their combination showed the best profile of acceptability. Stimulants and non-stimulants seemed well tolerated. Among medications, methylphenidate, amphetamine, atomoxetine, guanfacine and clonidine seemed significantly more efficacious than placebo. Methylphenidate and amphetamine seemed more efficacious than atomoxetine and guanfacine. Methylphenidate and clonidine seemed better accepted than placebo and atomoxetine. Most of the efficacious pharmacological treatments were associated with harms (anorexia, weight loss and insomnia), but an increased risk of serious adverse events was not observed. There is lack of evidence for cognitive training, neurofeedback, antidepressants, antipsychotics, dietary therapy, fatty acids, and other complementary and alternative medicine. Overall findings were limited by the clinical and methodological heterogeneity, small sample sizes of trials, short-term follow-up, and the absence of high-quality evidence; consequently, results should be interpreted with caution. Conclusions Clinical differences may exist between the pharmacological and non-pharmacological treatment used for the management of ADHD. Uncertainties about therapies and the balance between benefits, costs and potential harms should be considered before starting treatment. There is an urgent need for high-quality randomised trials of the multiple treatments for ADHD in children and adolescents. PROSPERO, number CRD42014015008.

Stop this waste of people, animals and money

Our evidence disputes this view. We spent 12 months rigorously characterizing nearly 2,000 biomedical articles from more than 200 journals thought likely to be predatory. More than half of the corresponding authors hailed from highand upper-middle-income countries as defined by the World Bank. Of the 17% of sampled articles that reported a funding source, the most frequently named funder was the US National Institutes of Health (NIH). The United States produced more articles in our sample than all other countries save India. Harvard University (with 9 articles) in Cambridge, Massachusetts, and the University of Texas (with Predatory journals are easy to please. They seem to accept papers with little regard for quality, at a fraction of the cost charged by mainstream openaccess journals. These supposedly scholarly publishing entities are murky operations, making money by collecting fees while failing to deliver on their claims of being open access and failing to provide services such as peer review and archiving. Despite abundant evidence that the bar is low, not much is known about who publishes in this shady realm, and what the papers are like. Common wisdom assumes that the hazard of predatory publishing is restricted mainly to the developing world. In one famous sting, a journalist for Science sent a purposely flawed paper to 140 presumed predatory titles (and to a roughly equal number of other open-access titles), pretending to be a biologist based in African capital cities. At least two earlier, smaller surveys found that most authors were in India or elsewhere in Asia. A campaign to warn scholars about predatory journals has concentrated its efforts in Africa, China, India, the Middle East and Russia. Frequent, aggressive solicitations from predatory publishers are generally considered merely a nuisance for scientists from rich countries, not a threat to scholarly integrity. Stop this waste of people, animals and money

Mass Production of Systematic Reviews and Meta‐analyses: An Exercise in Mega‐silliness?

I n 1978, the distinguished professor of psychology Hans Eysenck delivered a scathing critique of what was then a new method, that of meta-analysis, which he described as “an exercise in mega-silliness.” A provocative article by John Ioannidis in this issue of the journal1 suggests that “mega-silliness” may be an appropriate characterization of what the meta-analysis literature has become. With surveys of the PubMed database and other empirical evaluations, Ioannidis paints a disturbing picture of the current state of affairs, where researchers are producing, in epidemic proportions, systematic reviews and meta-analyses that are redundant, misleading, or serving vested interests. Ioannidis presents an astounding case of 21 different meta-analyses of statins for atrial fibrillation in cardiac surgery published within a period of 7 years, with some of these having practically identical results.1 Moreover, his findings are in line with our recent cross-sectional study of systematic reviews of biomedical research.2 We identified 682 systematic reviews indexed in MEDLINE in a single month (February 2014), which is equivalent to 22 reviews published per day. The majority of reviews did not consider study risk of biases or other reporting biases when drawing conclusions. Quality of reporting was highly variable: at least a third of reviews did not report use of a protocol, the search logic for at least one database, methods for data extraction and risk of bias assessment, or the funding source of the review. In addition, at least a third used statistical methods that are discouraged by leading systematic review organizations (eg, the Cochrane

A third of systematic reviews changed or did not specify the primary outcome: a PROSPERO register study

OBJECTIVES To examine outcome reporting bias of systematic reviews registered in PROSPERO. STUDY DESIGN AND SETTING Retrospective cohort study. The primary outcomes from systematic review publications were compared with those reported in the corresponding PROSPERO records; discrepancies in the primary outcomes were assessed as upgrades, additions, omissions, or downgrades. Relative risks (RRs) and 95% confidence intervals (CI) were calculated to determine the likelihood of having a change in primary outcome when the meta-analysis result was favorable and statistically significant. RESULTS Ninety-six systematic reviews were published. A discrepancy in the primary outcome occurred in 32% of the included reviews and 39% of the reviews did not explicitly specify a primary outcome(s); 6% of the primary outcomes were omitted. There was no significant increased risk of adding/upgrading (RR, 2.14; 95% CI: 0.53, 8.63) or decreased risk of downgrading (RR, 0.76; 95% CI: 0.27, 2.17) an outcome when the meta-analysis result was favorable and statistically significant. As well, there was no significant increased risk of adding/upgrading (RR, 0.89; 95% CI: 0.31, 2.53) or decreased risk of downgrading (RR, 0.56; 95% CI: 0.29, 1.08) an outcome when the conclusion was positive. CONCLUSIONS We recommend review authors carefully consider primary outcome selection, and journals are encouraged to focus acceptance on registered systematic reviews.