Elizabeth Gargon

Developing core outcome sets for clinical trials: issues to consider

The selection of appropriate outcomes or domains is crucial when designing clinical trials in order to compare directly the effects of different interventions in ways that minimize bias. If the findings are to influence policy and practice then the chosen outcomes need to be relevant and important to key stakeholders including patients and the public, health care professionals and others making decisions about health care. There is a growing recognition that insufficient attention has been paid to the outcomes measured in clinical trials. These issues could be addressed through the development and use of an agreed standardized collection of outcomes, known as a core outcome set, which should be measured and reported, as a minimum, in all trials for a specific clinical area. Accumulating work in this area has identified the need for general guidance on the development of core outcome sets. Key issues to consider in the development of a core outcome set include its scope, the stakeholder groups to involve, choice of consensus method and the achievement of a consensus.

Choosing important health outcomes for comparative effectiveness research: a systematic review.

Background A core outcome set (COS) is a standardised set of outcomes which should be measured and reported, as a minimum, in all effectiveness trials for a specific health area. This will allow results of studies to be compared, contrasted and combined as appropriate, as well as ensuring that all trials contribute usable information. The COMET (Core Outcome Measures for Effectiveness Trials) Initiative aims to support the development, reporting and adoption of COS. Central to this is a publically accessible online resource, populated with all available COS. The aim of the review we report here was to identify studies that sought to determine which outcomes or domains to measure in all clinical trials in a specific condition and to describe the methodological techniques used in these studies. Methods We developed a multi-faceted search strategy for electronic databases (MEDLINE, SCOPUS, and Cochrane Methodology Register). We included studies that sought to determine which outcomes/domains to measure in all clinical trials in a specific condition. Results A total of 250 reports relating to 198 studies were judged eligible for inclusion in the review. Studies covered various areas of health, most commonly cancer, rheumatology, neurology, heart and circulation, and dentistry and oral health. A variety of methods have been used to develop COS, including semi-structured discussion, unstructured group discussion, the Delphi Technique, Consensus Development Conference, surveys and Nominal Group Technique. The most common groups involved were clinical experts and non-clinical research experts. Thirty-one (16%) studies reported that the public had been involved in the process. The geographic locations of participants were predominantly North America (n = 164; 83%) and Europe (n = 150; 76%). Conclusions This systematic review identified many health areas where a COS has been developed, but also highlights important gaps. It is a further step towards a comprehensive, up-to-date database of COS. In addition, it shows the need for methodological guidance, including how to engage key stakeholder groups, particularly members of the public.

Driving up the quality and relevance of research through the use of agreed core outcomes

There is a growing recognition that insufficient attention has been paid to the selection of the outcomes to measure in clinical trials and clinical audit. Outcomes need to be relevant to patients, clinicians, purchasers and policy-makers if the findings of research are to influence practice and future research. In addition, standardization of outcomes is needed to combine data from different studies to allow evidence synthesis and to compare data sets. Inconsistent choice of outcome measures means that many meta-analyses are unable to include data from all the relevant studies. For example, the five most accessed Cochrane reviews in 2009, together with the top cited review in that year, all described inconsistencies in the outcomes reported in eligible trials. A call for the standardization of outcomes is a regular conclusion of systematic reviews. Furthermore, outcome reporting bias, defined as the bias arising from selecting outcomes for publication based on the results, affects many randomized trials and ‘is an under-recognized problem that affects the conclusions in a substantial proportion of Cochrane reviews’. That bias is likely to affect systematic reviews more widely as well as affecting individual studies when considered on their own. Similar problems occur with clinical audit, highlighting the importance of establishing national audits that use and report the same outcomes for all participants. All these issues could be addressed with the development and application of agreed standardized sets of outcomes that have been termed ‘core outcome sets’. These should be measured and reported, as a minimum, in all relevant clinical trials and national clinical audits for a specific condition. Adopting a core outcome set does not imply that a particular study, review or audit should be restricted to only those outcomes. Rather, the expectation is that, as a minimum, core outcomes will always be collected and reported to allow the results of trials to be compared, contrasted and combined as appropriate. The adoption of core outcomes would have implications across all areas of research in health and health care, reduce heterogeneity between trials, and lead to research that is more likely to have measured relevant outcomes. Importantly, they would enhance the value of evidence synthesis by reducing the risk of outcome reporting bias and ensuring that all trials contribute usable information. In addition, they will increase the efficiency and value of research. An important rationale for core outcome sets is that outcomes currently reported for trials do not consistently reflect endpoints that are meaningful for patients. Examples exist where trials failed to include all outcomes important to patients and where involvement of patients has identified an outcome that might not have been considered by practitioners on their own. Despite increasing recognition of the importance of incorporating patients’ opinions, their involvement has been limited. Recent regulatory guidance in the USA requires documented evidence of patient input during the development of instruments to measure patient reported outcomes (PROs). However, measurement of PROs in clinical trials is hampered by the multiplicity and heterogeneity of tools currently available. Many generic, diseaseand domain-specific instruments have been developed and validated, each containing multiple scales and items. As a result, synthesis of PRO data from trials is difficult and reviews aiming to summarize treatment effect according to PROs may fail. There is synergy between the development of core outcome sets for trials and work to select up to seven outcomes that are important to patients for inclusion in Summary of Findings tables in systematic reviews. Developed by the GRADE group (http://www.gradeworkinggroup.org), Cochrane reviews have featured such tables since 2008, and they play a key role in presenting research in guidelines, such as those produced by the World Health Organization (WHO). The development of core outcome sets needs to be accelerated and undertaken in ways that maximize efficiency. The COMET (Core Outcome Measures in Effectiveness Trials, http://www.comet-initiative.org) Initiative in the UK brings together researchers interested in the development and application of core outcome sets. These include key participants in a collaboration of researchers in rheumatology who have done the most notable work to date in this area. The COMET Initiative was launched in January 2010, with a second meeting in July 2011. Attendees included trialists, systematic reviewers, patients, clinicians, journal editors, research funders, policy-makers, people responsible for trials registries and regulators. Data on individual studies, both published and ongoing, are being included in a free, publically available internet-based resource. This will be updated periodically, to minimize the risk of duplication. Seventy-eight published or ongoing studies have already been entered into the repository. In addition, published reviews of outcomes used in clinical trials or studies examining patients’ views, will be entered.

The COMET (Core Outcome Measures in Effectiveness Trials) Initiative

Why standardise outcomes? The design of new trials would be simplified, the risk of measuring inappropriate outcomes would be reduced, and selective reporting of outcomes less likely. It would be easier to compare, contrast and combine studies in systematic reviews. Core outcome sets would help review authors to present their findings clearly and succinctly, for example within Summary of Findings tables.

Adverse drug reactions in children--a systematic review.

Background Adverse drug reactions in children are an important public health problem. We have undertaken a systematic review of observational studies in children in three settings: causing admission to hospital, occurring during hospital stay and occurring in the community. We were particularly interested in understanding how ADRs might be better detected, assessed and avoided. Methods and Findings We searched nineteen electronic databases using a comprehensive search strategy. In total, 102 studies were included. The primary outcome was any clinical event described as an adverse drug reaction to one or more drugs. Additional information relating to the ADR was collected: associated drug classification; clinical presentation; associated risk factors; methods used for assessing causality, severity, and avoidability. Seventy one percent (72/102) of studies assessed causality, and thirty four percent (34/102) performed a severity assessment. Only nineteen studies (19%) assessed avoidability. Incidence rates for ADRs causing hospital admission ranged from 0.4% to 10.3% of all children (pooled estimate of 2.9% (2.6%, 3.1%)) and from 0.6% to 16.8% of all children exposed to a drug during hospital stay. Anti-infectives and anti-epileptics were the most frequently reported therapeutic class associated with ADRs in children admitted to hospital (17 studies; 12 studies respectively) and children in hospital (24 studies; 14 studies respectively), while anti-infectives and non-steroidal anti-inflammatory drugs (NSAIDs) were frequently reported as associated with ADRs in outpatient children (13 studies; 6 studies respectively). Fourteen studies reported rates ranging from 7%–98% of ADRs being either definitely/possibly avoidable. Conclusions There is extensive literature which investigates ADRs in children. Although these studies provide estimates of incidence in different settings and some indication of the therapeutic classes most frequently associated with ADRs, further work is needed to address how such ADRs may be prevented.

Can a core outcome set improve the quality of systematic reviews? – a survey of the Co-ordinating Editors of Cochrane review groups

BackgroundMissing outcome data or the inconsistent reporting of outcome data in clinical research can affect the quality of evidence within a systematic review. A potential solution is an agreed standardized set of outcomes known as a core outcome set (COS) to be measured in all studies for a specific condition. We investigated the amount of missing patient data for primary outcomes in Cochrane systematic reviews, and surveyed the Co-ordinating Editors of Cochrane Review Groups (CRGs) on issues related to the standardization of outcomes in their CRG’s reviews. These groups are responsible for the more than 7,000 protocols and full versions of Cochrane Reviews that are currently available, and the several hundred new reviews published each year, presenting the world’s largest collection of standardized systematic reviews in health care.MethodsUsing an unselected cohort of Cochrane Reviews, we calculated and presented the percentage of missing patient data for the primary outcome measure chosen for each review published by each CRG. We also surveyed the CRG Co-ordinating Editors to see what their policies are with regards to outcome selection and outcomes to include in the Summary of Finding (SoF) tables in their Cochrane Reviews. They were also asked to list the main advantages and challenges of standardizing outcomes across all reviews within their CRG.ResultsIn one fifth of the 283 reviews in the sample, more than 50% of the patient data for the primary outcome was missing. Responses to the survey were received from 90% of Co-ordinating Editors. Thirty-six percent of CRGs have a centralized policy regarding which outcomes to include in the SoF table and 73% of Co-ordinating Editors thought that a COS for effectiveness trials should be used routinely for a SoF table.ConclusionsThe reliability of systematic reviews, in particular meta-analyses they contain, can be improved if more attention is paid to missing outcome data. The availability of COSs for specific health conditions might help with this and the concept has support from the majority of Co-ordinating Editors in CRGs.

The COMET Handbook: Version 1.0

The selection of appropriate outcomes is crucial when designing clinical trials in order to compare the effects of different interventions directly. For the findings to influence policy and practice, the outcomes need to be relevant and important to key stakeholders including patients and the public, health care professionals and others making decisions about health care. It is now widely acknowledged that insufficient attention has been paid to the choice of outcomes measured in clinical trials. Researchers are increasingly addressing this issue through the development and use of a core outcome set, an agreed standardised collection of outcomes which should be measured and reported, as a minimum, in all trials for a specific clinical area.Accumulating work in this area has identified the need for guidance on the development, implementation, evaluation and updating of core outcome sets. This Handbook, developed by the COMET Initiative, brings together current thinking and methodological research regarding those issues. We recommend a four-step process to develop a core outcome set. The aim is to update the contents of the Handbook as further research is identified.

Core Outcome Set–STAndards for Reporting: The COS-STAR Statement

Background Core outcome sets (COS) can enhance the relevance of research by ensuring that outcomes of importance to health service users and other people making choices about health care in a particular topic area are measured routinely. Over 200 COS to date have been developed, but the clarity of these reports is suboptimal. COS studies will not achieve their goal if reports of COS are not complete and transparent. Methods and Findings In recognition of these issues, an international group that included experienced COS developers, methodologists, journal editors, potential users of COS (clinical trialists, systematic reviewers, and clinical guideline developers), and patient representatives developed the Core Outcome Set–STAndards for Reporting (COS-STAR) Statement as a reporting guideline for COS studies. The developmental process consisted of an initial reporting item generation stage and a two-round Delphi survey involving nearly 200 participants representing key stakeholder groups, followed by a consensus meeting. The COS-STAR Statement consists of a checklist of 18 items considered essential for transparent and complete reporting in all COS studies. The checklist items focus on the introduction, methods, results, and discussion section of a manuscript describing the development of a particular COS. A limitation of the COS-STAR Statement is that it was developed without representative views of low- and middle-income countries. COS have equal relevance to studies conducted in these areas, and, subsequently, this guideline may need to evolve over time to encompass any additional challenges from developing COS in these areas. Conclusions With many ongoing COS studies underway, the COS-STAR Statement should be a helpful resource to improve the reporting of COS studies for the benefit of all COS users.

The COMET Initiative database: progress and activities from 2011 to 2013

The Core Outcome Measures in Effectiveness Trials (COMET) Initiative database is an international repository of studies relevant to the development of core outcome sets. By the end of 2013, it included a unique collection of 306 studies. The website is increasingly being used, with more than 12,000 visits in 2013 (a 55% increase over 2012), 8,369 unique visitors (a 53% increase) and 6,844 new visitors (a 48% increase). There has been a rise in visits from outside the United Kingdom, with 2,405 such visits in 2013 (30% of all visits). By December 2013, a total of 4,205 searches had been completed, with 2,139 in 2013 alone.

Choosing important health outcomes for comparative effectiveness research: An updated review and user survey.

Background A COS represents an agreed minimum set of outcomes that should be measured and reported in all trials of a specific condition. The COMET (Core Outcome Measures in Effectiveness Trials) initiative aims to collate and stimulate the development and application of COS, by including data on relevant studies within a publically available internet-based resource. In recent years, there has been an interest in increasing the development of COS. Therefore, this study aimed to provide an update of a previous review, and examine the quality of development of COS. A further aim was to understand the reasons why individuals are searching the COMET database. Methods A multi-faceted search strategy was followed, in order to identify studies that sought to determine which outcomes/domains to measure in clinical trials of a specific condition. Additionally, a pop up survey was added to the COMET website, to ascertain why people were searching the COMET database. Results Thirty-two reports relating to 29 studies were eligible for inclusion in the review. There has been an improvement in the description of the scope of a COS and an increase in the proportion of studies using literature/systematic reviews and the Delphi technique. Clinical experts continue to be the most common group involved in developing COS, however patient and public involvement has increased. The pop-up survey revealed the most common reasons for visiting the COMET website to be thinking about developing a COS and planning a clinical trial. Conclusions This update demonstrates that recent studies appear to have adopted a more structured approach towards COS development and public representation has increased. However, there remains a need for developers to adequately describe details about the scope of COS, and for greater public engagement. The COMET database appears to be a useful resource for both COS developers and users of COS.